If a user's session cookie expired, the next REST API request their
browser did would go into the json_unauthorized code path. This
returned a response with a WWW-Authenticate tag for HTTP Basic Auth
(since that's what the REST API uses), even for /json requests which
should only be authenticated using session auth.
We fix this by explicitly passing the desired WWW-Authenticate state.
Fixes: #800.
The recent changes to api_fetch_api_key to receive detailed data via
the "return_data" object did not properly update the LDAP backend to
accept that argument, causing mobile password authentication to not
work with the LDAP backend.
tools/hash_reqs.py generates a hash of a requirements file. It
does that by generating a sorted list of unique dependencies referred
to by that requirements file. To do that, it also recurses into other
requirements files specified inside that requirements file.
After annotating rate_limiter.rules, mypy complained that rules does
not support cmp. So use key to customize sort instead of cmp.
Python docs also recommend using key over cmp.
zerver.lib.initial_password.initial_password is supposed to return an
Optional[text_type], but it returns an Optional[binary_type] instead.
Encode the return value to make sure it returns an Optional[text_type].
Add a class 'BaseHandler' and make it a base class of OuterHandler,
QuoteHandler and CodeHandler. This will help annotate some functions
and improve type checking.
This was confusing, because we otherwise suggest consistently to users
that they should connect to their development environment on port
9991.
Fixes#1032.
Apparently, there are like 5 independently developed jquery-caret
plugins, none of which are great. The previous one we were using was
last modified in 2010. This new one comes from
https://github.com/acdvorak/jquery.caret and at least doesn't use
deprecated jQuery syntax and has a repository on GitHub.
This plugin is way larger than it needs to be for what it does, but we
can deal with that later.
Place all hotkey names into a set of three objects in hotkeys.js:
* hotkeys_shift_insensitive
These are keys where the behaviour is the same whether they are
pressed with shift or not.
* hotkeys_no_modifiers
These are keys where the event should only be fired when shift
is not being pressed.
* hotkeys_shift
These are keys where the event should only be fired when the key
is pressed simultaneously with shift.
Each object is a dictionary of key value pairs, with the key being
the keyscan code (e.which) for the key. This is normally the ASCII
key code. The value is an object with two properties, name (which
is the event name) and message_view_only, a boolean. Hotkeys with
message_view_only set to true will not be fired when the home tab
is obscured.
Following strings are marked translatable:
- All strings which are passed to `button.text` or which affect the
text of buttons.
- All strings passed to `placeholder`.
- All strings passed to `compose_error`.
Fixes#969
get_display_recipient's annotation clashes with other wrong annotations.
Fix those wrong annotations.
Since get_display_recipient returns a Union, use isinstance checks and
casts to make mypy checks succeed.
We'll likely want to follow this up with:
(1) Making Let's Encrypt the default instructions for getting a cert.
(2) Moving the "how to get an SSL cert" discussion to its own page.
(3) Eliminating the duplicate content in zulip.org/server.html
But this is definitely an improvement!
Identify functions which return QuerySets and give them a return type
`Sequence` with appropriate parameter. Typing them as QuerySet will
not be useful since generic stubs for QuerySets are not available and
not knowing the type of QuerySets is hardly useful for type checking.
generate_random_token used to return a value of type six.binary_type
and its return type was annotated as `str`. This commit fixes that
by making it return a value of type `six.text_type` and updating
the annotation accordingly.
Also fix clashing annnotations.
Change choices of UserProfile.avatar_sources and UserProfile.tutorial_status
from str literals to unicode literals. This is done because these fields
are CharFields, which are of type `six.text_type`. So the set of values
which they can take should also be of the type `six.text_type`.
Also fix clashing annotations.
Change `str` to `text_type` in annotations in zerver/models.py
related to realm emoji and realm filters.
Also fix clashing annotations in zerver/lib/bugdown/__init__.py.
Due to a cyclic dependency issue, functions having models as parameters
were annotated as Any.
That issue is fixed by importing models inside an `if False:` block,
so that mypy sees them but they are not imported at runtime.
In update_user_profile_caches, the return type in annotation was
marked as Any. Change that to None because, nothing is being returned
in that function.
This changes the type annotations for the cache keys in Zulip to be
consistently text_type, and updates the annotations for values that
are used as cache keys across the codebase.
str_utils.py has functions for converting strings from one type to
another. It also has a TypeVar called NonBinaryStr, which is like AnyStr
except that it doesn't allow bytes.
The previous Markdown syntax didn't work correctly on ReadTheDocs, and
now that we have Coveralls reporting backend test coverage as well,
describing our coverage as just "mypy static type" is no longer accurate.
Fixes#964.
Now that we have a working S3 mock and an effective way to toggle the
upload backend that Zulip is using, we can re-enable this important
end-to-end test of the Zulip S3 upload backend.
Previously, uploaded files were served:
* With S3UploadBackend, via get_uploaded_file (redirects to S3)
* With LocalUploadBackend in production, via nginx directly
* With LocalUploadBackend in development, via Django's static file server
This changes that last case to use get_uploaded_file in development,
which is a key step towards being able to do proper access control
authorization.
Does not affect production.
This has no functional changes; we just replace the old hacky
assignment of functions with assignment of the upload backend to a
variable.
I'm not totally happy with this, because we end up having to copy the
type annotations of the three methods 4 times each, but this should
make it a lot easier to test the (non-default-in-tests) S3 backend
using end-to-end tests, which would have caught
13bac1cc2a.
I expect we'll iterate on the interface over time; ideally, I'd like
all the code that checks LOCAL_UPLOADS_DIR to be inside upload.py, and
primarily in these classes.
* The warning contains a count of the number of people in the stream.
* An error appears if the warning is ignored and the user tries to
send the message anyway.
* The message cannot be sent until the warning is acknowledged or @all
/ @everyone is removed.
* This only applies to stream messages and not private messages.
Fixes#853.
Previously, we were checking if a particular user was the current user
in dozens of places in the codebase, and correct case-insensitive
checks were not used consistently, leading to bugs like #502.
Calling open() with mode 'w' or 'a' will create a file if it doesn't exist,
while mode 'r' will cause an exception. This can be easily tested with:
python -c 'open("test.tmp", "w")'
ls test.tmp
Also fix the annotation of zilencer.views.report_error.
The `report` arguments are a Dict containing both strings and the
`more_info` sub-dictionary, so we type them as Dict[str, Any].
[tweaked by tabbott]
Since relatively few systems have the typing module, this makes the
checks for whether the user is properly running our test scripts in
the virtualenv more likely to trigger well.
This should make users much more likely to be able to debug issues
where they ran Zulip outside the Vagrant environment or virtualenv.
[error messages tweaked by tabbott]
`makemessages` escapes the `%` sign in `.po` files, but Jinja2 does
not unescape it while replacing the tranlation strings. In Jinja2,
there is an updated implementation of gettext available called
new-style gettext which handles escaping better; this commit switches
to using that.
Fixes#906.
Type of parameter for function `is_recent`(line no.812) is `datetime`.
MyPy errors out, however, when the parameter is defined as `datetime`.
To get around, type `Any` is used.
Also, fixed a a small type annotation in users.py because email must
be a string because emails don't support UTF-8 at this time (according
a comment in gravatar_hash in avatar.py).
Currently this uses a Union type for connection_id; we need to figure
out what actually sets that and what its type is and fix that later
(see https://github.com/zulip/zulip/issues/896).
.message_edit_notice is too broad of a selector to
test if your most recent edit has posted. Also
check if the currently selected message is the one
that's been edited.
When you deactivate a user, visit another page, and then
come back, the user shows up in another table (the
"Deactivated Users" table), and no longer has the
strikeout styling of a recently deactivated user.
Using the strikeout class as a selector to test if the
user has been reactivated will of course not be a good
test, and if you have a slow machine, lose a
race condition.
When you deactivate a user, visit another page,
and then come back, the user shows up in another
table, and no longer has the strikeout styling
of a recently deactivated user. Using strikeout
class to test if the user has been reactivated
will of course not be a good test, and if you
have a slow machine, lose a race condition.
Also, fixed up the annotations for tornadoviews to better align with
how narrows was defined as `Iterable[Sequence[str]]` rather than
`List[Tuple[str, str]]`.
Had to add some "type: ignore" because the pattern used in match
doesn't affect the type returned. A fix for this issue has been pushed
to typeshed - https://github.com/python/typeshed/pull/244
These ones don't fix any bugs, because the mutable arg is never passed
outside of the callable or mutated. But it's good practice to not use
them in case those invariants are changed in the future.
[Substantially revised by tabbott]
This probably still has some bugs in it, but having mostly complete
annotations for models.py will help a lot for the annotations folks
are adding to other files.
In order to genericize use of Zulip outside companies,
all instances of coworkers have been changed to users.
NOTABLE EXCEPTION: When the Zulip instance is domain-
locked, the reference to coworkers remains. The reason
for this is twofold: first, the majority of Zulip instances
which require a particular domain will be locked to a
company, and second, the template variable for the domain
necessary should be added to the alert so it is clear
to the user what the domain needs to be for access.
Fixes: #861.
Currently (https://github.com/rtfd/recommonmark/issues/3)
the recommonmark bridge (which allows Sphinx to read
Markdown) does not support tables, so the directory structure
doc is now a bulleted list instead of a set of tables.
The order and structure of the Read The Docs documentation
was fairly arbitrary; here's a suggestion for how to change
its information architecture so it's more useful to a new
contributor. This adds a short Zulip overview, and a table
of contents and code contribution intro for new contributors.
Fixes: #668.
Signed-off-by: Sumana Harihareswara <sumanah@panix.com>
Fixes#747. Fixes#748.
This updates README.dev.md to include clear, step-by-step instructions
for setting up the Zulip dev environment Windows 10, OS X El Capitan,
Ubuntu 14.04 Trusty, and Ubuntu 16.04 Xenial. It is aimed at first-time
contributors.
I tested these instructions multiple times on each of the target
systems.
Also added is a "trobleshooting and common errors" section which
documents the most common errors contributors are likely to encounter
during setup and how to fix them.
Improvements based on feedback from @timabbott.
- Fixes whitespace issues so linter will pass.
- Updates memory requirement.
- Re-orders so operating systems are listed alphabetically.
- Updates headings to be clearer.
- Updates and adds ToC entries for clarity.
- Adds screen shot of Zulip dev environment running in browser.
- Adds details about using dev environment, including about logging.
- Misc other minor changes for clarity.
- Adds a stub for docs/logging.md
- Adds details about configuring Cygwin for native symlinks.
Just render the templates without the actual workflow to see if they
don't return a 500 error; this lets us catch various classes of
template bugs automatically.
Fixes#784.
Add two options to the `test-backend` script:
1. verbose
If given the `test-backend` script will give detailed output.
2. no-shallow
Default value is False. If given the `test-backend` script will
fail if it finds a template which is shallow tested.
This lets us cut out the line which hard-codes how deeply nested in
the tree the `run-mypy` script is, making it simpler to borrow these
scripts in other projects.
This stub file allows us to annotate view functions using the actual
types present in the bodies of the functions, rather than everything
having the type REQ.
In function bulk_add_subscriptions, some variables were named
`stream_name` but their type is Stream, not a string. Rename
those variables to `stream`.
Previously lister.py used to check whether the exclude path is a
substring of a path being considered. So it would fail when the
exclude path is an absolute path or uses '..' or '.'.
We need to update provision.py to compile the messages files, since
they are needed for the new i18n tests. And of course we need to
include the .mo files in release tarballs; there's a bit of complexity
there around how the tarball archives are created.
Since we merged cd2348e9ae more than a
month ago and haven't seen any noticable regresions as a result, it's
reasonable at this point to do a corresponding decrease in our
documented RAM requirements for the Zulip development environment.
Like the recent change blocking JSON endpoints for deactivated users
and users in deactivated realms, this change is a hardening
improvement. Those users should be unable to get an active session
anyway, but if somehow one is leaked, this means they won't be able to
access any user data.
Previously, api_fetch_api_key would not give clear error messages if
password auth was disabled or the user's realm had been deactivated;
additionally, the account disabled error stopped triggering when we
moved the active account check into the auth decorators.
While in theory users should be unable to get a valid session in order
to access these endpoints in the first place, this provides an extra
layer of hardering to prevent a deactivated user with a session from
accessing data via the old-style JSON API.
The security model for deactivated users (and users in deactivated
realms) being unable to access the service is intended to work via two
mechanisms:
* All active user sessions are deleted, and all login code paths
(where a user could get a new session) check whether the user (or
realm) is inactive before authorizing the request, preventing the
user from accessing the website and AJAX endpoints.
* All API code paths (which don't require a session) check whether the
user (and realm) are active.
However, this security model was not implemented correctly. In
particular, the check for whether a user has an active account in the
login process was done inside the login form's validators, which meant
that authentication mechanisms that did not use the login form
(e.g. Google and REMOTE_USER auth) could succeed in granting a session
even with an inactive account. The Zulip homepage would still fail to
load because the code for / includes an API call to Tornado authorized
by the user's token that would fail, but this mechanism could allow an
inactive user to access realm data or users to access data in a
deactivated realm.
This fixes the issue by adding explicit checks for inactive users and
inactive realms in all authentication backends (even those that were
already protected by the login form validator).
Mirror dummy users are already inactive, so we can remove the explicit
code around mirror dummy users.
The following commits add a complete set of tests for Zulip's inactive
user and realm security model.
In a deactivated realm, webhooks would still successfully send
messages, since there was no check for whether the realm was active in
api_key_only_webhook_view.
Long ago, there was work on an experimental integration model where
every user in a realm would have administrative control over all bots,
with the goal of simplifying the process of setting up communally
administered bots for smaller teams. While that new model was never
fully implemented (and thus never setup as an option), an error in
that original implementation meant that the data on all bots in a
realm, including their API keys, was sent to the browsers of users via
the `realm_bots` variable in `page_params`. The data wasn't displayed
in the UI for non-admin users, but was available via e.g. the
javascript console.
This commit updates this behavior to only send sensitive bot data like
API keys to the owner of the bot (and realm admins).
We may in the future implement a model simplifying communally
administered integrations, but if we do that, those bots should be
limited in their capabilities (e.g. only able to send webhook
messages).
This bug has been present since Zulip was released as open source.
The old code for this lookup was unnecessarily complicated because we
were working around Guardian, where the `is_realm_admin` check was
extremely expensive.
Previously we relied on having two matching list of fields for the
get_active_user_dicts_in_realm, one in the actual code and the other
in the caching system. By unifying these lists to have a single
source, we eliminate a class of caching bugs we might otherwise
regularly introduce.
This results in a substantial performance improvement for all of
Zulip's backend templates.
Changes in templates:
- Change `block.super` to `super()`.
- Remove `load` tag because Jinja2 doesn't support it.
- Use `minified_js()|safe` instead of `{% minified_js %}`.
- Use `compressed_css()|safe` instead of `{% compressed_css %}`.
- `forloop.first` -> `loop.first`.
- Use `{{ csrf_input }}` instead of `{% csrf_token %}`.
- Use `{# ... #}` instead of `{% comment %}`.
- Use `url()` instead of `{% url %}`.
- Use `_()` instead of `{% trans %}` because in Jinja `trans` is a block tag.
- Use `{% trans %}` instead of `{% blocktrans %}`.
- Use `{% raw %}` instead of `{% verbatim %}`.
Changes in tools:
- Check for `trans` block in `check-templates` instead of `blocktrans`
Changes in backend:
- Create custom `render_to_response` function which takes `request` objects
instead of `RequestContext` object. There are two reasons to do this:
1. `RequestContext` is not compatible with Jinja2
2. `RequestContext` in `render_to_response` is deprecated.
- Add Jinja2 related support files in zproject/jinja2 directory. It
includes a custom backend and a template renderer, compressors for js
and css and Jinja2 environment handler.
- Enable `slugify` and `pluralize` filters in Jinja2 environment.
Fixes#620.
In theory these should be the same, but in misconfigured environments
(such at Travis CI) where /etc/hosts has multiple entries for
"localhost", 127.0.0.1 is safer than "localhost".
This fixes an issue introduced by
b21454d05e, where the typed_ast module
wouldn't build properly when provisioning, because we didn't have the
necessary headers installed.
To avoid the potential for introducing regressions here, we carefully
pass a default to REQ or not based on how the existing webhook's
parsing code worked. In the longer term, we'll want to make the
behavior consistent.
This makes mypy about 15% faster running on the Zulip codebase (from
7s=>6s on my laptop), which seems worth it for losing a couple files.
This option requires a new dependency, which we add to the
mypy-specific requirements.txt file.
Also update the mypy command line to not use deprecated argument names.
This introduces a few errors, so we exclude the relevant files to keep
the mypy output clean.
This fixes an exception where client_id was never set in an error code
path. It shouldn't be needed, but I think this makes the code clearer
and this will help in debugging the actual problem.
Related to #753.
The error message for a test file that doesn't import properly was
previously pretty difficult to understand and it wasn't clear how to
debug the issue.
This should save a couple minutes on the time it takes to provision a
Zulip environment on a machine that has provisioned with the same
requirements.txt file content before.
We can't yet use this in Travis CI because Travis CI doesn't support
using both sudo and caching in the same build.
This commit adds the capability to keep track and remove uploaded
files. Unclaimed attachments are files that have been uploaded to the
server but are not referred in any messages. A management command to
remove old unclaimed files after a week is also included.
Tests for getting the file referred in messages are also included.
Camo is a caching image proxy, used in Zulip to avoid mixed-content
warnings by proxying HTTP image content over HTTPS. We've been using
it in zulip.com production for years; this change makes it available
in standalone Zulip deployments.
This should save several minutes off the Travis CI `production`
suite's runtime, since previously we were doing the full apt upgrade
process twice, resulting in things like multiple expensive rebuilds of
the initramfs.
Add 'six' to setup-py3k, because it is being used in tools/lister.py.
Add 'typing' to setup-py3k, so that tools/lister.py can be type
annotated in the future.
Since we don't have a stable way to get the Dropbox preview failure
image (and it was sorta a weird setup anyway), it seems best to just
remove the condition.
Previously, the user list would remain filtered after a user hit enter
to start composing a message to a user, leaving them in a state with a
partial user list.
Fixes#360.
Previously we needed to use a specified password when activating a
formerly mirror dummy user, in order for that user to be able to
(re)set their password and login. Now that we have our own password
reset form, this is no longer required.
Previously, if a user had only authenticated via Google auth, they
would be unable to reset their password in order to set one (which is
needed to setup the mobile apps, for example).
Now tools/check-py3 will by default run all fixers together. This is
quicker but doesn't indicate which fixers caused the failure. The
newly added option --find-fixers falls back to the old way of checking
each fixer separately if the quick check fails.
Fixes#710.
Apparently LXC 2 removed support for the `-B best` option in
lxc-create, and Vagrant hasn't been updated appropriately yet, so we
need to add a workaround to explicitly specify a backing store.
Fixes#718.
This manifested as errors of the form:
"""
There was an error executing ["sudo",
"/usr/local/bin/vagrant-lxc-wrapper", "lxc-create", "-B", "best",
"--template",
"/home/tabbott/.vagrant.d/gems/gems/vagrant-lxc-1.2.1/scripts/lxc-template",
"--name", "zulip_default_1461801696512_85064", "--", "--tarball",
"/home/tabbott/.vagrant.d/boxes/fgrehm-VAGRANTSLASH-trusty64-lxc/1.2.0/lxc/rootfs.tar.gz",
"--config",
"/home/tabbott/.vagrant.d/boxes/fgrehm-VAGRANTSLASH-trusty64-lxc/1.2.0/lxc/lxc-config"]
"""
Several recently merged webhooks were incorrectly not checking that
the actual webhook result didn't return an error. While they would
usually still fail in most cases when checking whether the message
came back correctly, this hid the root cause errors and thus made it
much harder to debug.
We were incorrectly applying the rate limiting rules to webhooks even
if rate limiting was disabled (as in the test suite), causing test
failures when the total number of webhook tests in Zulip got too high.
In theory, tools like populate_db should probably be in zerver, not
zilencer, but until we migrate them out, we need to include these in
EXTRA_INSTALLED_APPS in development.
The previous separated-out configuration wasn't helping us, and this
makes it easier to make the extra installed applications pluggable in
the following commits.
This will merge conflict with every new integraiton in flight, which
is unfortunate, but will make there be fewer merge conflicts as people
add new webhooks in the future (currently, every pair of new
integrations conflict because folks are adding them all at the end,
whereas after this change, there will only be merge conflicts when
adding two integrations near each other alphabetically).
This integration relies on the Teamcity "tcWebHooks" plugin which is
available at
https://netwolfuk.wordpress.com/category/teamcity/tcplugins/tcwebhooks/
It posts build fail and success notifications to a stream specified in
the webhook URL.
It uses the name of the build configuration as the topic.
For personal builds, it tries to map the Teamcity username to a Zulip
username, and sends a private message to that person.
As documented in https://github.com/zulip/zulip/issues/441, Guardian
has quite poor performance, and in fact almost 50% of the time spent
running the Zulip backend test suite on my laptop was inside Guardian.
As part of this migration, we also clean up the old API_SUPER_USERS
variable used to mark EMAIL_GATEWAY_BOT as an API super user; now that
permission is managed entirely via the database.
When rebasing past this commit, developers will need to do a
`manage.py migrate` in order to apply the migration changes before the
server will run again.
We can't yet remove Guardian from INSTALLED_APPS, requirements.txt,
etc. in this release, because otherwise the reverse migration won't
work.
Fixes#441.
When uploaded avatar image is not a valid image file, PIL raises
IOError. Catch the IOError raised by PIL and raise JsonableError.
This will return a response with status code 400.
S3Test is now only the S3-specific test (which isn't even run), so we
can now invest in making FileUploadTest have good coverage of the
(local) file upload code paths.
The main function of prompting inside `manage.py migrate` is to ask
the user if they want to delete stale content-types, which is
unimportant and likely scary, so we disable doing so.
Previously, the UserProfile objects were created in the order
generated by a Set, which meant tests would randomly start failing if
the code that runs before this part of populate_db changed (and thus
caused the Set object used to pass users into bulk_create_users to
have a different order when enumerated).
This fixes the issue in two ways -- one by sorting the users inside
bulk_create_users, and second by attaching subscriptions to users
based on a deterministic ordering.
The restarted Tornado processes seemed to escape the process group and
thus continue running after run-dev.py finished.
While we're at it, we don't need to dump/reload event queues in the
test suite either.
Move recenter_pointer_on_display, suppress_scroll_pointer_update,
fast_forward_pointer, furthest_read, and server_furthest_read to
a new pointer module in pointer.js.
The `with sh.sudo` pattern that we were using in python-sh was
deprecated, and emperically hangs on Ubuntu xenial. Since in general
the use of python-sh/python-pbs caused trouble (requiring extra
dependencies, confusing syntax), this just removes it.
We replace it with a new zulip_tools.py library function that echoes
the command line and streams the output.
We do the same to install-phantomjs so we can remove that dependency.
Previously, the Zulip subscriptions page's error bar would always be
at the very top of the scrollable view, and thus would likely be out
of view when an error happened. This fixes it by having the error bar
always placed below the search box (and thus visible regardless of
where in the scrollable streams view we are).
Fixes: #515.
[commit message and comments expanded by tabbott]
This reverts commit d936bf61f9.
We no longer need this since we've migrated to specifying the
dependencies in the typing module that we're actually using.
This fixes an issue where this worker wasn't even being installed
properly in a way that sets us up for doing further reorganization of
the Zulip Nagios plugins.
tools/travis/py3k used to always exit with exit code 0.
It should exit with 1 when fixers detect a compatibility issue.
py3k used [ -z "$failed" ] to check if there was a failure.
This is wrong, since if no failure has occured, failed=0,
and -z checks if a string is of zero length. This commit also
fixes this bug.
In py3k, "git reset --hard" was called only if
libmodernize.fixes.fix_dict_six changed files and some of those
changes are not considered false positives by py3k.
But if all of those changes are not considered false positives
by py3k, then "git reset --hard" is not called and the repository
is no longer clean.
This commit fixes this bug.
tools/travis/py3k used to only check files whose names ended with .py.
Now it also checks python scripts which don't have an extension.
It uses tools/lister.py to get a list of all python files.
The previous version of sanitize_name dropped all unicode characters
and mangled filenames with multiple `.`s in the extension, leading to
confusing URLs for files uploaded to Zulip.
Fixes#321.
[tweaked significantly by tabbott]
This has the side effect of making lint-all check all shell scripts,
not just those under scripts/, tools/, and bin/.
[commit message expanded by tabbott]
Make module tools/lister.py which lists all files in a directory
tracked by git. This is done because lister.py will be used by other
scripts in the future which have to introspect files in the repository,
like linters, static code checkers, etc.
It's always been the case that in production, Tornado dumps all the
event queues when shut down so that they can be reloaded by the
replacement Tornado process. This never worked in development because
the codepath for auto-reload didn't go through either a signal or
sys.exit (it re-execs the process instead).
This meant that we didn't have a mechanism for testing the event queue
dump/load functionality in the development environment. We fix this
by adding such dumping/loading. However, this breaks the automatic
reloading of open browser windows on a server restart, so we add that
back in by adjusting the special `restart` events to pass a special
`immediate` flag when used in development.
This also has the benefit of removing the "Bad event queue" errors one
would get on every file save induced restart on the Python console.
Apparently it isn't always the case that removal of jquery and the DOM
prevents cleanup_event_queue from being called via the postunload
hook, so add a check to avoid it being double-called.
Previously, the browser might restart a get_events operation even
while it was in the middle of executing a `DELETE /events` query to
cause its event queue to be de-allocated. This was a rare race
condition when we weren't notifying clients when event queues were
de-allocated, but this will become a common case in the next commit.
This is a no-op right now, but we'll want the new structure for the
next commit, and splitting this out makes it a lot easier to read what
is actually changed in the next commit.
This should fix a problem we've been having with errors downloading
the PhantomJS packages from their original hosting service.
Eventually we should move it to an S3 bucket.
cd2348e9ae broke installing Zulip in
production since it didn't correctly update the puppet configuration
to call the process_queue script using the new argument format.
This commit isn't ideal in that I'd prefer to not require updating
puppet in sync with the actual running code, but we don't have a great
mechanism for doing that.
Fixes#586.
Previously, we needed to update the installation instructions with the
current version of Zulip in production every time we did a release,
which was kinda a pain (and hadn't happened since 1.3.6).
Fixes#576.
[commit message details expanded by tabbott]
The original logic for incremental presence list updating from
668d0d9dfa incorrectly attempted to
insert the user 1 spot later than its proper index in the listing.
Now that we're doing presence updates in a performant fashion, we
don't need to throttle processing these events, and in fact the
throttling of these events created a correctness problem, since we're
now doing incremental updates rather than just rerendering everything
after each event.
The code in 668d0d9dfa for removing an
existing user from the user list to update the status didn't correctly
quote the email address of the user in its jquery selector.
While we already don't link to /terms anywhere on the site, they can still be
accessed if you navigate to /terms directly. Now, those routes will only be
exported on the Zulip.com service.
We should ideally provide a mechanism for deployments to specify their own
terms without modifying source code; in the interim, sites that have already
customised the provided Zulip.com terms can simply carry a patch reverting this
commit.
This change drops the memory used for Python processes run by Zulip in
development from about 1GB to 300MB on my laptop.
On the front of safety, http://pika.readthedocs.org/en/latest/faq.html
explains "Pika does not have any notion of threading in the code. If
you want to use Pika with threading, make sure you have a Pika
connection per thread, created in that thread. It is not safe to share
one Pika connection across threads.". Since this code only connects
to rabbitmq inside the individual threads, I believe this should be
safe.
Progress towards #32.
First user-fasing problem is that when user click to "Collapse" button
of message from narrowed list, buttons "Uncollapse" and "[More...]" does
not work. Second, is that when user collapse/uncollapse some message
from narrowed list, the collapsing/uncollapsing of the same message in
home list does not work in appropriate way.
In "popovers.js" there is the function that is called on click to the
buttons "Collapse" or "Un-collapse". It should show and hide body of a
message. If a message list is narrowed, it should show/hide message in
home list too. So, the first problem is that "toggle_row()" in this
function call methods "collapse(row)" or "uncollapse(row)" from
"condense.js" twice (for row and home_row) using condition
"if (message.collapsed)". When it happen the first time, the variable
"message.collapsed" is changed. That is why next call of "toggle_row()"
work incorrectly.
The second problem is that the function in "condense.js" that is
called on click to the button "[More...]" contains no code for
collapsing/uncollapsing message from home list. It just calls
"collapse(row)" or "uncollapse(row)" for row from narrowed list.
Now, functions "collapse(row)" and "uncollapse(row)" get row from
current list and change both messages (from current list and home
list). On-click functions call them just once for making all of needed
message changes. So, when user collapse or uncollapse message from
home or narrowed list it works correctly.
Fixes: #516
Apparently, our event queue garbage collection logic never actually
disconnected any existing handler objects.
We fix this by disconnecting the handler inside cleanup(), adding a
special check to avoid creating a pointless timeout object.
The new Tornado handler tracking logic properly handled requests that
threw an exception or followed the RespondAsynchronously code path,
but did not properly de-allocated the handler in the syncronous case.
An easy reproducer for this is to load a new Zulip browser window;
that will leak 2 handler objects for the 2 synchronous requests made
from Django to Tornado as part of initial state fetching.
This line appears to have been lost in rebasing from the original
implementation of 1396eb7022faec4c2d91553800a35781a96dd5bd; so the
previous fix actually only addressed the issue in a rare exception
case.
The recent Tornado memory leak fix
(1396eb7022) didn't use the correct
variable name for the current handler ID, causing this cleanup code to
fail in the event that a view raised an exception.
Replaced calls to ifilterfalse by list comprehensions because
ifilterfalse is not part of python 3. Also changed some lists to sets
for faster lookup.
Refer to #256.
This automatically loads settings, zerver.models.* and
zerver.lib.actions.* when you start `manage.py shell`, which should
save a bit of time basically every time someone uses it.
Fixes#275.
Previously, even though the Zulip digest emails were documented in the
settings, the cron job to run the script that actually sends the daily
digest emails wasn't included in the non-zulip.com part of the Zulip
production distribution. The overall consequence is that digest
emails didn't work for non-zulip.com users. This fixes that issue by
moving that cron job into the zulip manifests.
[commit message details expanded by tabbott]
In 2ea0daab19, handlers were moved to
being tracked via the handlers_by_id dict, but nothing cleared this
dict, resulting in every handler object being leaked. Since a Tornado
process uses a different handler object for every request, this
resulted in a significant memory leak. We fix this by clearing the
handlers_by_id dict in the two code paths that would result in a
Tornado handler being de-allocated: the exception codepath and the
handler disconnect codepath.
Fixes#463.
Saving the organization settings form in the administration did not
work due to a trivial form name mismatch caused by following
revisions: 472898c and 58aba59.
The only places we use the architecture were for finding the
tsearch_extras and phantomjs binaries; Luke Faraone kindly uploaded
both 32-bit and 64-bit binaries for tsearch_extras 0.1.3, so with a
bit of refactoring, we can now support 32-bit.
Fixes#505.
Add call to tools/generate-fixtures in tools/test-backend before
starting the tests. Previously, test-backend could fail if called
after tools/test-js-with-casper had failed.
Fixes#501.
Previously, we used shell quoting that would result in the shell variable not
being substituted. Instead, we use `"`s that will allow for variable
substitution.
We also explicitly include `ca-certificates`, as it is needed for the install
to complete. Usually this is brought in as a `Recommends` of `wget`, but some
systems may not automatically include such dependencies.
Fixes#470.
Whenever a user became active, this triggers an immediate presence
update event (to show that user as active). The implementation for
that event (running on the browsers of all other users in the realm)
would fully rerender the presence list, which can be an expensive
operation in a large realm, just to update the status for that one
user. This fixes that case to just remove the user from the list and
then re-insert it at the appropriate index.
[Commit message expanded with more details by Tim Abbott]
Add a function email_allowed_for_realm that checks whether a user with
given email is allowed to join a given realm (either because the email
has the right domain, or because the realm is open), and use it
whenever deciding whether to allow adding a user to a realm.
This commit is not intended to change any behavior, except in one case
where the Zulip realm's domain was not being converted to lowercase.
manage_args is set to a list of arguments a few lines later in the
function, making this initialization as the empty string useless and
confusing.
Discovered using mypy.
While I believe this actually produced correct output since users are
always subscribed to streams within their realm, this code was
definitely wrong.
Discovered using the mypy type-checking tool.
It's not clear whether this will end up being net negative in value in
the long term since it's kinda hard to understand the output, but in
the short term it should prevent regressions.
If the content wasn't rendered, both rendered_content and
rendered_content_version would be None. In addition to being
confusing, in Python 3, `None < 2` is an error and this code breaks.
At present, we only do a few simple checks on the client type inside
the event system, and this saves database/memcached queries.
Note that this preserves the structure of the marshalled name in
to_dict/from_dict as client_type to avoid an unnecessary migration.
Previously, client descriptors were referenced directly from the
handler object. Once we split the Tornado process into separate queue
and connection servers, these will no longer be in the same process,
so we need to reference them by ID instead.
This commit is somewhat ugly, but its purpose is to be early
preparation for splitting Tornado into a queue server and a frontend
server, and this code belongs, by and large, in the queue server
component.
89a2765553 didn't include the database
migration corresponding to the change, which means it didn't take full
effect when it was merged.
I noticed this because `manage.py makemigrations` would generate these
migrations; that suggests a good idea for a test to add.
This solves the problem reported in #331 with needing to specify
--provider=lxc to use the LXC provider in an Ubuntu Linux environment;
additionally, it adds the LXC option needed to run LXC on Ubuntu
15.10, but not on 14.04 where that option is unavailable and would
totally break LXC.
It's possible we should just eliminate this mechanism, but this fixes
a proximal problem where the multi-line get_subscribers endpoint
description was being handled wrong.
Previously these were hardcoded in zproject/settings.py to be accessed
on localhost.
[Modified by Tim Abbott to adjust comments and fix configure-rabbitmq]
Travis CI's model of installing every version of postgres on the test
VM and then shutting all the versions other than the one requested
down seems to not work very well with doing apt upgrades. It seems
the best way to resolve this is to just uninstall the versions we
don't need.
A common issue when doing a Zulip upgrade is trying to pass
upgrade-zulip a tarball path under /root, which doesn't work because
the Zulip user doesn't have permission to read the tarball. We
could fix this by just unpacking the tarballs as root, but it seemed
like a nicer approach would be to archive the release tarballs
somewhere readable by the Zulip user (/home/zulip/archives) and unpack
them from there.
Fixes#208.
The point of the lock is to prevent two deployments happening at the
same time and racing with each other, not to prevent doing any future
deployments after an error happens (which is what the current
implementation does in practice).
Addresses part of #208.
This link was broken when we hardened the access model for user file
uploads to not work cross-realm. The right solution is just to
include the image in the codebase so it's guaranteed to exist.
Fixes#205.
Previously:
* It wouldn't raise an exception if the stream didn't exist
* It didn't correctly handle being passed a stream name
that differed in case from the stream name in the database.
The previous implementation didn't work because HomepageForm rejected
the email as not having a domain. Additionally, the logic in
accounts_register didn't work with Google auth because that code path
doesn't pass through accounts_home. Since whether there's a unique
open realm for the server is effectively a configuration property, we
can fix the bug and make the logic clearer by moving it into the
"figure out the user's realm" function.
The browser registers for events via loading the home view, not this
interface, and this functionality is available via the API-format
register route anyway.
This removes from our cache a moderate amount of totally useless alert
word data corresponding to users who don't have any alert words.
Thanks to @dbiollo for the suggestion!
Just doing the database query is more readable, and has about the same
performance as before in the case where active user dicts for the
realm are in cache (and is substantially better in the rare case that
this isn't in the cache).
Thanks to @dbiollo for the perf investigation and suggestion!
This makes it possible to use DevAuthBackend when doing
performance/scalability testing on Zulip with many thousands of users.
It's unlikely that anyone testing this backend will find it valuable
to have more than 100 login buttons on the same page, and if they do,
they can always just change this limit.
Thanks to @dbiollo for the suggestion!
This fixes a performance issue looking up UserProfile objects for
realms with a large number of users in the case that a UserProfile
object is not in the cache.
Thanks to @dbiollo for the suggestion!
We ran into a bug with the Travis CI infrastructure where it postgres
9.1 is installed on the system, and so when we'd do an apt upgrade
with a new version of 9.1, the 9.1 daemon would end up getting started
and conflict with the 9.3 daemon we were trying to run.
860cf68716 introduced calls to
notifications.redraw_title() on narrow activation. This introduced a
bug when the Zulip desktop app reloads while narrowed --
new_message_count would still be set to undefined when
narrow.activate() is called as the page (re)loads, and thus we'd call
window.bridge.updateCount(undefined), resulting in a traceback.
We fix this by just initializing it to 0, rather than using the old
default value of undefined.
This allows full-screen mode when launching from a saved app link
(mobile browser -> save link to home screen). This works on Android,
too, despite the "apple-" prefix.
Django's `manage.py runserver` prints a relatively low-information log
line for every request of the form:
[14/Dec/2015 00:43:06]"GET /static/js/message_list.js HTTP/1.0" 200 21969
This is pretty spammy, especially given that we already have our own
middleware printing a more detailed version of the same log lines:
2015-12-14 00:43:06,935 INFO 127.0.0.1 GET 200 0ms /static/js/message_list.js (unauth via ?)
Since runserver doesn't have support controlling whether these log
lines are printed, we wrap it with a small bit of code that silences
the log lines for 200/304 requests (aka the uninteresting ones).
It's needed for the tornado server. Otherwise, you get errors like
2015-12-20 09:33:55,124 ERROR Internal Server Error: /api/v1/events
Traceback (most recent call last):
File "/home/zulip/deployments/2015-12-20-13-44-47/zerver/management/commands/runtornado.py", line 209, in get_response
response = middleware_method(request)
File "/usr/lib/python2.7/dist-packages/django/middleware/common.py", line 62, in process_request
host = request.get_host()
File "/usr/lib/python2.7/dist-packages/django/http/request.py", line 101, in get_host
raise DisallowedHost(msg)
DisallowedHost: Invalid HTTP_HOST header: 'localhost:9993'. You may need to add u'localhost' to ALLOWED_HOSTS.
Like the Stream Subject lists, Private messages are now shown
when the user clicks on the "Private message" link. User can drill in
to get more than 5 conversations. Selecting PMs from the user or group
PM lists on the right sidebar also opens the list & highlights the
selected conversation.
[Edited by tabbott@mit.edu to fix some small bugs.]
Several of these rules only apply to one of Python and Javascript, and
this simplifies the logic and should make our linter code more readable.
In the process, we add support for per-rule/file pair exclusions to
handle the tab exception for codehilite.py.
By default we are placed inside a virtualenv by the .bash_profile using
/usr/bin/python forces the provisioning script to run outside of this
virtualenv.
These routes previously didn't follow our standard convention of
sending arguments in JSON format, and so broke when we started
checking the argument format in
123d51e3aa.
Fixes#333.
Apparently, previously nginx was only compressing text/html content.
This should result in a substantial savings in network traffic -- some
quick testing I did found it cut the total data transferred for
loading a logged-in zulip.com instance from 3MB to 1.2MB.
`tools/test-js-with-node cover` needs istanbul to be installed in
order to work; we might as well install it by default rather than
having it be an extra step users need to deal with.
Of course, since this is only needed in the development environment,
this could suggest we want to fork/conditionalize package.json, but I
think for now it's reasonable to just install everything we use
somewhere -- the npm list is still pretty short and we have that issue
anyway with webpack-dev-server.
If running on Django 1.8, running these plugins would die with the below. A fix
for this is to run `django.setup()` before interacting with Django.
Refs:
https://docs.djangoproject.com/en/1.8/ref/applications/#troubleshooting
```
Traceback (most recent call last):
File "/usr/lib/nagios/plugins/check_send_receive_time", line 103, in <module>
sender = get_user_profile_by_email(settings.NAGIOS_SEND_BOT)
File "/home/zulip/deployments/current/zerver/lib/cache.py", line 113, in func_with_caching
val = func(*args, **kwargs)
File "/home/zulip/deployments/current/zerver/models.py", line 1073, in get_user_profile_by_email
return UserProfile.objects.select_related().get(email__iexact=email.strip())
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 328, in get
num = len(clone)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 144, in __len__
self._fetch_all()
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 977, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 238, in iterator
results = compiler.execute_sql()
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 829, in execute_sql
sql, params = self.as_sql()
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 378, in as_sql
extra_select, order_by, group_by = self.pre_sql_setup()
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 48, in pre_sql_setup
self.setup_query()
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 39, in setup_query
self.select, self.klass_info, self.annotation_col_map = self.get_select()
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 206, in get_select
related_klass_infos = self.get_related_selections(select)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 700, in get_related_selections
[f.name], opts, root_alias)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1471, in setup_joins
names, opts, allow_many, fail_on_missing=True)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1372, in names_to_path
if field.is_relation and not field.related_model:
File "/usr/lib/python2.7/dist-packages/django/utils/functional.py", line 60, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 110, in related_model
apps.check_models_ready()
File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 131, in check_models_ready
raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
```
Primarily this makes sure all the log files we generate are ignored.
A good follow-up project to this would be to move all the log files to
a fixed directory so that they're not creating a mess in the main
filespace.
The #! line processing interpreted the argument to pass to `env` as
"python2.7 -u", which obviously isn't a real program.
We fix this by setting the PYTHONUNBUFFERED environment variable
inside the program, which has the same effect.
Thanks to Dan Fedele for the bug report and suggested solution!
2015-11-16 08:46:48 -08:00
859 changed files with 54860 additions and 12791 deletions
Browsing this list can be a great way to find feature ideas to
implement that other Zulip users are excited about.
* [2016 roadmap milestone](http://zulip.readthedocs.io/en/latest/roadmap.html): The
projects that are [priorities for the Zulip project](https://zulip.readthedocs.io/en/latest/roadmap.html). These are great projects if you're looking to make an impact.
If you're excited about helping with an open issue, just post on the
conversation thread that you're working on it. You're encouraged to
@@ -123,11 +172,10 @@ looking at the new feature tutorial and coding style guidelines on
ReadTheDocs.
Feedback on how to make this development process more efficient, fun,
and friendly to new contributors is very welcome! Just shoot an email
and friendly to new contributors is very welcome! Just send an email
Contribution of a step-by-step guide for setting this up (and moving
this configuration to be available in the main `puppet/zulip/` tree)
would be very welcome!
### Monitoring Zulip
The complete Nagios configuration (sans secret keys) used to
monitor zulip.com is available under `puppet/zulip_internal` in the
Zulip Git repository (those files are not installed in the release
tarballs).
The Nagios plugins used by that configuration are installed
automatically by the Zulip installation process in subdirectories
under `/usr/lib/nagios/plugins/`. The following is a summary of the
various Nagios plugins included with Zulip and what they check:
Application server and queue worker monitoring:
* check_send_receive_time (sends a test message through the system
between two bot users to check that end-to-end message sending works)
* check_rabbitmq_consumers and check_rabbitmq_queues (checks for
rabbitmq being down or the queue workers being behind)
* check_queue_worker_errors (checks for errors reported by the queue workers)
* check_worker_memory (monitors for memory leaks in queue workers)
* check_email_deliverer_backlog and check_email_deliverer_process
(monitors for whether outgoing emails are being sent)
Database monitoring:
* check_postgres_replication_lag (checks streaming replication is up
to date).
* check_postgres (checks the health of the postgres database)
* check_postgres_backup (checks backups are up to date; see above)
* check_fts_update_log (monitors for whether full-text search updates
are being processed)
Standard server monitoring:
* check_website_response.sh (standard HTTP check)
* check_debian_packages (checks apt repository is up to date)
If you're using these plugins, bug reports and pull requests to make
it easier to monitor Zulip and maintain it in production are
encouraged!
### Scalability of Zulip
This section attempts to address the considerations involved with
running Zulip with a large team (>1000 users).
* We recommend using a [remote postgres
database](#postgres-database-details) for isolation, though it is
not required. In the following, we discuss a relatively simple
configuration with two types of servers: application servers
(running Django, Tornado, RabbitMQ, Redis, Memcached, etc.) and
database servers.
* You can scale to a pretty large installation (O(~1000) concurrently
active users using it to chat all day) with just a single reasonably
large application server (e.g. AWS c3.2xlarge with 8 cores and 16GB
of RAM) sitting mostly idle (<10% CPU used and only 4GB of the 16GB
RAM actively in use). You can probably get away with half that
(e.g. c3.xlarge), but ~8GB of RAM is highly recommended at scale.
Beyond a 1000 active users, you will eventually want to increase the
memory cap in `memcached.conf` from the default 512MB to avoid high
rates of memcached misses.
* For the database server, we highly recommend SSD disks, and RAM is
the primary resource limitation. We have not aggressively tested
for the minimum resources required, but 8 cores with 30GB of RAM
(e.g. AWS's m3.2xlarge) should suffice; you may be able to get away
with less especially on the CPU side. The database load per user is
pretty optimized as long as `memcached` is working correctly. This
has not been tested, but from extrapolating the load profile, it
should be possible to scale a Zulip installation to 10,000s of
active users using a single large database server without doing
anything complicated like sharding the database.
* For reasonably high availability, it's easy to run a hot spare
application server and a hot spare database (using Postgres
streaming replication; see the section on configuring this). Be
sure to check out the section on backups if you're hoping to run a
spare application server; in particular you probably want to use the
S3 backend for storing user-uploaded files and avatars and will want
to make sure secrets are available on the hot spare.
* Zulip does not support dividing traffic for a given Zulip realm
between multiple application servers. There are two issues: you
need to share the memcached/redis/rabbitmq instance (these should
can be moved to a network service shared by multiple servers with a
bit of configuration) and the Tornado event system for pushing to
browsers currently has no mechanism for multiple frontend servers
(or event processes) talking to each other. One can probably get a
factor of 10 in a single server's scalability by [supporting
multiple tornado processes on a single
server](https://github.com/zulip/zulip/issues/372), which is also
likely the first part of any project to support exchanging events
amongst multiple servers.
Questions, concerns, and bug reports about this area of Zulip are very
welcome! This is an area we are hoping to improve.
### Security Model
This section attempts to document the Zulip security model. Since
this is new documentation, it likely does not cover every issue; if
there are details you're curious about, please feel free to ask
questions on the Zulip development mailing list (or if you think
you've found a security bug, please report it to support@zulip.com so
we can do a responsible security announcement).
#### Secure your Zulip server like your email server
* It's reasonable to think about security for a Zulip server like you
do security for a team email server -- only trusted administrators
within an organization should have shell access to the server.
In particular, anyone with root access to a Zulip application server
or Zulip database server, or with access to the `zulip` user on a
Zulip application server, has complete control over the Zulip
installation and all of its data (so they can read messages, modify
history, etc.). It would be difficult or impossible to avoid this,
because the server needs access to the data to support features
expected of a group chat system like the ability to search the
entire message history, and thus someone with control over the
server has access to that data as well.
#### Encryption and Authentication
* Traffic between clients (web, desktop and mobile) and the Zulip is
encrypted using HTTPS. By default, all Zulip services talk to each
other either via a localhost connection or using an encrypted SSL
connection.
* The preferred way to login to Zulip is using an SSO solution like
Google Auth, LDAP, or similar. Zulip stores user passwords using
the standard PBKDF2 algorithm. Password strength is checked and
weak passwords are visually discouraged using the zxcvbn library,
but Zulip does not by default have strong requirements on user
password strength. Modify `static/js/common.js` to adjust the
password strength requirements (Patches welcome to make controlled
by an easy setting!).
* Zulip requires CSRF tokens in all interactions with the web API to
prevent CSRF attacks.
#### Messages and History
* Zulip message content is rendering using a specialized Markdown
parser which escapes content to protect against cross-site scripting
attacks.
* Zulip supports both public streams and private ("invite-only")
streams. Any Zulip user can join any public stream in the realm
(and can view the complete message of any public stream history
without joining the stream).
* Users who are not members of a private stream cannot read messages
on the stream, send messages to the stream, or join the stream, even
if they are a Zulip administrator. However, any member of a private
stream can invite other users to the stream. When a new user joins
a private stream, they can see future messages sent to the stream,
but they do not receive access to the stream's message history.
* Zulip supports editing the content or topics of messages that have
already been sent (and even updating the topic of messages sent by
other users when editing the topic of the overall thread).
While edited messages are synced immediately to open browser
windows, editing messages is not a safe way to redact secret content
(e.g. a password) unintentionally shared via Zulip, because other
users may have seen and saved the content of the original message
(for example, they could have taken a screenshot immediately after
you sent the message, or have an API tool recording all messages
they receive).
Zulip stores and sends to clients the content of every historical
version of a message, so that future versions of Zulip could support
displaying the diffs between previous versions.
#### Users and Bots
* There are three types of users in a Zulip realm: Administrators,
normal users, and bots. Administrators have the ability to
deactivate and reactivate other human and bot users, delete streams,
add/remove administrator privileges, as well as change configuration
for the overall realm (e.g. whether an invitation is required to
join the realm). Being a Zulip administrator does not provide the
ability to interact with other users' private messages or the
messages sent to private streams to which the administrator is not
subscribed. However, a Zulip administrator subscribed to a stream
can toggle whether that stream is public or private. Also, Zulip
realm administrators have administrative access to the API keys of
all bots in the realm, so a Zulip administrator may be able to
access messages sent to private streams that have bots subscribed,
by using the bot's credentials.
In the future, Zulip's security model may change to allow realm
administrators to access private messages (e.g. to support auditing
functionality).
* Every Zulip user has an API key, available on the settings page.
This API key can be used to do essentially everything the user can
do; for that reason, users should keep their API key safe. Users
can rotate their own API key if it is accidentally compromised.
* To properly remove a user's access to a Zulip team, it does not
suffice to change their password or deactivate their account in the
SSO system, since neither of those prevents authenticating with the
user's API key or those of bots the user has created. Instead, you
should deactivate the user's account in the Zulip administration
interface (/#administration); this will automatically also
deactivate any bots the user had created.
* The Zulip mobile apps authenticate to the server by sending the
user's password and retrieving the user's API key; the apps then use
the API key to authenticate all future interactions with the site.
Thus, if a user's phone is lost, in addition to changing passwords,
you should rotate the user's Zulip API key.
* Zulip bots are used for integrations. A Zulip bot can do everything
a normal user in the realm can do including reading other, with a
few exceptions (e.g. a bot cannot login to the web application or
create other bots). In particular, with the API key for a Zulip
bot, one can read any message sent to a public stream in that bot's
realm. A likely future feature for Zulip is [limited bots that can
only send messages](https://github.com/zulip/zulip/issues/373).
* Certain Zulip bots can be marked as "API super users"; these special
bots have the ability to send messages that appear to have been sent
by another user (an important feature for implementing integrations
like the Jabber, IRC, and Zephyr mirrors).
#### User-uploaded content
* Zulip supports user-uploaded files; ideally they should be hosted
from a separate domain from the main Zulip server to protect against
various same-domain attacks (e.g. zulip-user-content.example.com)
using the S3 integration.
The URLs of user-uploaded files are secret; if you are using the
"local file upload" integration, anyone with the URL of an uploaded
file can access the file. This means the local uploads integration
is vulnerable to a subtle attack where if a user clicks on a link in
a secret .PDF or .HTML file that had been uploaded to Zulip, access
to the file might be leaked to the other server via the Referrer
header (see https://github.com/zulip/zulip/issues/320).
The Zulip S3 file upload integration is relatively safe against that
attack, because the URLs of files presented to users don't host the
content. Instead, the S3 integration checks the user has a valid
Zulip session in the relevant realm, and if so then redirects the
browser to a one-time S3 URL that expires a short time later.
Keeping the URL secret is still important to avoid other users in
the Zulip realm from being able to access the file.
* Zulip supports using the Camo image proxy to proxy content like
inline image previews that can be inserted into the Zulip message
feed by other users over HTTPS.
* By default, Zulip will provide image previews inline in the body of
messages when a message contains a link to an image. You can
control this using the `INLINE_IMAGE_PREVIEW` setting.
#### Final notes and security response
If you find some aspect of Zulip that seems inconsistent with this
security model, please report it to support@zulip.com so that we can
investigate and coordinate an appropriate security release if needed.
Zulip security announcements will be sent to
zulip-announce@googlegroups.com, so you should subscribe if you are
running Zulip in production.
Remote User SSO Authentication
==============================
@@ -508,3 +891,138 @@ understanding what's going on as you try to debug:
Again, most issues with this setup tend to be subtle issues with the
hostname/DNS side of the configuration. Suggestions for how to
improve this SSO setup documentation are very welcome!
Postgres database details
=========================
#### Remote Postgres database
This is a bit annoying to setup, but you can configure Zulip to use a
dedicated postgres server by setting the `REMOTE_POSTGRES_HOST`
variable in /etc/zulip/settings.py, and configuring Postgres
certificate authentication (see
http://www.postgresql.org/docs/9.1/static/ssl-tcp.html and
http://www.postgresql.org/docs/9.1/static/libpq-ssl.html for
documentation on how to set this up and deploy the certificates) to
make the DATABASES configuration in `zproject/settings.py` work (or
override that configuration).
If you want to use a remote Postgresql database, you should configure
the information about the connection with the server. You need a user
called "zulip" in your database server. You can configure these
options in /etc/zulip/settings.py:
* REMOTE_POSTGRES_HOST: Name or IP address of the remote host
* REMOTE_POSTGRES_SSLMODE: SSL Mode used to connect to the server, different options you can use are:
* disable: I don't care about security, and I don't want to pay the overhead of encryption.
* allow: I don't care about security, but I will pay the overhead of encryption if the server insists on it.
* prefer: I don't care about encryption, but I wish to pay the overhead of encryption if the server supports it.
* require: I want my data to be encrypted, and I accept the overhead. I trust that the network will make sure I always connect to the server I want.
* verify-ca: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server that I trust.
* verify-full: I want my data encrypted, and I accept the overhead. I want to be sure that I connect to a server I trust, and that it's the one I specify.
Then you should specify the password of the user zulip for the database in /etc/zulip/zulip-secrets.conf:
```
postgres_password = xxxx
```
Finally, you can stop your database on the Zulip server via:
```
sudo service postgresql stop
sudo update-rc.d postgresql disable
```
In future versions of this feature, we'd like to implement and
document how to the remote postgres database server itself
automatically by using the Zulip install script with a different set
of puppet manifests than the all-in-one feature; if you're interested
in working on this, post to the Zulip development mailing list and we
can give you some tips.
#### Debugging postgres database issues
When debugging postgres issues, in addition to the standard `pg_top`
tool, often it can be useful to use this query:
```
SELECT procpid,waiting,query_start,current_query FROM pg_stat_activity ORDER BY procpid;
```
which shows the currently running backends and their activity. This is
similar to the pg_top output, with the added advantage of showing the
complete query, which can be valuable in debugging.
To stop a runaway query, you can run `SELECT pg_cancel_backend(pid
int)` or `SELECT pg_terminate_backend(pid int)` as the 'postgres'
user. The former cancels the backend's current query and the latter
terminates the backend process. They are implemented by sending SIGINT
and SIGTERM to the processes, respectively. We recommend against
sending a Postgres process SIGKILL. Doing so will cause the database
to kill all current connections, roll back any pending transactions,
and enter recovery mode.
#### Stopping the Zulip postgres database
To start or stop postgres manually, use the pg_ctlcluster command:
```
pg_ctlcluster 9.1 [--force] main {start|stop|restart|reload}
```
By default, using stop uses "smart" mode, which waits for all clients
to disconnect before shutting down the database. This can take
prohibitively long. If you use the --force option with stop,
pg_ctlcluster will try to use the "fast" mode for shutting
down. "Fast" mode is described by the manpage thusly:
With the --force option the "fast" mode is used which rolls back all
active transactions, disconnects clients immediately and thus shuts
down cleanly. If that does not work, shutdown is attempted again in
"immediate" mode, which can leave the cluster in an inconsistent state
and thus will lead to a recovery run at the next start. If this still
does not help, the postmaster process is killed. Exits with 0 on
success, with 2 if the server is not running, and with 1 on other
failure conditions. This mode should only be used when the machine is
about to be shut down.
Many database parameters can be adjusted while the database is
running. Just modify /etc/postgresql/9.1/main/postgresql.conf and
issue a reload. The logs will note the change.
#### Debugging issues starting postgres
pg_ctlcluster often doesn't give you any information on why the
database failed to start. It may tell you to check the logs, but you
won't find any information there. pg_ctlcluster runs the following
command underneath when it actually goes to start Postgres:
|``zerver/management/commands/`` | Management commands one might run at a production deployment site (e.g. scripts to change a value or deactivate a user properly) |
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
