The main purpose of the "var" convention is to make it easy to write stuff
inside of our git repo when running a dev instance, and then "var" gets
excluded from checkins. For production, that's not as much of a concern.
For upgrades we don't want to be changing the directory around and confusing
matters, especially with the extra moving part of nginx configs (which have
their own issues in terms of being overwritten by accident when admins go to
S3).
Between releases 1.3.13 and 1.4.0, local_settings.py was renamed to
prod_settings.py. The upgrade scripts were adjusted to reflect this name
change. But because the first part of the upgrade script is run with the
currently installed version's code, the symlink to /etc/zulip/settings.py is
created with the old name. This was causing upgrade-zulip-stage-2 to fail.
Now upgrade-zulip-stage-2 creates the symlink at zproject/prod_settings.py
if it doesn't already exist.
Fixes#1731.
- Adds heading and reorganizes sections for improved clarity. Sections
now ordered: Create user, grant admin access, default realm settings,
changing realm settings, other manage.py commands, if you can't login.
- Adds screen shots of Zulip web interface.
Apparently, we had incorrectly concluded that our highlight_string
search result highlighting offsets coming from tsearch_extras were
measured in bytes, whereas in fact it is measured in characters.
This was left as a placeholder when we moved the production
documentation to ReadTheDocs, but is no longer required since we have
eliminated links to the old path.
- Moves Let's Encrypt option for ssl certs above self-signed option.
Updates Let's Excrypt instructions to user Certbot (new recommended
method). Fixes#1426.
- Adds and updates headings for clarity.
- Adds new step 6 about configuring authentication (from
prod-auth-first-login.html).
- Adds emphasis on using `su zulip`, `sudo -u zulip -i`, and `sudo -i`
where appropriate. Fixes#1356.
Fixes#1245.
- Reformats bullets as regular paragraphs for improved readability.
- Adds headings to subsections to make them easier to read and linkable.
- Clarifies minor points throughout.
- Add headings for each of the six major sections: integrations, streams
and topics, notification settings, mobile and desktop apps, other
great features, and enjoy your Zulip installation.
- Add links to each of these sections to top of page, with first and
most important four formatted as a list.
- Adds headings for 'supervisorctl' and 'troubleshooting'.
- Rearranges to include most important info at top: that Supervisor is
used, a reminder to review components architecture, log location, link
to troubleshooting services section, and a request to report any
issues not already documented on this page.
The previous default configuration resulted in delivery problems if
the Zulip server was authorized in the SPF records for the domains of
all users on the Zulip server.
Also changes all links to /integrations to be relative rather than
absolute, so that users will primarily access the /integrations page
for their subdomain.
This has the nice side effect of getting rid of the now-unnecessary
ADMIN_DOMAIN from this codepath -- we really just want whichever
realm ERROR_BOT is in.
Due to the border-top being tied to the .message-header-contents
element rather than the recipient_row in private message groups, the
floating container would be 1px more vertical when representing private
message groups than with public message groups.
Extract dispatch_normal_event() from the inner
function dispatch_event() in get_events_success()
in server_events.js. Also, alphabetize the cases in
the switch statement. This starts to address #1541,
but see the discussion on the ticket for how we can
continue to clean up our event dispatching.
This commit also starts to build out the infrastructure for
helping Zulip contributors to more easily author bots in
a way that sets up for running some bots on the server itself.
Since this delayed sending feature is the only thing
settings.MANDRILL_API_KEY is used for, it seems reasonable for that to
be the gate as to whether we actually use Mandrill.
This is a convenient tool to have around.
We require an unusual argument value of "YES" to send to everyone on
the server, since that's something one should do with a great deal of
care.
These error messages are pretty spammy because most servers on the
public Internet receive some amount of HTTP(S) scanning traffic.
We still log them, just don't email the administrators.
This sends an event when a new avatar is uploaded that refreshes the
avatar for all browser clients without the need to reload the browser.
Fixes: #1359.
!avatar, !modal_link, !gravatar, etc. were incorrectly being processed
before the escape character for code blocks.
While we're at it, we add tests for these special syntaxes.
This starts to address 1533. I still think the </p> tags
should be on their own line lined up with the start tag,
so the linter won't let through the specific example
shown in the ticket.
Arguably we should figure out a way to make this fetch the support
email from the server (or something), but in the short term it seems
worth making the email address at least correct.
This fixes a nasty bug where exporting messages sent by a single user
might only contain some of the messages in the event that the
unspecified sort order by the database didn't happen to be sorted by
message ID.
This file, installed back when we had node_modules checked into git,
was leaked when we deleted node_modules/ from git and added it to
.gitignore. This caused occasional weird errors for anyone who tried
to `rm -rf node_modules` as part of reprovisioning.
This fixes a confusing bug that we ran into where the output from
certain child processes wouldn't appear when running `lint-all` via
`ssh` (without a tty) into the Vagrant environment.
This commit only addresses tables that currently derive from
user_profile_config in get_realm_config:
zerver_userpresence
zerver_useractivity
zerver_useractivityinterval
zerver_subscription
zerver_recipient
zerver_stream
zerver_huddle
It also introduces an entry in realm.json for a virtual
table called "zerver_userprofile_mirrordummy" for dummy users,
which include prior dummy users and users excluded from the call
to do_export_realm().
Note that this feature is not yet exposed in the management command.
When the feedback module is hidden the #userlist_header border would
brush against the navbar. Check if the feedback header isn’t there and
remove the border top.
The policy here is essentially:
* Pages on the central server (documentation, etc.) are server_uri.
* Pages associated with a user's realm server (anything logged-in, plus
things like their login page) are realm_uri.
Changes relative path to an absolute path (that doesn't contain the
subdomain) for various links to
/create_realm, /api, /apps, /integrations, /hello, /terms, and the logged
out / (the Zulip in the upper left corner of portico)
I typically left links internal to the relevant pages (e.g. a link from
integrations.html to a subpage of integrations/) as relative links, and
changed external links from within the app to the absolute path (e.g. the
link to integrations from the gear menu).
Helps in case the development environment is not using https.
We get the value from zerver/context_processors.py (which is
ultimately reading it from settings)
This adds a few new helpful context variables that we can use to
compute URLs in all of our templates:
* external_uri_scheme: http(s)://
* server_uri: The base URL for the server's canonical name
* realm_uri: The base URL for the user's realm
This is preparatory work for making realm_uri != server_uri when we
add support for subdomains.
The message cache filling script actually used both database and
memcached queries as part of filling the cache (all used to compute
the needed display_recipient values). We would ideally fix this by
using bulk operations to fill the display_recipient cache, but until
we do so, this cache filler is counterproductive.
I believe this disabling fixes an issue where memcached would get
overloaded and stop handling requests during a server restart on busy
servers.
Now attachment data gets written to its own json file. We are
splitting this out so that will be easier for us to cross-check
attachments against messages without holding up writing a lot
of the other realm data. (message cross-checking is coming soon)
This commit doesn't change any behavior; it just moves fetching
attachments out of the Config scheme and into its own method.
This prepares us to start writing attachment data to its own
file and cross-checking against message ids (coming soon).
We now just have a single configuration get_realm_config() that
handles most of the top-down realm export tables. (It basically
does everything not related to messages or uploads/avatars.)
Unifying the configs allows us to be more strict in our
configuration about checking for anomalies. In the future
we may need to loosen up some of those restrictions again,
but for now we are picky and paranoid.
Fetch stream data only for stream recipients, instead of
getting streams via realm_id.
(This change is kind of moot for now, since our stream recipients
include all possible stream recipients in the realm, but this
sets us up for when we start restricting users that we export
within the realm.)
The previous model for these Nagios checks was kinda crazy -- every
minute, we'd run a full `rabbitmctl list_consumers` for each of the
dozen+ consumers that we have, and then do the exact same parsing
logic for each to determine whether the target queue has a running
consumer to write out a state file.
Because `rabbitmctl list_consumers` takes a small amount of resources,
on systems where CPU is very limited (e.g. t2 style AWS instances),
this minor CPU wastage could be problematic.
Now we just do that `rabbitmqctl list_consumers` once per minute, and
output all the state files from a single command.
Further TODO items on this front include removing the hardcoded list
of queues.
Previously, missed message emails with multiple senders would
incorrectly have a "," outside the quoted sender name part of the from
address string, resulting in confusing email output.
This commit introduces the ability to do custom fetches
and to essentially use temp tables for intermediate results.
(The temp table stuff deals with recipients/subscriptions
having three different flavors--user, stream, and huddle.)
Because rabbitmq doesn't support changing the nodename of a running
rabbitmq node, Zulip installations suffered a plague of issues where
e.g. a Zulip server would reboot, the hostname would change, and
suddenly the local rabbitmq instance being used by Zulip would stop
working.
We address this problem by using, by default, a fixed rabbitmq
nodename, but providing server administrators the option to set the
rabbitmq nodename used by Zulip however they choose.
To upgrade an existing server to use this new configuration, one will
need to add something like the following to /etc/zulip/zulip.conf:
[rabbitmq]
nodename = zulip@localhost
However, I don't believe we have the puppet code in place to make this
work correctly at initial installation without rabbitmq-server being
already installed (but off), as we can easily setup in Travis CI but I
haven't been willing to do for the installer. So for now, this just
fixes our Travis CI problems.
Fixes: #1579.
Travis CI seems to have changed the way the snakeoil SSL certs are
generated in their infrastructure, so we need to update our expected
"success" HTTP headers accordingly.
Most directly useful for the migration to zulipchat.com.
Creates a new field in UserProfile to store the tos_version, as well as two
new settings TOS_VERSION and FIRST_TIME_TOS_TEMPLATE. We check for a version
mismatch between what the user has signed and the current
settings.TOS_VERSION whenever the user hits the home page, and redirect them
if needed.
Note that accounts_accept_terms.html and
zerver.views.accounts_accept_terms were unused before this commit
(they date from c327446537)
Previously, we used a fixed memcached memory allocation of 512MB,
regardless of the size of the server. While that is a good allocation
for a server with 4GB of RAM, for servers with less, we should
decrease the allocation, and for a large server with much more RAM, we
should increase it. We still support the user overriding the
configuration setting, but this produces more sensible defaults.
Zulip had only patches the redis configuration in one small way, which
resulted in unnecessary portability issues for using Redis on
different versions of Linux. We replace this with just a adding an
include mechanism to the redis config.
While we're at it, we configure this to take advantage of the
new REDIS_PASSWORD secret to automatically configure redis passwords.
The function to create the message partial files has been
renamed to export_partial_message_files(). It now gets its own
list of user profile ids and recipient ids from the response,
so that we can de-clutter do_export_realm().
The name avatar_bucket was confusing for a boolean, and
in some places it was used for non-S3 paths.
I considered the more concise 'is_avatar', but that
was still confusing when you are processing multiple
files, because you think it's a calculated property
on one file instead of an overall codepath switch.
I also considered splitting up some functions, but
there is a lot of common logic between handling
file uploads and avatars that's not trivial to extract
into helpers, especially on the S3 side.
I did some minor moving around of code that made us have
one fewer function without any additional conditional
logic. The names are more explicit about saying
"from_local" and "from_s3". Also, there is less clutter
now in do_export_realm(), which is evolving into more of
a dispatcher and less of a worker.
When we refactored zilencer to use a single urls.py file in
bf50dd7771, we accidentally lost the
prefix on the API urls.
This broke sending error report emails if zilencer was enabled.
It seems that we no longer get the message, 'zerver/lib/actions.py
modified; restarting server', but the server reloads successfully
nonetheless.
Fixes: #1341.
This is pretty minor cleanup, but it makes it a little more
explicit what we're writing to the shard file, and it allows
us to use a more specific mypy type when calling
floatify_datetime_fields.
We no longer have an in-process code path to export
UserMessage rows. We want to only maintain the
subprocess code, which we'll always use in production,
and which will work fine in dev.
Adds a new field default language in the zerver_realm model.
This realm level default language will be used as default language
for newly created users. Realm level default language can be
changed from the administration page.
Fixes#1372.
I also fixed some small things like removing unnecessary return
statements, and adding a TODO.
In some cases I explicitly cast stuff at run-time to set() or
str() to appease mypy, as well as make it clear to somebody
reading the code that the callee might not respect ordering
or tolerate unicode.
When in debug mode, previously an explicit `blueslip.error()` call
would not display the original stack trace of the error, making
debugging difficult.
This isn't perfect, in that it seems to display some tracebacks
multiple times, but at least the trace is available.
Adding the container attribute to the color picker options parameter
modifies the behavior of the spectrum function to allow the color
picker to be attached to the #subscriptions_table element rather than
the default document.body element.
This allows for the color picker to scroll with the subscriptions page,
fixing the bug where the color picker would not scroll up/down the page.
Fixes#1293.
This allows you to select the container that the color picker should
append to and therefore be absolutely positioned in accordance with.
This still defaults to document body, which was the previously default
container to append the color picker to.
The color picker original z-index was set to (2^31)-1 which overrode
everything on the site. This behavior is unwanted as the color picker
should not override the z-index of the navbar.
In changing the z-index to 100 the navbar remains above the color
picker.
The find-add-class tool, when in lint mode, verifies that we can
understand all calls to addClass from our JS code.
When in non-lint mode, i.e. verbose mode, the tool prints out a
list of tuples of (fn, class) that we can use as we wish in other
tools.
We were ignoring singleton tags like "input" tags in
html-grep. This was an artifact of our tokenizer originally
being built to check indentation of templates, for which
singleton tags had been a distraction. This fix actually cleans up
the template checking logic as well, since it can now rely
on the tokenizer to classify special tags and singleton tags.
The tokenizer is more complete and more specific.
The previous export tool would only work properly for small realms,
and was missing a number of important features:
* Export of avatars and uploads from S3
* Export of presence data, activity data, etc.
* Faithful export/import of timestamps
* Parallel export of messages
* Not OOM killing for large realms
The new tool runs as a pair of documented management commands, and
solves all of those problems.
Also we add a new management command for exporting the data of an
individual user.
Often, users will copy email addresses with a name (rather than pure
email addresses) into the Zulip "invite users" UI. Previously, that
would throw an error.
This change also adds a get_invitee_emails_set function for parsing
emails content and a test suite for this new feature.
Fixes: #1419.
This reverts commit 3f95e567c1.
Apparently `apt-add-repository` fails periodically in CI. I suspect
this is some sort of silly networking problem, but given that all
we're saving is a few lines of code, the old version was better if
this fails basically ever.
Create `media.css` using media queries that had been at the bottom
of `zulip.css`, then update miscellaneous setttings/docs files.
I also add `.screen-medium-show` and `.screen-narrow-show` to
`media.css`, as they seem to be an important part of our
responsive design.
Fixes#1532.
Define Integration and WebhookIntegration classes.
Change webhook part of integration's guide.
Replace hardcoded webhook urls to generating
based on WEBHOOKS list.
Now, `tools/test-all` calls a new program called `tools/tests-tools`
that runs unit tests in `test_css_parser.py` and 'test_template_parser.py`.
This puts 100% line coverage on tools/lib/css_parser.py.
This puts about 50% line coverage on tools/lib/template_parser.py.
* Fixes passing a string argument rather than an actual Python
argument.
* Switches to hardcoding the database to connect to rather than the
user, so this check can be run as an arbitrary user.
Both key and secret settings of team and organization default to
SOCIAL_AUTH_GITHUB_KEY and SOCIAL_AUTH_GITHUB_SECRET respectively.
SOCIAL_AUTH_GITHUB_TEAM_ID and SOCIAL_AUTH_GITHUB_ORG_NAME default
to `None`.
`tools/lint-all` now calls the new `tools/check-css`
The css_parser library parsers CSS into a data structure
that remembers line numbers and columns of semantically
meaningful tokens and adjoining white space/tokens. It
is intended to be used for various linting tasks.
The file `tools/check-css` runs a few files through the
parser and makes sure they round trip. This has some value
right away, as files that fail to parse will cause an
exception to be thrown and thus alert developers to syntax
errors. We expect to grow this into more advanced linting
tasks eventually.
The old behavior was to raise an exception, but Django was catching
the exception and doing unexpected things. For instance, in the
manage.py shell, printing out a ModelReprMixin object (with
__unicode__ not implemented) would result in nothing being printed,
rather than it raising a error or otherwise alerting the programmer as
to what was going on.
This fixes a regression where missed message emails would not be sent
at all in the event that EMAIL_GATEWAY_PATTERN was unset.
The overall experience still isn't great, but it's better than crashing.
Fixes: #1411
[commit message expanded by tabbott]
These tests would work as part of the whole suite, but
not standalone, because they were getting objects out
of node's require cache that a previous test had cleaned up.
Now they should work standalone as well, and the tests
are more explicit about their dependencies.
Some node tests used to pass as long as prior tests ran,
but then they would fail if you ran them standalone. Now
we are more aggressive about cleaning up node's require
cache after each individual test runs.
This introduces a very minor different in behavior if you specify
an invalid filename as a command line argument. We now show
warnings for those *before* running the rest of the tests.
`npm install` fails nondeterministically occasionally, and this makes
such failures likely to be automatically resolved in most cases by
simple retrying.
In templates.js we want to enforce outputting just
one output file per template, and we also keep the source
alphabetical by template name. This isn't a permanent
decision, but it makes organizing the ouput a little
easier for now.
This exists primarily in order to allow us to mock settings.DEBUG for
the purposes of rate limiting, without actually mocking
settings.DEBUG, which I suspect Django never intended one to do, and
thus caused some very strange test failures (see
https://github.com/zulip/zulip/pull/776 for details).
This will lead to minor differences in the warnings that
people see when they run tests that are slow. We call out
the slowness a little more clearly from a visual standpoint,
and we simplify the calculation of the slowness threshold.
We still allow more time for tests with the `@slow` decorator
to run, but we don't use their expected_run_time.
When running ./tools/test-backend, the script to generate
fixtures, ./tools/setup/generate-fixtures, looks for a file
called migration-status to determine whether it can short
circuit doing database migrations. This file got moved as
part of the effort to put files in "var," but the existence
check was still looking for that file in its old location.
While logging through GitHub if the realm of the user doesn't
exist then we are redirected to registration page but the action
points to the complete url of the GitHub oAuth overflow.
All other zulip management command names have underscores, so
rename email-mirror to email_mirror.
This will also make it possible to import this module, which will
help in writing tests for it.
We install service-identity to prevent twisted from giving spammy
warnings. However, we don't need the security features provided
by service-identity. So we can remove most of its cryptographic
dependencies.
Since twisted is only used for running tools/run-dev.py, we only
need it in development. Also twisted is not python 3 compatible.
So it should be shifter to requirements/py2_dev.txt.
This was in AdminCreateUserTest.test_create_user_backend().
For end to end tests we are logged in, and we need to verify
that our decorators add UserProfile to the parameters of
the view on our behalf, so that we don't get false positives.
In an upcoming commit, we will want to be able to serialize
the parameters for client_put to produce url coverage reports,
so that is another reason not to pass in the UserProfile
object. (That was how this was discovered.)
Our flush functions update user profile cache entries which can cause
confusing race conditions (see e.g. #1257). To resolve this, we move
all the user_profile flush functions to delete the entry instead of
updating it -- it will then be fetched as part of the next request
that needs to access the user object.
There are still races here, and there is perhaps an argument that a
better fix for this would be to re-fetch the object and then put it
into the cache, but this resolves the main cache correctness problem
we had with the previous implementation.
Fixes: #1322.
Run '/puppet/zulip/files/nagios_plugins/zulip_app_frontend/check_send_receive_time'
script as 'zulip' user so that the connection to the database can be
made correctly.
This reverts commit be93b6ea28.
Unfortunately, the newer jquery comes with a huge performance
regression affecting the hotkeys code, which has the effect of making
typing super slow.
Fixes: #1449.
This makes us more consistent, since we have other wrappers
like client_patch, client_put, and client_delete.
Wrapping also will facilitate instrumentation of our posting code.
Upgrade alabaster and sphinx to latest version.
commonmark can't be updated because recommonmark uses an old
version of commonmark which is incompatible with latest version.
The MitUser model caused a constant series of little problems for
users with mit.edu email addresses trying to sign up for different
Zulip servers.
The new implementation just uses conditionals on the realm object when
selecting the confirmation template to use.
This function is only called in cases where user_profile isn't None,
and the code reads better if we just check that first rather than
checking it on every line that accesses user_profile.
In python 3, subprocess uses bytes for input and output if
universal_newlines=False (the default). It uses str for input and
output if universal_newlines=True.
Since we mostly deal with strings, add universal_newlines=True to
subprocess.check_output.
When email mirroring is done via polling, the IMAP account's
password should be stored in zulip-secrets.conf in
email_gateway_password, not in email_gateway_login.
This new helper combines two old helpers, one of which was misnamed
and the other of which was always called after the first, so it
made sense to just combine the helpers.
Fixes: #1386
Initialize Record by using __init__ instead of setting attributes
in validate. This is needed because mypy complains when we set
new attributes outside __init__.
This commit adds these two tests:
test_use_first_unread_anchor_with_some_unread_messages
test_use_first_unread_anchor_with_no_unread_messages
The new tests add coverage to the conditional logic in
get_old_messages_backend() that looks at first_unread_result
when use_first_unread_anchor is set to True.
The test is now called test_use_first_unread_anchor_with_muted_topics().
Before this commit, the test exercised setting
use_first_unread_anchor to True, but it didn't inspect the
most relevant query affected by the flag. Now it does.
This test is still kind of hard to read, and it's far from ideal,
but I'm reluctant to remove it from the test suite.
Updates Zulip dev setup for first-time contributors as follows:
- Recommends and links to Vagrant version 1.8.4 (from 1.8.1)
- Consolidates Ubuntu 14.04 and Ubuntu 16.04 instructions since they are
now identical.
- Updates memory requirement to 2GB to address out of memory related
errors (see issue #1333).
Fixes#1288.
Factor out the code in tools/provision.py which installs a python2
and python3 venv into a module (tools/setup/setup_venvs.py) which
can also be used as a script.
This doc follows the different layers that Zulip uses
to handle a request, detailing parts of those layers that
would be most commonly interesting to developers.
Currently if you try to use a non-existent handlebars
template, you get this error:
Uncaught TypeError: Handlebars.templates[name] is not a function
This change makes it a little more clear what the issue is
and what the fix would be.
Sphinx/RTD creates different anchor tags for subheadings than
Markdown does, and treats image embedding differently.
This commit updates hyperlinks to affected anchors,
and fixes an image embed for move to RTD.
We aim to consolidate developer documentation on Read The Docs. As a
start, this commit turns the first part of README.dev.md into a topic
page within /docs/ , and links to it from the doc index.
See #669.
This adds support for passing a client cert (in the format expected by the
`requests` library) to the `Client` constructor, as well as for specifying
one on the command line or in .zuliprc (through new `client_cert` and
`client_cert_key` options).
Twisted is not python 3 compatible. So for now create a python2
venv and install twisted in it when running provision.py in python3
mode and use twisted from the python2 venv.
As they stand now, alert words tests will cause a race condition with
all subsequent tests which access the UserProfile object these tests
modify. Currently, if we modify alert words, we don't get any
notification from the server, issue reported at #1269. Consequently, we
can't wait on any condition to avoid the race condition. The best option
is to wait for the fix of #1269 and modify the tests in that issue.
Fixes: #1244
This increases test coverage by exercising highlight_string().
It also gives deeper test coverage to NarrowBuilder.by_search(),
which had test coverage before, but only in terms of inspecting
the SQL that was generated. This test actually runs the SQL
under the hood.
This partly fixes#1006.
This allows the frontend to fetch data on the subscribers list (etc.)
for streams where the user has never been subscribed, making it
possible to implement UI showing details like subscribe counts on the
subscriptions page.
This is likely a performance regression for very large teams with
large numbers of streams; we'll want to do some testing to determine
the impact (and thus whether we should make this feature only fully
enabled for larger realms).
Clear memcached when tools/run-dev.py is run. This prevents
errors on using a different python version because values are
pickled before being stored in memcached and different python
versions implement pickling differently.
Also provide a command-line option --no-clear-mc to prevent
memcached from being cleared.
tools/provision.py: Create directory var/uploads.
zproject/local_settings_template.py: Update Upload dir to var/uploads.
zproject/dev_settings.py: Update upload dir to var/uploads.
The muting logic in approximate_unread_count() was confusing
stream/subject and only using the first of many stream/subject
pairs, so it was rarely excluding rows from the count, and when
it did exclude rows, they were the wrong rows.
This fixes part of #1300, but we may want to keep the issue open.
This dependency is only needed if USING_STATSD is set, and so had
apparently been missed from our dependency list previously (since few
sites use that setting).
There is a hard to reproduce race condition causing these tests to
occasionally fail. We believe it is caused by switching to the home tab and
not properly waiting for all the messages to load; see Issue #1243. The
tests are for the following pathway (not a high priority to test):
1. User starts editing a message.
2. allow_message_editing is turned off for the realm (in this case, by the
user going to the admin page and turning it off).
3. User finishes editing the message and hits send.
Running `./manage.py email-mirror` used to fail on python 3
because twisted.mail.imap4 is not python 3 compatible.
Display a message informing the user that email-mirror is not
available on python 3 instead of failing with a traceback.
Also add tools/test-management to py3-backend.
runtornado unbuffers its output using
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0).
This is not python 3 compatible since we can't specify
buffering on a text stream in python 3. So use the '-u'
option of python when calling runtornado.py to make output
unbuffered.
There were a bunch of authorization and well-formedness checks in
zerver.lib.actions.do_update_message that I moved to
zerver.views.messages.update_message_backend.
Reason: by convention, functions in actions.py complete their actions;
error checking should be done outside the file when possible.
Fixes: #1150.
This is controlled through the admin tab and a new field in the Realms table.
Notes:
* The admin tab setting takes a value in minutes, whereas the backend stores it
in seconds.
* This setting is unused when allow_message_editing is false.
* There is some generosity in how the limit is enforced. For instance, if the
user sees the hovering edit button, we ensure they have at least 5 seconds to
click it, and if the user gets to the message edit form, we ensure they have
at least 10 seconds to make the edit, by relaxing the limit.
* This commit also includes a countdown timer in the message edit form.
Resolves#903.
Taiga's webhook integration would give output events in a random
order which caused test failures on python 3 (seems like python
3 is more prone to non-deterministic failures). Fix that by
sorting the outputs obtained from events before concatenating them.
Use ujson.dumps to render raw messages sent by the PagerDuty
integration instead of using pprint.pformat. pprint.pformat
gives different results on python 2 and 3.
Correctly encode and decode strings in convert_html_to_markdown.
It wasn't possible to use universal_newlines=True since
Popen.communicate doesn't encode/decode strings correctly on
python 2.
There is a hard to reproduce race condition causing these tests to
occasionally fail. We believe it is caused by switching to the home tab and
not properly waiting for all the messages to load; see Issue #1243. The
tests are for the following pathway (not a high priority to test):
1. User starts editing a message.
2. allow_message_editing is turned off for the realm (in this case, by the
user going to the admin page and turning it off).
3. User finishes editing the message and hits send.
The purpose of this is to move a lot of the log and other generated
files used by the Zulip development environment into a consistent
hierarchy.
We also need to create this in tools/build-release-tarball as well,
since that runs a development environment out of a temporary
directory.
For longer running servers, searching the backlog can become ambiguous
since the date stamps that demarcate the messages only include the month
and day. This commit changes the behavior to include the year for
messages which are more than a year old.
We were using data for both the request and response data in
$(".administration").on("submit", "form.admin-realm-form".
Left the request data as data to be consistent with the rest of the
file, renamed the response data to response_data.
Bitbucket changed the format of their API. The old format is still
useful for BitBucket enterprise, but for the main cloud verison of
Bitbucket, we need a new BitBucket integration supporting the new API.
To begin the process of consolidating Zulip documentation on Read
the Docs, the requirements are now their own topic page, and
index.rst links to it.
Fixes: #669.
Previously, the install script would fail if you passed various
non-default puppet rules, since the code to configure and restart
services that runs later on in the install script largely ran
unconditionally, regardless of whether the relevant service was
actually installed on the target system.
This should make the main install script reusable for installing
e.g. a dedicated Postgres server for use with Zulip.
Add methods assert_equals_response and assert_in_response to
AuthedTestCase. These methods make it convenient to check if
a string equals the contents of an HttpResponse's body or if a
string is a substring of the contents of an HttpResponse's body.
response.content is binary data, but code usually assumes it to
be text. Fix this by decoding response.content where required.
Don't do this in tests yet.
This is controlled through the admin tab and a new field in the Realms
table. This mirrors the behavior of the old hardcoded setting
feature_flags.disable_message_editing. Partially resolves#903.
This reverts commit f1f48f305e.
The use of sklearn unfortunately caused a substantial slowdown to the
Zulip provisioning process, which didn't seem worth it for a
relatively minor feature.
In write_log_line, error_content can be binary_type and
error_content_iter can be a Sequence of binary_type. Handle
this this in a python 3 compatible way. Also change annotations
to reflect this fact.
If settings.RATE_LIMITING is False, short circuit rate
limiting earlier in rate_limit(). This change particularly
avoids inspect request.user and possibly spamming the error
log for sites that don't care about rate limiting.
Instead of just checking .html files in the templates directory,
we now also check them everywhere, including .html files used
for Casper tests and some static files.
Originally this cache was used to transmit data from Django to Tornado
(and also for general message caching purposes), but now nothing
actually reads from this cache, so we can eliminate it.
Apparently, the message cache we were filling was completely useless
and unused, and furthermore, the cache we were filling as part of
restarting the server was also totally useless, since it didn't have
the messages users would be requesting.
Apparently, puppet has messed up exit codes and doesn't by default
return the usual 0=success, nonzero=failure codes. By default, it
seems to always return 0; and with `--detailed-exitcodes`, it returns
the complicated thing documented in the comments.
We fix this by checking the exit code and translating it to what we
actually care about, namely whether errors occurred.
See https://tickets.puppetlabs.com/browse/PUP-2754 for details.
Fixes#1094.
* get_realm returns None if no matching realm is present, but
create_stream.py assumed it raises Realm.DoesNotExist.
* encoded/decode strings properly.
* Replace filter by list comprehension.
* Add '# type: ignore' to statements which use attributes from
modeule `posix`, since stubs for posix are missing on python 3.
* get_realm returns None if no matching realm is present, but
create_stream.py assumed it raises Realm.DoesNotExist.
* encoded/decode strings properly.
The functions truncate_content, truncate_body and truncate_topic
are only meant to be used on text strings. So change its
parameter types from AnyStr to text_type.
Many stubs in xml.etree.ElementTree use Union[str, bytes] as
return type. Mypy wants us to correctly handle each case. This
is correct, but not useful for us since we know that we'll always
get str. So force the return value to text_type, to supress mypy
errors.
We originally waited for .message_edit_notice to appear, now we wait for
textarea.message_edit_content to disappear.
This is better because the previous code didn't correctly handle
editing the same message twice (the "EDITED" tag would still be there
from the first edit, so it wouldn't wait at all the second time!).
When provision.py is run using python 2, retain original behavior
of creating a python 2 venv using requirements/py2_dev.txt and
creating a python 3 venv using requirements/mypy.txt.
When provision.py is run using python 3, install a single python 3
venv with requirements/py3_dev.txt.
In python 3, subprocess uses bytes for input and output if
universal_newlines=False (the default). It uses str for input and
output if universal_newlines=True.
Since we're dealing with strings here, add universal_newlines=True
to subprocess.check_output calls.
Squash the AlterField on UserProfile.groups in 0002 into the
AddField in 0001. This is done to avoid a probable bug in Django,
where running migrations in python 3 sometimes led to a KeyError.
Subprocess.CalledProcessError's __init__ takes 4 arguments out
of which 2 are optional. It is being passed 2 arguments in
zulip_tools.py. However, python 3 stubs are incorrect and only
1 argument has been marked as default.
See https://github.com/python/typeshed/pull/329.
Add '# type: ignore' to that line so that mypy check passes in
python 3 mode.
The subscribers list is appended to in `peer_add` events with not
regard for preserving the ordering, and ordering isn't really
important here, so it seems best to just sort it in these checks.
This works around a nasty problem with Webpack that you can't run two
copies of the Webpack development server on the same project at the
same time (even if on different ports). The second copy doesn't fail,
it just hangs waiting for some lock, which is confusing; but even if
that were to be solved, we don't actually need the webpack development
server running to run the Casper tests; we just need bundle.js built.
So the easy solution is to just run webpack manually and be sure to
include bundle.js in the JS_SPECS entry.
As a follow-up to this change, we should clean up how test_settings.py
is implemented to not require duplicating code from settings.py.
Fixes#878.
This fixes some tracebacks I got while testing the Zulip htpasswd SSO
functionality.
I think that this stopped working as a result of the Jinja2 migration.
Installing `nginx` is problematic, because it means something is using
port 80 and thus `letsencrypt-auto --standalone` doesn't work. But we
do want `openssl` so that the directory trees we create symlinks in
will exist.
Also encode/decode strings appropriately when using api_keys to generate
basic auth header.
Also fix clashing annotations in zerver/tests/test_external.py.
* Add Optional where required.
* Set type of req_redis_key as `(text_type) -> text_type` for consistency.
Almost all our cache keys and redis keys use this signature.
This is important for both ensuring the Nagios checks work correctly
in production, as well as making sure the `zulip` user can access the
virtualenv (owned by the `travis` user) in Travis CI.
Rather than looking at which venvs are used by this particular build,
we instead look at which venvs have a hash that is the hash_reqs value
of a current requirements.txt file.
The manage.py change effectively switches the Zulip production server
to use the virtualenv, since all of our supervisord commands for the
various Python services go through manage.py.
Additionally, this migrates the production scripts and Nagios plugins
to use the virtualenv as well.
netifaces is used in puppet/zulip_internal/files/zulip-ec2-configure-interfaces.
flup is used for fastcgi.
python-dateutils is used in some puppet scripts.
We would like to know which kind of authentication backends the server
supports.
This is information you can get from /login, but not in a way easily
parseable by API apps (e.g. the Zulip mobile apps).
This prototype from Dropbox Hack Week turned out to be too inefficient
to be used for realms with any significant amount of history, so we're
removing it.
It will be replaced by https://github.com/zulip/zulip/pull/673.
This reverts commit e985b57259.
This commit will break production when we next do a release, because
we haven't done a migration to create Attachment objects for
previously uploaded files.
For a long time, rest_dispatch has had this hack where we have to
create a copy of it in each views file using it, in order to directly
access the globals list in that file. This removes that hack, instead
making rest_dispatch just use Django's import_string to access the
target method to use.
[tweaked and reorganized from acrefoot's original branch in various
ways by tabbott]
Currently we use the deprecated django pattern() prefix pattern.
This make it hard to read the router logic in zproject/urls.py
This commit denormalizes the urls so that they can be read
more easily, at the expense of some verbosity. This also makes it
easier to reorganize urls in that file.
We skip denomalizing rest_dispatch due to its unique complications.
Apparently, c74a74dc74 introduced a bug
where we are no longer correctly depending on build-essential as part
of the Zulip development environment installation process.
Fixes#1111.
Assigns hotkey 'w' to search streams.
Only show search box when active. Activate with hotkey or by clicking
STREAMS.
Filter matches at the beginning of words in stream name.
Behaviour is otherwise almost identical to user search.
Casper tests.
The `email` parameter to this template is only set if the user is
redirected to the login form after attempting to register for a new
account with an email address that already has an account.
We only use zxcvbn in the main webapp for checking the user's password
in the change password form. Since zxcvbn is a very large javascript
library (~700KB), loading it asynchronously only when a user is trying
to change their password results in a significant performance
improvement for loading the Zulip webapp on a slow network.
Fixes#263.
Previously, we were wasting time every time we installed packages,
because `apt-get upgrade` would only install most of the packages
`apt-get dist-upgrade` would.
This is needed because hash_reqs.py is used to create a virtualenv.
Currently we only use virtualenv in development, but we will soon
start using it in production. Scripts used in production should be
put in scripts/.
If a user's session cookie expired, the next REST API request their
browser did would go into the json_unauthorized code path. This
returned a response with a WWW-Authenticate tag for HTTP Basic Auth
(since that's what the REST API uses), even for /json requests which
should only be authenticated using session auth.
We fix this by explicitly passing the desired WWW-Authenticate state.
Fixes: #800.
The recent changes to api_fetch_api_key to receive detailed data via
the "return_data" object did not properly update the LDAP backend to
accept that argument, causing mobile password authentication to not
work with the LDAP backend.
tools/hash_reqs.py generates a hash of a requirements file. It
does that by generating a sorted list of unique dependencies referred
to by that requirements file. To do that, it also recurses into other
requirements files specified inside that requirements file.
After annotating rate_limiter.rules, mypy complained that rules does
not support cmp. So use key to customize sort instead of cmp.
Python docs also recommend using key over cmp.
zerver.lib.initial_password.initial_password is supposed to return an
Optional[text_type], but it returns an Optional[binary_type] instead.
Encode the return value to make sure it returns an Optional[text_type].
Add a class 'BaseHandler' and make it a base class of OuterHandler,
QuoteHandler and CodeHandler. This will help annotate some functions
and improve type checking.
This was confusing, because we otherwise suggest consistently to users
that they should connect to their development environment on port
9991.
Fixes#1032.
Apparently, there are like 5 independently developed jquery-caret
plugins, none of which are great. The previous one we were using was
last modified in 2010. This new one comes from
https://github.com/acdvorak/jquery.caret and at least doesn't use
deprecated jQuery syntax and has a repository on GitHub.
This plugin is way larger than it needs to be for what it does, but we
can deal with that later.
Place all hotkey names into a set of three objects in hotkeys.js:
* hotkeys_shift_insensitive
These are keys where the behaviour is the same whether they are
pressed with shift or not.
* hotkeys_no_modifiers
These are keys where the event should only be fired when shift
is not being pressed.
* hotkeys_shift
These are keys where the event should only be fired when the key
is pressed simultaneously with shift.
Each object is a dictionary of key value pairs, with the key being
the keyscan code (e.which) for the key. This is normally the ASCII
key code. The value is an object with two properties, name (which
is the event name) and message_view_only, a boolean. Hotkeys with
message_view_only set to true will not be fired when the home tab
is obscured.
Following strings are marked translatable:
- All strings which are passed to `button.text` or which affect the
text of buttons.
- All strings passed to `placeholder`.
- All strings passed to `compose_error`.
Fixes#969
get_display_recipient's annotation clashes with other wrong annotations.
Fix those wrong annotations.
Since get_display_recipient returns a Union, use isinstance checks and
casts to make mypy checks succeed.
We'll likely want to follow this up with:
(1) Making Let's Encrypt the default instructions for getting a cert.
(2) Moving the "how to get an SSL cert" discussion to its own page.
(3) Eliminating the duplicate content in zulip.org/server.html
But this is definitely an improvement!
Identify functions which return QuerySets and give them a return type
`Sequence` with appropriate parameter. Typing them as QuerySet will
not be useful since generic stubs for QuerySets are not available and
not knowing the type of QuerySets is hardly useful for type checking.
generate_random_token used to return a value of type six.binary_type
and its return type was annotated as `str`. This commit fixes that
by making it return a value of type `six.text_type` and updating
the annotation accordingly.
Also fix clashing annnotations.
Change choices of UserProfile.avatar_sources and UserProfile.tutorial_status
from str literals to unicode literals. This is done because these fields
are CharFields, which are of type `six.text_type`. So the set of values
which they can take should also be of the type `six.text_type`.
Also fix clashing annotations.
Change `str` to `text_type` in annotations in zerver/models.py
related to realm emoji and realm filters.
Also fix clashing annotations in zerver/lib/bugdown/__init__.py.
Due to a cyclic dependency issue, functions having models as parameters
were annotated as Any.
That issue is fixed by importing models inside an `if False:` block,
so that mypy sees them but they are not imported at runtime.
In update_user_profile_caches, the return type in annotation was
marked as Any. Change that to None because, nothing is being returned
in that function.
This changes the type annotations for the cache keys in Zulip to be
consistently text_type, and updates the annotations for values that
are used as cache keys across the codebase.
str_utils.py has functions for converting strings from one type to
another. It also has a TypeVar called NonBinaryStr, which is like AnyStr
except that it doesn't allow bytes.
The previous Markdown syntax didn't work correctly on ReadTheDocs, and
now that we have Coveralls reporting backend test coverage as well,
describing our coverage as just "mypy static type" is no longer accurate.
Fixes#964.
Now that we have a working S3 mock and an effective way to toggle the
upload backend that Zulip is using, we can re-enable this important
end-to-end test of the Zulip S3 upload backend.
Previously, uploaded files were served:
* With S3UploadBackend, via get_uploaded_file (redirects to S3)
* With LocalUploadBackend in production, via nginx directly
* With LocalUploadBackend in development, via Django's static file server
This changes that last case to use get_uploaded_file in development,
which is a key step towards being able to do proper access control
authorization.
Does not affect production.
This has no functional changes; we just replace the old hacky
assignment of functions with assignment of the upload backend to a
variable.
I'm not totally happy with this, because we end up having to copy the
type annotations of the three methods 4 times each, but this should
make it a lot easier to test the (non-default-in-tests) S3 backend
using end-to-end tests, which would have caught
13bac1cc2a.
I expect we'll iterate on the interface over time; ideally, I'd like
all the code that checks LOCAL_UPLOADS_DIR to be inside upload.py, and
primarily in these classes.
* The warning contains a count of the number of people in the stream.
* An error appears if the warning is ignored and the user tries to
send the message anyway.
* The message cannot be sent until the warning is acknowledged or @all
/ @everyone is removed.
* This only applies to stream messages and not private messages.
Fixes#853.
Previously, we were checking if a particular user was the current user
in dozens of places in the codebase, and correct case-insensitive
checks were not used consistently, leading to bugs like #502.
Calling open() with mode 'w' or 'a' will create a file if it doesn't exist,
while mode 'r' will cause an exception. This can be easily tested with:
python -c 'open("test.tmp", "w")'
ls test.tmp
Also fix the annotation of zilencer.views.report_error.
The `report` arguments are a Dict containing both strings and the
`more_info` sub-dictionary, so we type them as Dict[str, Any].
[tweaked by tabbott]
Since relatively few systems have the typing module, this makes the
checks for whether the user is properly running our test scripts in
the virtualenv more likely to trigger well.
This should make users much more likely to be able to debug issues
where they ran Zulip outside the Vagrant environment or virtualenv.
[error messages tweaked by tabbott]
`makemessages` escapes the `%` sign in `.po` files, but Jinja2 does
not unescape it while replacing the tranlation strings. In Jinja2,
there is an updated implementation of gettext available called
new-style gettext which handles escaping better; this commit switches
to using that.
Fixes#906.
Type of parameter for function `is_recent`(line no.812) is `datetime`.
MyPy errors out, however, when the parameter is defined as `datetime`.
To get around, type `Any` is used.
Also, fixed a a small type annotation in users.py because email must
be a string because emails don't support UTF-8 at this time (according
a comment in gravatar_hash in avatar.py).
Currently this uses a Union type for connection_id; we need to figure
out what actually sets that and what its type is and fix that later
(see https://github.com/zulip/zulip/issues/896).
.message_edit_notice is too broad of a selector to
test if your most recent edit has posted. Also
check if the currently selected message is the one
that's been edited.
When you deactivate a user, visit another page, and then
come back, the user shows up in another table (the
"Deactivated Users" table), and no longer has the
strikeout styling of a recently deactivated user.
Using the strikeout class as a selector to test if the
user has been reactivated will of course not be a good
test, and if you have a slow machine, lose a
race condition.
When you deactivate a user, visit another page,
and then come back, the user shows up in another
table, and no longer has the strikeout styling
of a recently deactivated user. Using strikeout
class to test if the user has been reactivated
will of course not be a good test, and if you
have a slow machine, lose a race condition.
Also, fixed up the annotations for tornadoviews to better align with
how narrows was defined as `Iterable[Sequence[str]]` rather than
`List[Tuple[str, str]]`.
Had to add some "type: ignore" because the pattern used in match
doesn't affect the type returned. A fix for this issue has been pushed
to typeshed - https://github.com/python/typeshed/pull/244
These ones don't fix any bugs, because the mutable arg is never passed
outside of the callable or mutated. But it's good practice to not use
them in case those invariants are changed in the future.
[Substantially revised by tabbott]
This probably still has some bugs in it, but having mostly complete
annotations for models.py will help a lot for the annotations folks
are adding to other files.
In order to genericize use of Zulip outside companies,
all instances of coworkers have been changed to users.
NOTABLE EXCEPTION: When the Zulip instance is domain-
locked, the reference to coworkers remains. The reason
for this is twofold: first, the majority of Zulip instances
which require a particular domain will be locked to a
company, and second, the template variable for the domain
necessary should be added to the alert so it is clear
to the user what the domain needs to be for access.
Fixes: #861.
Currently (https://github.com/rtfd/recommonmark/issues/3)
the recommonmark bridge (which allows Sphinx to read
Markdown) does not support tables, so the directory structure
doc is now a bulleted list instead of a set of tables.
The order and structure of the Read The Docs documentation
was fairly arbitrary; here's a suggestion for how to change
its information architecture so it's more useful to a new
contributor. This adds a short Zulip overview, and a table
of contents and code contribution intro for new contributors.
Fixes: #668.
Signed-off-by: Sumana Harihareswara <sumanah@panix.com>
Fixes#747. Fixes#748.
This updates README.dev.md to include clear, step-by-step instructions
for setting up the Zulip dev environment Windows 10, OS X El Capitan,
Ubuntu 14.04 Trusty, and Ubuntu 16.04 Xenial. It is aimed at first-time
contributors.
I tested these instructions multiple times on each of the target
systems.
Also added is a "trobleshooting and common errors" section which
documents the most common errors contributors are likely to encounter
during setup and how to fix them.
Improvements based on feedback from @timabbott.
- Fixes whitespace issues so linter will pass.
- Updates memory requirement.
- Re-orders so operating systems are listed alphabetically.
- Updates headings to be clearer.
- Updates and adds ToC entries for clarity.
- Adds screen shot of Zulip dev environment running in browser.
- Adds details about using dev environment, including about logging.
- Misc other minor changes for clarity.
- Adds a stub for docs/logging.md
- Adds details about configuring Cygwin for native symlinks.
Just render the templates without the actual workflow to see if they
don't return a 500 error; this lets us catch various classes of
template bugs automatically.
Fixes#784.
Add two options to the `test-backend` script:
1. verbose
If given the `test-backend` script will give detailed output.
2. no-shallow
Default value is False. If given the `test-backend` script will
fail if it finds a template which is shallow tested.
This lets us cut out the line which hard-codes how deeply nested in
the tree the `run-mypy` script is, making it simpler to borrow these
scripts in other projects.
This stub file allows us to annotate view functions using the actual
types present in the bodies of the functions, rather than everything
having the type REQ.
In function bulk_add_subscriptions, some variables were named
`stream_name` but their type is Stream, not a string. Rename
those variables to `stream`.
Previously lister.py used to check whether the exclude path is a
substring of a path being considered. So it would fail when the
exclude path is an absolute path or uses '..' or '.'.
We need to update provision.py to compile the messages files, since
they are needed for the new i18n tests. And of course we need to
include the .mo files in release tarballs; there's a bit of complexity
there around how the tarball archives are created.
Since we merged cd2348e9ae more than a
month ago and haven't seen any noticable regresions as a result, it's
reasonable at this point to do a corresponding decrease in our
documented RAM requirements for the Zulip development environment.
Like the recent change blocking JSON endpoints for deactivated users
and users in deactivated realms, this change is a hardening
improvement. Those users should be unable to get an active session
anyway, but if somehow one is leaked, this means they won't be able to
access any user data.
Previously, api_fetch_api_key would not give clear error messages if
password auth was disabled or the user's realm had been deactivated;
additionally, the account disabled error stopped triggering when we
moved the active account check into the auth decorators.
While in theory users should be unable to get a valid session in order
to access these endpoints in the first place, this provides an extra
layer of hardering to prevent a deactivated user with a session from
accessing data via the old-style JSON API.
The security model for deactivated users (and users in deactivated
realms) being unable to access the service is intended to work via two
mechanisms:
* All active user sessions are deleted, and all login code paths
(where a user could get a new session) check whether the user (or
realm) is inactive before authorizing the request, preventing the
user from accessing the website and AJAX endpoints.
* All API code paths (which don't require a session) check whether the
user (and realm) are active.
However, this security model was not implemented correctly. In
particular, the check for whether a user has an active account in the
login process was done inside the login form's validators, which meant
that authentication mechanisms that did not use the login form
(e.g. Google and REMOTE_USER auth) could succeed in granting a session
even with an inactive account. The Zulip homepage would still fail to
load because the code for / includes an API call to Tornado authorized
by the user's token that would fail, but this mechanism could allow an
inactive user to access realm data or users to access data in a
deactivated realm.
This fixes the issue by adding explicit checks for inactive users and
inactive realms in all authentication backends (even those that were
already protected by the login form validator).
Mirror dummy users are already inactive, so we can remove the explicit
code around mirror dummy users.
The following commits add a complete set of tests for Zulip's inactive
user and realm security model.
In a deactivated realm, webhooks would still successfully send
messages, since there was no check for whether the realm was active in
api_key_only_webhook_view.
Long ago, there was work on an experimental integration model where
every user in a realm would have administrative control over all bots,
with the goal of simplifying the process of setting up communally
administered bots for smaller teams. While that new model was never
fully implemented (and thus never setup as an option), an error in
that original implementation meant that the data on all bots in a
realm, including their API keys, was sent to the browsers of users via
the `realm_bots` variable in `page_params`. The data wasn't displayed
in the UI for non-admin users, but was available via e.g. the
javascript console.
This commit updates this behavior to only send sensitive bot data like
API keys to the owner of the bot (and realm admins).
We may in the future implement a model simplifying communally
administered integrations, but if we do that, those bots should be
limited in their capabilities (e.g. only able to send webhook
messages).
This bug has been present since Zulip was released as open source.
The old code for this lookup was unnecessarily complicated because we
were working around Guardian, where the `is_realm_admin` check was
extremely expensive.
Previously we relied on having two matching list of fields for the
get_active_user_dicts_in_realm, one in the actual code and the other
in the caching system. By unifying these lists to have a single
source, we eliminate a class of caching bugs we might otherwise
regularly introduce.
This results in a substantial performance improvement for all of
Zulip's backend templates.
Changes in templates:
- Change `block.super` to `super()`.
- Remove `load` tag because Jinja2 doesn't support it.
- Use `minified_js()|safe` instead of `{% minified_js %}`.
- Use `compressed_css()|safe` instead of `{% compressed_css %}`.
- `forloop.first` -> `loop.first`.
- Use `{{ csrf_input }}` instead of `{% csrf_token %}`.
- Use `{# ... #}` instead of `{% comment %}`.
- Use `url()` instead of `{% url %}`.
- Use `_()` instead of `{% trans %}` because in Jinja `trans` is a block tag.
- Use `{% trans %}` instead of `{% blocktrans %}`.
- Use `{% raw %}` instead of `{% verbatim %}`.
Changes in tools:
- Check for `trans` block in `check-templates` instead of `blocktrans`
Changes in backend:
- Create custom `render_to_response` function which takes `request` objects
instead of `RequestContext` object. There are two reasons to do this:
1. `RequestContext` is not compatible with Jinja2
2. `RequestContext` in `render_to_response` is deprecated.
- Add Jinja2 related support files in zproject/jinja2 directory. It
includes a custom backend and a template renderer, compressors for js
and css and Jinja2 environment handler.
- Enable `slugify` and `pluralize` filters in Jinja2 environment.
Fixes#620.
In theory these should be the same, but in misconfigured environments
(such at Travis CI) where /etc/hosts has multiple entries for
"localhost", 127.0.0.1 is safer than "localhost".
This fixes an issue introduced by
b21454d05e, where the typed_ast module
wouldn't build properly when provisioning, because we didn't have the
necessary headers installed.
To avoid the potential for introducing regressions here, we carefully
pass a default to REQ or not based on how the existing webhook's
parsing code worked. In the longer term, we'll want to make the
behavior consistent.
This makes mypy about 15% faster running on the Zulip codebase (from
7s=>6s on my laptop), which seems worth it for losing a couple files.
This option requires a new dependency, which we add to the
mypy-specific requirements.txt file.
Also update the mypy command line to not use deprecated argument names.
This introduces a few errors, so we exclude the relevant files to keep
the mypy output clean.
This fixes an exception where client_id was never set in an error code
path. It shouldn't be needed, but I think this makes the code clearer
and this will help in debugging the actual problem.
Related to #753.
The error message for a test file that doesn't import properly was
previously pretty difficult to understand and it wasn't clear how to
debug the issue.
This should save a couple minutes on the time it takes to provision a
Zulip environment on a machine that has provisioned with the same
requirements.txt file content before.
We can't yet use this in Travis CI because Travis CI doesn't support
using both sudo and caching in the same build.
This commit adds the capability to keep track and remove uploaded
files. Unclaimed attachments are files that have been uploaded to the
server but are not referred in any messages. A management command to
remove old unclaimed files after a week is also included.
Tests for getting the file referred in messages are also included.
Camo is a caching image proxy, used in Zulip to avoid mixed-content
warnings by proxying HTTP image content over HTTPS. We've been using
it in zulip.com production for years; this change makes it available
in standalone Zulip deployments.
This should save several minutes off the Travis CI `production`
suite's runtime, since previously we were doing the full apt upgrade
process twice, resulting in things like multiple expensive rebuilds of
the initramfs.
Add 'six' to setup-py3k, because it is being used in tools/lister.py.
Add 'typing' to setup-py3k, so that tools/lister.py can be type
annotated in the future.
Since we don't have a stable way to get the Dropbox preview failure
image (and it was sorta a weird setup anyway), it seems best to just
remove the condition.
Previously, the user list would remain filtered after a user hit enter
to start composing a message to a user, leaving them in a state with a
partial user list.
Fixes#360.
Previously we needed to use a specified password when activating a
formerly mirror dummy user, in order for that user to be able to
(re)set their password and login. Now that we have our own password
reset form, this is no longer required.
Previously, if a user had only authenticated via Google auth, they
would be unable to reset their password in order to set one (which is
needed to setup the mobile apps, for example).
Now tools/check-py3 will by default run all fixers together. This is
quicker but doesn't indicate which fixers caused the failure. The
newly added option --find-fixers falls back to the old way of checking
each fixer separately if the quick check fails.
Fixes#710.
Apparently LXC 2 removed support for the `-B best` option in
lxc-create, and Vagrant hasn't been updated appropriately yet, so we
need to add a workaround to explicitly specify a backing store.
Fixes#718.
This manifested as errors of the form:
"""
There was an error executing ["sudo",
"/usr/local/bin/vagrant-lxc-wrapper", "lxc-create", "-B", "best",
"--template",
"/home/tabbott/.vagrant.d/gems/gems/vagrant-lxc-1.2.1/scripts/lxc-template",
"--name", "zulip_default_1461801696512_85064", "--", "--tarball",
"/home/tabbott/.vagrant.d/boxes/fgrehm-VAGRANTSLASH-trusty64-lxc/1.2.0/lxc/rootfs.tar.gz",
"--config",
"/home/tabbott/.vagrant.d/boxes/fgrehm-VAGRANTSLASH-trusty64-lxc/1.2.0/lxc/lxc-config"]
"""
Several recently merged webhooks were incorrectly not checking that
the actual webhook result didn't return an error. While they would
usually still fail in most cases when checking whether the message
came back correctly, this hid the root cause errors and thus made it
much harder to debug.
We were incorrectly applying the rate limiting rules to webhooks even
if rate limiting was disabled (as in the test suite), causing test
failures when the total number of webhook tests in Zulip got too high.
In theory, tools like populate_db should probably be in zerver, not
zilencer, but until we migrate them out, we need to include these in
EXTRA_INSTALLED_APPS in development.
The previous separated-out configuration wasn't helping us, and this
makes it easier to make the extra installed applications pluggable in
the following commits.
This will merge conflict with every new integraiton in flight, which
is unfortunate, but will make there be fewer merge conflicts as people
add new webhooks in the future (currently, every pair of new
integrations conflict because folks are adding them all at the end,
whereas after this change, there will only be merge conflicts when
adding two integrations near each other alphabetically).
This integration relies on the Teamcity "tcWebHooks" plugin which is
available at
https://netwolfuk.wordpress.com/category/teamcity/tcplugins/tcwebhooks/
It posts build fail and success notifications to a stream specified in
the webhook URL.
It uses the name of the build configuration as the topic.
For personal builds, it tries to map the Teamcity username to a Zulip
username, and sends a private message to that person.
As documented in https://github.com/zulip/zulip/issues/441, Guardian
has quite poor performance, and in fact almost 50% of the time spent
running the Zulip backend test suite on my laptop was inside Guardian.
As part of this migration, we also clean up the old API_SUPER_USERS
variable used to mark EMAIL_GATEWAY_BOT as an API super user; now that
permission is managed entirely via the database.
When rebasing past this commit, developers will need to do a
`manage.py migrate` in order to apply the migration changes before the
server will run again.
We can't yet remove Guardian from INSTALLED_APPS, requirements.txt,
etc. in this release, because otherwise the reverse migration won't
work.
Fixes#441.
When uploaded avatar image is not a valid image file, PIL raises
IOError. Catch the IOError raised by PIL and raise JsonableError.
This will return a response with status code 400.
S3Test is now only the S3-specific test (which isn't even run), so we
can now invest in making FileUploadTest have good coverage of the
(local) file upload code paths.
The main function of prompting inside `manage.py migrate` is to ask
the user if they want to delete stale content-types, which is
unimportant and likely scary, so we disable doing so.
Previously, the UserProfile objects were created in the order
generated by a Set, which meant tests would randomly start failing if
the code that runs before this part of populate_db changed (and thus
caused the Set object used to pass users into bulk_create_users to
have a different order when enumerated).
This fixes the issue in two ways -- one by sorting the users inside
bulk_create_users, and second by attaching subscriptions to users
based on a deterministic ordering.
The restarted Tornado processes seemed to escape the process group and
thus continue running after run-dev.py finished.
While we're at it, we don't need to dump/reload event queues in the
test suite either.
Move recenter_pointer_on_display, suppress_scroll_pointer_update,
fast_forward_pointer, furthest_read, and server_furthest_read to
a new pointer module in pointer.js.
The `with sh.sudo` pattern that we were using in python-sh was
deprecated, and emperically hangs on Ubuntu xenial. Since in general
the use of python-sh/python-pbs caused trouble (requiring extra
dependencies, confusing syntax), this just removes it.
We replace it with a new zulip_tools.py library function that echoes
the command line and streams the output.
We do the same to install-phantomjs so we can remove that dependency.
Previously, the Zulip subscriptions page's error bar would always be
at the very top of the scrollable view, and thus would likely be out
of view when an error happened. This fixes it by having the error bar
always placed below the search box (and thus visible regardless of
where in the scrollable streams view we are).
Fixes: #515.
[commit message and comments expanded by tabbott]
This reverts commit d936bf61f9.
We no longer need this since we've migrated to specifying the
dependencies in the typing module that we're actually using.
This fixes an issue where this worker wasn't even being installed
properly in a way that sets us up for doing further reorganization of
the Zulip Nagios plugins.
tools/travis/py3k used to always exit with exit code 0.
It should exit with 1 when fixers detect a compatibility issue.
py3k used [ -z "$failed" ] to check if there was a failure.
This is wrong, since if no failure has occured, failed=0,
and -z checks if a string is of zero length. This commit also
fixes this bug.
In py3k, "git reset --hard" was called only if
libmodernize.fixes.fix_dict_six changed files and some of those
changes are not considered false positives by py3k.
But if all of those changes are not considered false positives
by py3k, then "git reset --hard" is not called and the repository
is no longer clean.
This commit fixes this bug.
tools/travis/py3k used to only check files whose names ended with .py.
Now it also checks python scripts which don't have an extension.
It uses tools/lister.py to get a list of all python files.
The previous version of sanitize_name dropped all unicode characters
and mangled filenames with multiple `.`s in the extension, leading to
confusing URLs for files uploaded to Zulip.
Fixes#321.
[tweaked significantly by tabbott]
This has the side effect of making lint-all check all shell scripts,
not just those under scripts/, tools/, and bin/.
[commit message expanded by tabbott]
Make module tools/lister.py which lists all files in a directory
tracked by git. This is done because lister.py will be used by other
scripts in the future which have to introspect files in the repository,
like linters, static code checkers, etc.
It's always been the case that in production, Tornado dumps all the
event queues when shut down so that they can be reloaded by the
replacement Tornado process. This never worked in development because
the codepath for auto-reload didn't go through either a signal or
sys.exit (it re-execs the process instead).
This meant that we didn't have a mechanism for testing the event queue
dump/load functionality in the development environment. We fix this
by adding such dumping/loading. However, this breaks the automatic
reloading of open browser windows on a server restart, so we add that
back in by adjusting the special `restart` events to pass a special
`immediate` flag when used in development.
This also has the benefit of removing the "Bad event queue" errors one
would get on every file save induced restart on the Python console.
Apparently it isn't always the case that removal of jquery and the DOM
prevents cleanup_event_queue from being called via the postunload
hook, so add a check to avoid it being double-called.
Previously, the browser might restart a get_events operation even
while it was in the middle of executing a `DELETE /events` query to
cause its event queue to be de-allocated. This was a rare race
condition when we weren't notifying clients when event queues were
de-allocated, but this will become a common case in the next commit.
This is a no-op right now, but we'll want the new structure for the
next commit, and splitting this out makes it a lot easier to read what
is actually changed in the next commit.
This should fix a problem we've been having with errors downloading
the PhantomJS packages from their original hosting service.
Eventually we should move it to an S3 bucket.
cd2348e9ae broke installing Zulip in
production since it didn't correctly update the puppet configuration
to call the process_queue script using the new argument format.
This commit isn't ideal in that I'd prefer to not require updating
puppet in sync with the actual running code, but we don't have a great
mechanism for doing that.
Fixes#586.
Previously, we needed to update the installation instructions with the
current version of Zulip in production every time we did a release,
which was kinda a pain (and hadn't happened since 1.3.6).
Fixes#576.
[commit message details expanded by tabbott]
The original logic for incremental presence list updating from
668d0d9dfa incorrectly attempted to
insert the user 1 spot later than its proper index in the listing.
Now that we're doing presence updates in a performant fashion, we
don't need to throttle processing these events, and in fact the
throttling of these events created a correctness problem, since we're
now doing incremental updates rather than just rerendering everything
after each event.
The code in 668d0d9dfa for removing an
existing user from the user list to update the status didn't correctly
quote the email address of the user in its jquery selector.
While we already don't link to /terms anywhere on the site, they can still be
accessed if you navigate to /terms directly. Now, those routes will only be
exported on the Zulip.com service.
We should ideally provide a mechanism for deployments to specify their own
terms without modifying source code; in the interim, sites that have already
customised the provided Zulip.com terms can simply carry a patch reverting this
commit.
This change drops the memory used for Python processes run by Zulip in
development from about 1GB to 300MB on my laptop.
On the front of safety, http://pika.readthedocs.org/en/latest/faq.html
explains "Pika does not have any notion of threading in the code. If
you want to use Pika with threading, make sure you have a Pika
connection per thread, created in that thread. It is not safe to share
one Pika connection across threads.". Since this code only connects
to rabbitmq inside the individual threads, I believe this should be
safe.
Progress towards #32.
First user-fasing problem is that when user click to "Collapse" button
of message from narrowed list, buttons "Uncollapse" and "[More...]" does
not work. Second, is that when user collapse/uncollapse some message
from narrowed list, the collapsing/uncollapsing of the same message in
home list does not work in appropriate way.
In "popovers.js" there is the function that is called on click to the
buttons "Collapse" or "Un-collapse". It should show and hide body of a
message. If a message list is narrowed, it should show/hide message in
home list too. So, the first problem is that "toggle_row()" in this
function call methods "collapse(row)" or "uncollapse(row)" from
"condense.js" twice (for row and home_row) using condition
"if (message.collapsed)". When it happen the first time, the variable
"message.collapsed" is changed. That is why next call of "toggle_row()"
work incorrectly.
The second problem is that the function in "condense.js" that is
called on click to the button "[More...]" contains no code for
collapsing/uncollapsing message from home list. It just calls
"collapse(row)" or "uncollapse(row)" for row from narrowed list.
Now, functions "collapse(row)" and "uncollapse(row)" get row from
current list and change both messages (from current list and home
list). On-click functions call them just once for making all of needed
message changes. So, when user collapse or uncollapse message from
home or narrowed list it works correctly.
Fixes: #516
Apparently, our event queue garbage collection logic never actually
disconnected any existing handler objects.
We fix this by disconnecting the handler inside cleanup(), adding a
special check to avoid creating a pointless timeout object.
The new Tornado handler tracking logic properly handled requests that
threw an exception or followed the RespondAsynchronously code path,
but did not properly de-allocated the handler in the syncronous case.
An easy reproducer for this is to load a new Zulip browser window;
that will leak 2 handler objects for the 2 synchronous requests made
from Django to Tornado as part of initial state fetching.
This line appears to have been lost in rebasing from the original
implementation of 1396eb7022faec4c2d91553800a35781a96dd5bd; so the
previous fix actually only addressed the issue in a rare exception
case.
The recent Tornado memory leak fix
(1396eb7022) didn't use the correct
variable name for the current handler ID, causing this cleanup code to
fail in the event that a view raised an exception.
Replaced calls to ifilterfalse by list comprehensions because
ifilterfalse is not part of python 3. Also changed some lists to sets
for faster lookup.
Refer to #256.
This automatically loads settings, zerver.models.* and
zerver.lib.actions.* when you start `manage.py shell`, which should
save a bit of time basically every time someone uses it.
Fixes#275.
Previously, even though the Zulip digest emails were documented in the
settings, the cron job to run the script that actually sends the daily
digest emails wasn't included in the non-zulip.com part of the Zulip
production distribution. The overall consequence is that digest
emails didn't work for non-zulip.com users. This fixes that issue by
moving that cron job into the zulip manifests.
[commit message details expanded by tabbott]
In 2ea0daab19, handlers were moved to
being tracked via the handlers_by_id dict, but nothing cleared this
dict, resulting in every handler object being leaked. Since a Tornado
process uses a different handler object for every request, this
resulted in a significant memory leak. We fix this by clearing the
handlers_by_id dict in the two code paths that would result in a
Tornado handler being de-allocated: the exception codepath and the
handler disconnect codepath.
Fixes#463.
Saving the organization settings form in the administration did not
work due to a trivial form name mismatch caused by following
revisions: 472898c and 58aba59.
The only places we use the architecture were for finding the
tsearch_extras and phantomjs binaries; Luke Faraone kindly uploaded
both 32-bit and 64-bit binaries for tsearch_extras 0.1.3, so with a
bit of refactoring, we can now support 32-bit.
Fixes#505.
Add call to tools/generate-fixtures in tools/test-backend before
starting the tests. Previously, test-backend could fail if called
after tools/test-js-with-casper had failed.
Fixes#501.
Previously, we used shell quoting that would result in the shell variable not
being substituted. Instead, we use `"`s that will allow for variable
substitution.
We also explicitly include `ca-certificates`, as it is needed for the install
to complete. Usually this is brought in as a `Recommends` of `wget`, but some
systems may not automatically include such dependencies.
Fixes#470.
Whenever a user became active, this triggers an immediate presence
update event (to show that user as active). The implementation for
that event (running on the browsers of all other users in the realm)
would fully rerender the presence list, which can be an expensive
operation in a large realm, just to update the status for that one
user. This fixes that case to just remove the user from the list and
then re-insert it at the appropriate index.
[Commit message expanded with more details by Tim Abbott]
Add a function email_allowed_for_realm that checks whether a user with
given email is allowed to join a given realm (either because the email
has the right domain, or because the realm is open), and use it
whenever deciding whether to allow adding a user to a realm.
This commit is not intended to change any behavior, except in one case
where the Zulip realm's domain was not being converted to lowercase.
manage_args is set to a list of arguments a few lines later in the
function, making this initialization as the empty string useless and
confusing.
Discovered using mypy.
While I believe this actually produced correct output since users are
always subscribed to streams within their realm, this code was
definitely wrong.
Discovered using the mypy type-checking tool.
It's not clear whether this will end up being net negative in value in
the long term since it's kinda hard to understand the output, but in
the short term it should prevent regressions.
If the content wasn't rendered, both rendered_content and
rendered_content_version would be None. In addition to being
confusing, in Python 3, `None < 2` is an error and this code breaks.
At present, we only do a few simple checks on the client type inside
the event system, and this saves database/memcached queries.
Note that this preserves the structure of the marshalled name in
to_dict/from_dict as client_type to avoid an unnecessary migration.
Previously, client descriptors were referenced directly from the
handler object. Once we split the Tornado process into separate queue
and connection servers, these will no longer be in the same process,
so we need to reference them by ID instead.
This commit is somewhat ugly, but its purpose is to be early
preparation for splitting Tornado into a queue server and a frontend
server, and this code belongs, by and large, in the queue server
component.
89a2765553 didn't include the database
migration corresponding to the change, which means it didn't take full
effect when it was merged.
I noticed this because `manage.py makemigrations` would generate these
migrations; that suggests a good idea for a test to add.
This solves the problem reported in #331 with needing to specify
--provider=lxc to use the LXC provider in an Ubuntu Linux environment;
additionally, it adds the LXC option needed to run LXC on Ubuntu
15.10, but not on 14.04 where that option is unavailable and would
totally break LXC.
It's possible we should just eliminate this mechanism, but this fixes
a proximal problem where the multi-line get_subscribers endpoint
description was being handled wrong.
Previously these were hardcoded in zproject/settings.py to be accessed
on localhost.
[Modified by Tim Abbott to adjust comments and fix configure-rabbitmq]
Travis CI's model of installing every version of postgres on the test
VM and then shutting all the versions other than the one requested
down seems to not work very well with doing apt upgrades. It seems
the best way to resolve this is to just uninstall the versions we
don't need.
A common issue when doing a Zulip upgrade is trying to pass
upgrade-zulip a tarball path under /root, which doesn't work because
the Zulip user doesn't have permission to read the tarball. We
could fix this by just unpacking the tarballs as root, but it seemed
like a nicer approach would be to archive the release tarballs
somewhere readable by the Zulip user (/home/zulip/archives) and unpack
them from there.
Fixes#208.
The point of the lock is to prevent two deployments happening at the
same time and racing with each other, not to prevent doing any future
deployments after an error happens (which is what the current
implementation does in practice).
Addresses part of #208.
This link was broken when we hardened the access model for user file
uploads to not work cross-realm. The right solution is just to
include the image in the codebase so it's guaranteed to exist.
Fixes#205.
Previously:
* It wouldn't raise an exception if the stream didn't exist
* It didn't correctly handle being passed a stream name
that differed in case from the stream name in the database.
The previous implementation didn't work because HomepageForm rejected
the email as not having a domain. Additionally, the logic in
accounts_register didn't work with Google auth because that code path
doesn't pass through accounts_home. Since whether there's a unique
open realm for the server is effectively a configuration property, we
can fix the bug and make the logic clearer by moving it into the
"figure out the user's realm" function.
The browser registers for events via loading the home view, not this
interface, and this functionality is available via the API-format
register route anyway.
This removes from our cache a moderate amount of totally useless alert
word data corresponding to users who don't have any alert words.
Thanks to @dbiollo for the suggestion!
Just doing the database query is more readable, and has about the same
performance as before in the case where active user dicts for the
realm are in cache (and is substantially better in the rare case that
this isn't in the cache).
Thanks to @dbiollo for the perf investigation and suggestion!
This makes it possible to use DevAuthBackend when doing
performance/scalability testing on Zulip with many thousands of users.
It's unlikely that anyone testing this backend will find it valuable
to have more than 100 login buttons on the same page, and if they do,
they can always just change this limit.
Thanks to @dbiollo for the suggestion!
This fixes a performance issue looking up UserProfile objects for
realms with a large number of users in the case that a UserProfile
object is not in the cache.
Thanks to @dbiollo for the suggestion!
We ran into a bug with the Travis CI infrastructure where it postgres
9.1 is installed on the system, and so when we'd do an apt upgrade
with a new version of 9.1, the 9.1 daemon would end up getting started
and conflict with the 9.3 daemon we were trying to run.
860cf68716 introduced calls to
notifications.redraw_title() on narrow activation. This introduced a
bug when the Zulip desktop app reloads while narrowed --
new_message_count would still be set to undefined when
narrow.activate() is called as the page (re)loads, and thus we'd call
window.bridge.updateCount(undefined), resulting in a traceback.
We fix this by just initializing it to 0, rather than using the old
default value of undefined.
This allows full-screen mode when launching from a saved app link
(mobile browser -> save link to home screen). This works on Android,
too, despite the "apple-" prefix.
Django's `manage.py runserver` prints a relatively low-information log
line for every request of the form:
[14/Dec/2015 00:43:06]"GET /static/js/message_list.js HTTP/1.0" 200 21969
This is pretty spammy, especially given that we already have our own
middleware printing a more detailed version of the same log lines:
2015-12-14 00:43:06,935 INFO 127.0.0.1 GET 200 0ms /static/js/message_list.js (unauth via ?)
Since runserver doesn't have support controlling whether these log
lines are printed, we wrap it with a small bit of code that silences
the log lines for 200/304 requests (aka the uninteresting ones).
It's needed for the tornado server. Otherwise, you get errors like
2015-12-20 09:33:55,124 ERROR Internal Server Error: /api/v1/events
Traceback (most recent call last):
File "/home/zulip/deployments/2015-12-20-13-44-47/zerver/management/commands/runtornado.py", line 209, in get_response
response = middleware_method(request)
File "/usr/lib/python2.7/dist-packages/django/middleware/common.py", line 62, in process_request
host = request.get_host()
File "/usr/lib/python2.7/dist-packages/django/http/request.py", line 101, in get_host
raise DisallowedHost(msg)
DisallowedHost: Invalid HTTP_HOST header: 'localhost:9993'. You may need to add u'localhost' to ALLOWED_HOSTS.
Like the Stream Subject lists, Private messages are now shown
when the user clicks on the "Private message" link. User can drill in
to get more than 5 conversations. Selecting PMs from the user or group
PM lists on the right sidebar also opens the list & highlights the
selected conversation.
[Edited by tabbott@mit.edu to fix some small bugs.]
Several of these rules only apply to one of Python and Javascript, and
this simplifies the logic and should make our linter code more readable.
In the process, we add support for per-rule/file pair exclusions to
handle the tab exception for codehilite.py.
By default we are placed inside a virtualenv by the .bash_profile using
/usr/bin/python forces the provisioning script to run outside of this
virtualenv.
These routes previously didn't follow our standard convention of
sending arguments in JSON format, and so broke when we started
checking the argument format in
123d51e3aa.
Fixes#333.
Apparently, previously nginx was only compressing text/html content.
This should result in a substantial savings in network traffic -- some
quick testing I did found it cut the total data transferred for
loading a logged-in zulip.com instance from 3MB to 1.2MB.
`tools/test-js-with-node cover` needs istanbul to be installed in
order to work; we might as well install it by default rather than
having it be an extra step users need to deal with.
Of course, since this is only needed in the development environment,
this could suggest we want to fork/conditionalize package.json, but I
think for now it's reasonable to just install everything we use
somewhere -- the npm list is still pretty short and we have that issue
anyway with webpack-dev-server.
If running on Django 1.8, running these plugins would die with the below. A fix
for this is to run `django.setup()` before interacting with Django.
Refs:
https://docs.djangoproject.com/en/1.8/ref/applications/#troubleshooting
```
Traceback (most recent call last):
File "/usr/lib/nagios/plugins/check_send_receive_time", line 103, in <module>
sender = get_user_profile_by_email(settings.NAGIOS_SEND_BOT)
File "/home/zulip/deployments/current/zerver/lib/cache.py", line 113, in func_with_caching
val = func(*args, **kwargs)
File "/home/zulip/deployments/current/zerver/models.py", line 1073, in get_user_profile_by_email
return UserProfile.objects.select_related().get(email__iexact=email.strip())
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 328, in get
num = len(clone)
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 144, in __len__
self._fetch_all()
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 977, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 238, in iterator
results = compiler.execute_sql()
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 829, in execute_sql
sql, params = self.as_sql()
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 378, in as_sql
extra_select, order_by, group_by = self.pre_sql_setup()
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 48, in pre_sql_setup
self.setup_query()
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 39, in setup_query
self.select, self.klass_info, self.annotation_col_map = self.get_select()
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 206, in get_select
related_klass_infos = self.get_related_selections(select)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 700, in get_related_selections
[f.name], opts, root_alias)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1471, in setup_joins
names, opts, allow_many, fail_on_missing=True)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 1372, in names_to_path
if field.is_relation and not field.related_model:
File "/usr/lib/python2.7/dist-packages/django/utils/functional.py", line 60, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File "/usr/lib/python2.7/dist-packages/django/db/models/fields/related.py", line 110, in related_model
apps.check_models_ready()
File "/usr/lib/python2.7/dist-packages/django/apps/registry.py", line 131, in check_models_ready
raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
```
Primarily this makes sure all the log files we generate are ignored.
A good follow-up project to this would be to move all the log files to
a fixed directory so that they're not creating a mess in the main
filespace.
sudo dpkg -i vagrant*.deb # in directory where you downloaded vagrant
vagrant plugin install vagrant-lxc
```
* For other Linux hosts with a kernel above 3.12, [follow the Vagrant
LXC installation
instructions](https://github.com/fgrehm/vagrant-lxc) to get Vagrant
with LXC for your platform.
* If your host is OS X or older Linux, [download VirtualBox](https://www.virtualbox.org/wiki/Downloads),
[download Vagrant](https://www.vagrantup.com/downloads.html), and install them both.
* If you're on OS X and have VMWare, it should be possible to patch
Vagrantfile to use the VMWare vagrant provider which should perform
much better than Virtualbox. Patches to do this by default if
VMWare is available are welcome!
* On Windows: You can use Vagrant and Virtualbox/VMWare on Windows
with Cygwin, similar to the Mac setup. Be sure to create your git
clone using `git clone https://github.com/zulip/zulip.git -c
core.autocrlf=false` to avoid Windows line endings being added to
files (this causes weird errors).
Once that's done, simply change to your zulip directory and run
`vagrant up` in your terminal to install the development server. This
will take a long time on the first run because Vagrant needs to
download the Ubuntu Trusty base image, but later you can run `vagrant
destroy` and then `vagrant up` again to rebuild the environment and it
will be much faster.
Once that finishes, you can run the development server as follows:
```
vagrant ssh -- -L9991:localhost:9991
# Now inside the container
cd /srv/zulip
source /srv/zulip-venv/bin/activate
./tools/run-dev.py --interface=''
```
To get shell access to the virtual machine running the server to run
lint, management commands, etc., use `vagrant ssh`.
(A small note on tools/run-dev.py: the `--interface=''` option will make
the development server listen on all network interfaces. While this
is correct for the Vagrant guest sitting behind a NAT, you probably
don't want to use that option when using run-dev.py in other environments).
At this point you should [read about using the development environment](https://github.com/zulip/zulip/blob/master/README.dev.md#using-the-development-environment).
Using provision.py without Vagrant
----------------------------------
If you'd like to install a Zulip development environment on a server
that's already running Ubuntu 14.04 Trusty, you can do that by just
running:
```
sudo apt-get update
sudo apt-get install -y python-pbs
python /srv/zulip/provision.py
cd /srv/zulip
source /srv/zulip-venv/bin/activate
./tools/run-dev.py
```
Note that there is no supported uninstallation process without Vagrant
(with Vagrant, you can just do `vagrant destroy` to clean up the
development environment).
By hand
-------
If you really want to install everything by hand, the below
instructions should work.
Install the following non-Python dependencies:
* libffi-dev — needed for some Python extensions
* postgresql 9.1 or later — our database (also install development headers)
* memcached (and headers)
* rabbitmq-server
* libldap2-dev
* python-dev
* redis-server — rate limiting
* tsearch-extras — better text search
* libfreetype6-dev - needed before you pip install Pillow to properly generate emoji PNGs
./tools/test-js-with-node # Runs all node tests but is very fast
```
The above setup instructions include the first-time setup of test
databases, but you may need to rebuild the test database occasionally
if you're working on new database migrations. To do this, run:
```
./tools/postgres-init-test-db
./tools/do-destroy-rebuild-test-database
```
Possible testing issues
=======================
- When running the test suite, if you get an error like this:
```
sqlalchemy.exc.ProgrammingError: (ProgrammingError) function ts_match_locs_array(unknown, text, tsquery) does not exist
LINE 2: ...ECT message_id, flags, subject, rendered_content, ts_match_l...
^
```
… then you need to install tsearch-extras, described
above. Afterwards, re-run the `init*-db` and the
`do-destroy-rebuild*-database` scripts.
- When building the development environment using Vagrant and the LXC provider, if you encounter permissions errors, you may need to `chown -R 1000:$(whoami) /path/to/zulip` on the host before running `vagrant up` in order to ensure that the synced directory has the correct owner during provision. This issue will arise if you run `id username` on the host where `username` is the user running Vagrant and the output is anything but 1000.
This seems to be caused by Vagrant behavior; more information can be found here https://github.com/fgrehm/vagrant-lxc/wiki/FAQ#help-my-shared-folders-have-the-wrong-owner
Browsing this list can be a great way to find feature ideas to
implement that other Zulip users are excited about.
* [2016 roadmap milestone](http://zulip.readthedocs.io/en/latest/roadmap.html): The
projects that are [priorities for the Zulip project](https://zulip.readthedocs.io/en/latest/roadmap.html). These are great projects if you're looking to make an impact.
If you're excited about helping with an open issue, just post on the
conversation thread that you're working on it. You're encouraged to
@@ -123,11 +201,10 @@ looking at the new feature tutorial and coding style guidelines on
ReadTheDocs.
Feedback on how to make this development process more efficient, fun,
and friendly to new contributors is very welcome! Just shoot an email
and friendly to new contributors is very welcome! Just send an email
Zulip offers a wide range of options for how to install the
development environment:
* [Detailed tutorial for Vagrant development environment](dev-env-first-time-contributors.html). Recommended for first-time contributors.
* [Brief installation instructions for Vagrant development environment](brief-install-vagrant-dev.html)
* [Installing on Ubuntu 14.04 Trusty or 16.04 Xenial directly](install-ubuntu-without-vagrant-dev.html) (convenient but more work to maintain/uninstall).
* [Installing manually on other UNIX platforms](install-generic-unix-dev.html)
This page documents the Zulip directory structure, where to find
things, and how to decide where to put a file.
You may also find the [new application feature
tutorial](new-feature-tutorial.html) helpful for understanding the
flow through these files.
### Core Python files
Zulip uses the [Django web
framework](https://docs.djangoproject.com/en/1.8/), so a lot of these
paths will be familiar to Django developers.
*`zproject/urls.py` Main [Django routes file](https://docs.djangoproject.com/en/1.8/topics/http/urls/). Defines which URLs are handled by which view functions or templates.
*`zerver/models.py` Main [Django models](https://docs.djangoproject.com/en/1.8/topics/db/models/) file. Defines Zulip's database tables.
*`zerver/lib/actions.py` Most code doing writes to user-facing database tables.
*`zerver/views/*.py` Most [Django views](https://docs.djangoproject.com/en/1.8/topics/http/views/).
*`zerver/views/webhooks/` Webhook views for [Zulip integrations](integration-guide.html).
You can consult the repository's `.gitattributes` file to see exactly
which components are excluded from production releases (release
tarballs are generated using `tools/build-release-tarball`).
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
