Commit Graph

382 Commits

Author SHA1 Message Date
Brainrecursion
30eaed0378 saml: Add option to restrict subdomain access based on SAML attributes.
Adds the ability to set a SAML attribute which contains a
list of subdomains the user is allowed to access. This allows a Zulip
server with multiple organizations to filter using SAML attributes
which organization each user can access.

Cleaned up and adapted by Mateusz Mandera to fit our conventions and
needs more.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-23 17:14:31 -07:00
Anders Kaseorg
f6c73ed45a auth: Avoid unchecked casts.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 13:30:09 -07:00
Anders Kaseorg
7e9db327b3 request: Improve validator type so mypy can check it against REQ.
Old: a validator returns None on success and returns an error string
on error.

New: a validator returns the validated value on success and raises
ValidationError on error.

This allows mypy to catch mismatches between the annotated type of a
REQ parameter and the type that the validator actually validates.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-20 22:29:15 -07:00
Dinesh
0445311430 auth: Make apple log in and sign up buttons consistent with others. 2020-06-18 13:06:10 -07:00
Anders Kaseorg
1ed2d9b4a0 logging: Use logging.exception and exc_info for unexpected exceptions.
logging.exception() and logging.debug(exc_info=True),
etc. automatically include a traceback.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-14 23:27:22 -07:00
Dinesh
d308c12ae2 auth: Add native flow support for Apple authentication.
Overrides some of internal functions of python-social-auth
to handle native flow.

Credits to Mateusz Mandera for the overridden functions.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-14 16:20:12 -07:00
Anders Kaseorg
6358339581 backends: Convert ExternalAuthDataDict to Python 3.6 style.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-13 15:37:25 -07:00
Anders Kaseorg
365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg
69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Anders Kaseorg
67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg
6480deaf27 python: Convert more "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, with more
restrictions patched out.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 14:48:09 -07:00
Anders Kaseorg
826ca3bfdf settings: Move LDAP_DEACTIVATE_NON_MATCHING_USERS default to default_settings.
Tweaked by tabbott to fix an incorrect translation to ONLY_SSO.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-09 22:27:55 -07:00
Dinesh
dc90d54b08 auth: Add Sign in with Apple support.
This implementation overrides some of PSA's internal backend
functions to handle `state` value with redis as the standard
way doesn't work because of apple sending required details
in the form of POST request.

Includes a mixin test class that'll be useful for testing
Native auth flow.

Thanks to Mateusz Mandera for the idea of using redis and
other important work on this.

Documentation rewritten by tabbott.

Co-authored-by: Mateusz Mandera <mateusz.mandera@zulip.com>
2020-06-09 17:29:35 -07:00
Dinesh
04671f40ae auth: Add auth_name attribute for non external auth backends.
This commit adds `name` attribute for the backends that do not
have them.

This is just a kind of prep commit in case if we want to use
`self.logger.xxxx()` in the future which is dependent on the
`name` attribute. But right now these logging calls aren't used
anywhere in those backends.
2020-06-08 17:42:07 -07:00
Dinesh
e22ca9483e logging: Log exception name also in auth_complete.
`HTTPError` has empty string for `str(HTTPError())`. Logging it
as it is would not be much helpful. So, this commits adds code
to log the name of error also.
2020-06-08 17:42:07 -07:00
Dinesh
d30f11888a logging: Set up a different logger for each backend.
Adds a top-level logger in `settings.LOGGING` `zulip.auth`
with the default handlers `DEFAULT_ZULIP_HANDLERS` and
an extra hanlder that writes to `/var/log/zulip/auth.log`.

Each auth backend uses it's own logger, `self.logger` which
is in form 'zulip.auth.<backend name>'.

This way it's clear which auth backend generated the log
and is easier to look for all authentication logs in one file.

Besides the above mentioned changes, `name` attribute is added to
`ZulipAuthMixin` so that these logging kind of calls wouldn't raise
any issues when logging is tried in a class without `name` attribute.

Also in the tests we use a new way to check if logger calls are made
i.e. we use `assertLogs` to test if something is logged.

Thanks to Mateusz Mandera for the idea of having a seperate logger
for auth backends and suggestion of using `assertLogs`.
2020-06-08 17:42:07 -07:00
Anders Kaseorg
8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Anders Kaseorg
7a53da7526 capitalization: Fix OAuth capitalization.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-03 16:39:12 -07:00
Sahil Batra
77d4be56a4 users: Modify do_create_user and create_user to accept role.
We change do_create_user and create_user to accept
role as a parameter instead of 'is_realm_admin' and 'is_guest'.
These changes are done to minimize data conversions between
role and boolean fields.
2020-06-02 16:11:36 -07:00
Mateusz Mandera
3e7fc17788 auth: Delegate RemoteUser SSO to browser when using the desktop app. 2020-06-02 13:00:17 -07:00
Dinesh
9f2fb3a48a auth: Move standard_relay_params of SAMLAuthBackend to SocialAuthMixin.
Earlier this `standard_relay_params` was used only for SAML auth,
now "Sign in with Apple" also requires this to store those params
in session for reuse. So, this acts as a prep commit for "Sign in
with Apple" auth support.
2020-05-29 16:02:53 -07:00
Vishnu KS
8784539d53 free trial: Send users to /upgrade after realm creation. 2020-05-26 17:01:32 -07:00
Anders Kaseorg
cf923b49d3 python: Remove extra pass statements with autoflake.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Anders Kaseorg
8bcdf4ca97 python: Convert TypedDict declarations to Python 3.6 style.
A subset of the diff generated by pyupgrade --py36-plus
--keep-percent-format.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:43:40 -07:00
Mateusz Mandera
b66dc9de50 saml: Support IdP-initiated SSO. 2020-05-25 16:09:30 -07:00
Dinesh
288921d425 auth: Log when a user tries to login with deactivated account.
Helps to see if users are often trying to login with deactived
accounts.
A use case: Trackdown whether any deactivated bot users are still
trying to access the API.

This implementation adds a new key `inactive_user_id`
to `return_data` in the function `is_user_active` which
check if a `user_profile` is active. This reduces the effort
of getting `user_id` just before logging.

Modified tests for line coverage.
2020-05-24 17:27:19 -07:00
Mateusz Mandera
dac4a7a70b saml: Figure out the idp from SAMLResponse.
Instead of plumbing the idp to /complete/saml/ through redis, it's much
more natural to just figure it out from the SAMLResponse, because the
information is there.
This is also a preparatory step for adding IdP-initiated sign in, for
which it is important for /complete/saml/ to be able to figure out which
IdP the request is coming from.
2020-05-24 16:40:28 -07:00
Mateusz Mandera
c74f8363e2 saml: Gracefully handle bad SAMLResponses. 2020-05-24 16:40:28 -07:00
Mateusz Mandera
2f5fd272aa auth: Gracefully handle bad http responses from IdP in social auth.
If the IdP authentication API is flaky for some reason, it can return
bad http responses, which will raise HTTPError inside
python-social-auth. We don't want to generate a traceback
in those cases, but simply log the exception and fail gracefully.
2020-05-20 09:30:56 -07:00
Anders Kaseorg
7271fb68aa logging: Pass format arguments to unconventionally-named loggers too.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 16:37:07 -07:00
Dinesh
a3329f288c auth: Modify filter_usable_emails to only exclude noreply github emails.
Instead of having to filter `@noreply.github.com` emails in
`get_unverified_emails`, it's good to make `filter_usable_emails`
just filter `@noreply.github.com` and handle verified/unverified
part in their respective functions because of `@noreply.github.com`
exception being a fiddly special-case detail.
Also renamed `filter_usable_emails` to `get_usable_email_objects`
as a line that gets all associated github emails is removed in
`get_verified_emails` and `get_unverified_emails` and added to
`filter_usable_emails`. The name `filter_usable_emails` suggests
that it just filters given emails, whereas here it's getting all
associated email objects and returning usable emails.
2020-05-02 14:30:31 -07:00
Dinesh
5c1fe776c3 auth: Extend the template for "choose email" in GitHub auth flow.
This commit extends the template for "choose email" to mention for
users who have unverified emails that they need to verify them before
using them for Zulip authentication.

Also modified `social_auth_test_finish` to assert if all emails
are present in "choose email" screen as we need unverified emails
to be shown to user and verified emails to login/signup.

Fixes #12638 as this was the last task for that issue.
2020-05-02 14:30:31 -07:00
Dinesh
4a07a6def7 auth: Separate code to get all emails from get_verified_emails.
This separates the part of code that gets all the emails associated
to GitHub from `get_verified_emails` in `GitHubAuthBackend`.
Improves readability of code and acts as a preparatory commit for
extending the template for "choose email" in GitHub auth flow to also
list any unverified emails that have an associated Zulip account in
the organization.
2020-05-02 13:40:57 -07:00
Anders Kaseorg
bdc365d0fe logging: Pass format arguments to logging.
https://docs.python.org/3/howto/logging.html#optimization

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-02 10:18:02 -07:00
Mateusz Mandera
f1ec02b40a auth: Add ExternalAuthResult to manage data in authentication flows.
This new type eliminates a bunch of messy code that previously
involved passing around long lists of mixed positional keyword and
arguments, instead using a consistent data object for communicating
about the state of an external authentication (constructed in
backends.py).

The result is a significantly more readable interface between
zproject/backends.py and zerver/views/auth.py, though likely more
could be done.

This has the side effect of renaming fields for internally passed
structures from name->full_name, next->redirect_to; this results in
most of the test codebase changes.

Modified by tabbott to add comments and collaboratively rewrite the
initialization logic.
2020-04-28 22:19:02 -07:00
Anders Kaseorg
fead14951c python: Convert assignment type annotations to Python 3.6 style.
This commit was split by tabbott; this piece covers the vast majority
of files in Zulip, but excludes scripts/, tools/, and puppet/ to help
ensure we at least show the right error messages for Xenial systems.

We can likely further refine the remaining pieces with some testing.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-    invoiced_through: Optional[LicenseLedger] = models.ForeignKey(
+    invoiced_through: Optional["LicenseLedger"] = models.ForeignKey(

-_apns_client: Optional[APNsClient] = None
+_apns_client: Optional["APNsClient"] = None

-    notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
-    signup_notifications_stream: Optional[Stream] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)
+    signup_notifications_stream: Optional["Stream"] = models.ForeignKey('Stream', related_name='+', null=True, blank=True, on_delete=CASCADE)

-    author: Optional[UserProfile] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)
+    author: Optional["UserProfile"] = models.ForeignKey('UserProfile', blank=True, null=True, on_delete=CASCADE)

-    bot_owner: Optional[UserProfile] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)
+    bot_owner: Optional["UserProfile"] = models.ForeignKey('self', null=True, on_delete=models.SET_NULL)

-    default_sending_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
-    default_events_register_stream: Optional[Stream] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_sending_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)
+    default_events_register_stream: Optional["Stream"] = models.ForeignKey('zerver.Stream', null=True, related_name='+', on_delete=CASCADE)

-descriptors_by_handler_id: Dict[int, ClientDescriptor] = {}
+descriptors_by_handler_id: Dict[int, "ClientDescriptor"] = {}

-worker_classes: Dict[str, Type[QueueProcessingWorker]] = {}
-queues: Dict[str, Dict[str, Type[QueueProcessingWorker]]] = {}
+worker_classes: Dict[str, Type["QueueProcessingWorker"]] = {}
+queues: Dict[str, Dict[str, Type["QueueProcessingWorker"]]] = {}

-AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional[LDAPSearch] = None
+AUTH_LDAP_REVERSE_EMAIL_SEARCH: Optional["LDAPSearch"] = None

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 11:02:32 -07:00
Anders Kaseorg
f8c95cda51 mypy: Add specific codes to type: ignore annotations.
https://mypy.readthedocs.io/en/stable/error_codes.html

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg
029bfb9fee mypy: Remove unnecessary type: ignore annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-22 10:46:33 -07:00
Anders Kaseorg
088f7ee5d6 python: Convert type checks to isinstance checks.
Generated by autopep8 --aggressive, with the setup.cfg configuration
from #14532.  In general, an isinstance check may not be equivalent to
a type check because it includes subtypes; however, that’s usually
what you want.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Anders Kaseorg
1cf63eb5bf python: Whitespace fixes from autopep8.
Generated by autopep8, with the setup.cfg configuration from #14532.
I’m not sure why pycodestyle didn’t already flag these.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-21 17:58:09 -07:00
Mateusz Mandera
62c0ab3f9d saml: Change which IdPs are returned to get_external_method_dicts.
If queried without a realm, get_external_method_dicts should only
have IdPs that can be used on all realms.
2020-04-21 13:49:34 -07:00
Mateusz Mandera
7ed3c3f9f0 saml: Add setting to require limit_to_subdomains on configured IdPs.
If SAML_REQUIRE_LIMIT_TO_SUBDOMAINS is enabled, the configured IdPs will
be validated and cleaned up when the saml backend is initialized.
settings.py would be a tempting and more natural place to do this
perhaps, but in settings.py we don't do logging and we wouldn't be able
to write a test for it.
2020-04-16 17:04:12 -07:00
Mateusz Mandera
143db68422 saml: Implement limiting of IdP to specified realms.
Through the limit_to_subdomains setting on IdP dicts it's now possible
to limit the IdP to only allow authenticating to the specified realms.

Fixes #13340.
2020-04-16 17:04:08 -07:00
Anders Kaseorg
c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Mateusz Mandera
e86cfbdbd7 rate_limiter: Store data in request._ratelimits_applied list.
The information used to be stored in a request._ratelimit dict, but
there's no need for that, and a list is a simpler structure, so this
allows us to simplify the plumbing somewhat.
2020-04-08 10:29:18 -07:00
Stefan Weil
d2fa058cc1 text: Fix some typos (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-27 17:25:56 -07:00
Mateusz Mandera
8069133f88 rate_limit: Remove __str__ methods of RateLimitedObjects.
These were clunky from the start and are no longer used, as keys are now
used directly for logging purposes.
2020-03-22 18:42:35 -07:00
Mateusz Mandera
4e9f77a6c4 rate_limit: Adjust keys() of some RateLimitedObjects.
type().__name__ is sufficient, and much readable than type(), so it's
better to use the former for keys.
We also make the classes consistent in forming the keys in the format
type(self).__name__:identifier and adjust logger.warning and statsd to
take advantage of that and simply log the key().
2020-03-22 18:42:35 -07:00
Mateusz Mandera
2c6b1fd575 rate_limit: Rename key_fragment() method to key(). 2020-03-22 18:42:35 -07:00
Mateusz Mandera
9c9f8100e7 rate_limit: Add the concept of RateLimiterBackend.
This will allow easily swapping and using various implementations of
rate-limiting, and separate the implementation logic from
RateLimitedObjects.
2020-03-22 18:42:35 -07:00