58bf2a7935
Tornado requests try hard to not make SQL queries -- and if they're necessary, to minimize the number of them. Specifically, both session objects and user objects are cached in memcached, and we expect that both of them will have been filled there by Django before any requests are made to Tornado. In the event that memcached is flushed, or data is otherwise evicted, we perform two database queries -- one for the session, and one for the user. However, the *width* of the latter query has grown significantly over time, as the Realm object grew more fields, and recently with the addition of role groups, which require multiple joins each. This leads to a query which is over 12k of text long, and results in 319 columns. In the event of a memcached flush, this can result in a *significant* amount of SQL traffic, as nearly every active Tornado request will make that query. We do not wish to narrow the default query for Django; we instead tag the request in the REST wrapper, and use that to use a much narrower user cache entry. That narrower cache entry is filled before the queue is created in Django; we also use it to explicitly set the log data, so the second "half" of the continued Tornado request does not need to fetch any user data either when writing its log line. Because they use different cache keys, this only affects the session-based `/json/events` endpoint, which caches by user-id; the `/api/v1/events` endpoint, which uses an API-key cache, keeps its wide user object. The former is 50% of the total request volume, whereas the latter is only 2%, so adding an additional cache for it is unnecessary complexity.
Zulip overview
Zulip is an open-source team collaboration tool with unique topic-based threading that combines the best of email and chat to make remote work productive and delightful. Fortune 500 companies, leading open source projects, and thousands of other organizations use Zulip every day. Zulip is the only modern team chat app that is designed for both live and asynchronous conversations.
Zulip is built by a distributed community of developers from all around the world, with 74+ people who have each contributed 100+ commits. With over 1000 contributors merging over 500 commits a month, Zulip is the largest and fastest growing open source team chat project.
Come find us on the development community chat!
Getting started
-
Contributing code. Check out our guide for new contributors to get started. We have invested in making Zulip’s code highly readable, thoughtfully tested, and easy to modify. Beyond that, we have written an extraordinary 150K words of documentation for Zulip contributors.
-
Contributing non-code. Report an issue, translate Zulip into your language, or give us feedback. We'd love to hear from you, whether you've been using Zulip for years, or are just trying it out for the first time.
-
Checking Zulip out. The best way to see Zulip in action is to drop by the Zulip community server. We also recommend reading about Zulip's unique approach to organizing conversations.
-
Running a Zulip server. Self-host Zulip directly on Ubuntu or Debian Linux, in Docker, or with prebuilt images for Digital Ocean and Render. Learn more about self-hosting Zulip.
-
Using Zulip without setting up a server. Learn about Zulip Cloud hosting options. Zulip sponsors free Zulip Cloud Standard for hundreds of worthy organizations, including fellow open-source projects.
-
Participating in outreach programs like Google Summer of Code and Outreachy.
-
Supporting Zulip. Advocate for your organization to use Zulip, become a sponsor, write a review in the mobile app stores, or help others find Zulip.
You may also be interested in reading our blog, and following us on Twitter and LinkedIn.
Zulip is distributed under the Apache 2.0 license.