shared: Bump version to 0.0.11.

These are based on my reading of the poll_data.js implementation.

.mailmap

@@ -12,6 +12,8 @@
 #     # shows raw names/emails, filtered by mapped name:
 #   $ git log --format='%an %ae' --author=$NAME | uniq -c
 
+Adam Benesh <Adam.Benesh@gmail.com> <Adam-Daniel.Benesh@t-systems.com>
+Adam Benesh <Adam.Benesh@gmail.com>
 Alex Vandiver <alexmv@zulip.com> <alex@chmrr.net>
 Alex Vandiver <alexmv@zulip.com> <github@chmrr.net>
 Allen Rabinovich <allenrabinovich@yahoo.com> <allenr@humbughq.com>
@@ -20,6 +22,9 @@ Alya Abbott <alya@zulip.com> <2090066+alya@users.noreply.github.com>
 Aman Agrawal <amanagr@zulip.com> <f2016561@pilani.bits-pilani.ac.in>
 Anders Kaseorg <anders@zulip.com> <anders@zulipchat.com>
 Anders Kaseorg <anders@zulip.com> <andersk@mit.edu>
+Aryan Shridhar <aryanshridhar7@gmail.com> <53977614+aryanshridhar@users.noreply.github.com>
+Aryan Shridhar <aryanshridhar7@gmail.com>
+Ashwat Kumar Singh <ashwat.kumarsingh.met20@itbhu.ac.in>
 Austin Riba <austin@zulip.com> <austin@m51.io>
 BIKI DAS <bikid475@gmail.com>
 Brock Whittaker <brock@zulipchat.com> <bjwhitta@asu.edu>
@@ -37,6 +42,7 @@ Jeff Arnold <jbarnold@gmail.com> <jbarnold@humbughq.com>
 Jeff Arnold <jbarnold@gmail.com> <jbarnold@zulip.com>
 Jessica McKellar <jesstess@mit.edu> <jesstess@humbughq.com>
 Jessica McKellar <jesstess@mit.edu> <jesstess@zulip.com>
+Julia Bichler <julia.bichler@tum.de> <74348920+juliaBichler01@users.noreply.github.com>
 Kevin Mehall <km@kevinmehall.net> <kevin@humbughq.com>
 Kevin Mehall <km@kevinmehall.net> <kevin@zulip.com>
 Kevin Scott <kevin.scott.98@gmail.com>
@@ -45,11 +51,13 @@ Mateusz Mandera <mateusz.mandera@zulip.com> <mateusz.mandera@protonmail.com>
 m-e-l-u-h-a-n <purushottam.tiwari.cd.cse19@itbhu.ac.in>
 Palash Raghuwanshi <singhpalash0@gmail.com>
 Parth <mittalparth22@gmail.com>
+Priyam Seth <sethpriyam1@gmail.com> <b19188@students.iitmandi.ac.in>
 Ray Kraesig <rkraesig@zulip.com> <rkraesig@zulipchat.com>
+Reid Barton <rwbarton@gmail.com> <rwbarton@humbughq.com>
 Rishi Gupta <rishig@zulipchat.com> <rishig+git@mit.edu>
 Rishi Gupta <rishig@zulipchat.com> <rishig@kandralabs.com>
 Rishi Gupta <rishig@zulipchat.com> <rishig@users.noreply.github.com>
-Reid Barton <rwbarton@gmail.com> <rwbarton@humbughq.com>
+Rishabh Maheshwari <b20063@students.iitmandi.ac.in>
 Sayam Samal <samal.sayam@gmail.com>
 Scott Feeney <scott@oceanbase.org> <scott@humbughq.com>
 Scott Feeney <scott@oceanbase.org> <scott@zulip.com>
@@ -71,3 +79,5 @@ Sahil Batra <sahil@zulip.com> <sahilbatra839@gmail.com>
 Yash RE <33805964+YashRE42@users.noreply.github.com> <YashRE42@github.com>
 Yash RE <33805964+YashRE42@users.noreply.github.com>
 Yogesh Sirsat <yogeshsirsat56@gmail.com>
+Zeeshan Equbal <equbalzeeshan@gmail.com> <54993043+zee-bit@users.noreply.github.com>
+Zeeshan Equbal <equbalzeeshan@gmail.com>

docs/overview/changelog.md

@@ -1,18 +1,44 @@
 # Version history
 
-This page contains the release history for the Zulip 5.x stable
-release series. See the [current Zulip changelog][latest-changelog]
-for newer release series, or the [commit log][commit-log] for an
-up-to-date list of raw changes.
+This page the release history for the Zulip server. See also the
+[Zulip release lifecycle](../overview/release-lifecycle.md).
 
-## Zulip 5.x series
+## Zulip 6.x series
 
-### 5.0 -- 2022-03-29
+### 6.0 -- unreleased
 
 This section is an incomplete draft of the release notes for the next
 major release, and is only updated occasionally. See the [commit
 log][commit-log] for an up-to-date list of raw changes.
 
+#### Upgrade notes for 6.0
+
+- None yet.
+
+## Zulip 5.x series
+
+### 5.1 -- 2022-04-01
+
+- Fixed upgrade bug where preexisting animated emoji would still
+  always animate in statuses.
+- Improved check that prevents servers from accidentally downgrading,
+  to not block upgrading servers that originally installed Zulip
+  Server prior to mid-2017.
+- Fixed email address de-duplication in Slack imports.
+- Prevented an extraneous scrollbar when a notification banner was
+  present across the top.
+- Fixed installation in LXC containers, which failed due to `chrony`
+  not being runnable there.
+- Prevented a "push notifications not configured" warning from
+  appearing in the new user default settings panel even when push
+  notifications were configured.
+- Fixed a bug which, in uncommon configurations, would prevent Tornado
+  from being restarted during upgrades; users would be able to log in,
+  but would immediately be logged out.
+- Updated translations.
+
+### 5.0 -- 2022-03-29
+
 #### Highlights
 
 - New [resolve topic](https://zulip.com/help/resolve-a-topic) feature
@@ -1566,7 +1592,7 @@ Zulip installations; it has minimal changes for existing servers.
   disruption by running this migration first, before beginning the
   user-facing downtime. However, if you'd like to watch the downtime
   phase of the upgrade closely, we recommend
-  [running them first manually](https://zulip.readthedocs.io/en/1.9.0/production/expensive-migrations.html)
+  running them first manually
   as well as the usual trick of doing an apt upgrade first.
 
 #### Full feature changelog
@@ -1955,7 +1981,7 @@ running a version from before 1.7 should upgrade directly to 1.7.1.
   minimizes disruption by running these first, before beginning the
   user-facing downtime. However, if you'd like to watch the downtime
   phase of the upgrade closely, we recommend
-  [running them first manually](https://zulip.readthedocs.io/en/1.9.0/production/expensive-migrations.html)
+  running them first manually
   as well as the usual trick of doing an apt upgrade first.
 
 - We've removed support for an uncommon legacy deployment model where
@@ -2580,15 +2606,17 @@ running a version from before 1.7 should upgrade directly to 1.7.1.
 This section links to the upgrade notes from past releases, so you can
 easily read them all when upgrading across multiple releases.
 
-- [Upgrade notes for 5.0](#upgrade-notes-for-50)
-- [Upgrade notes for 4.0](#upgrade-notes-for-40)
-- [Upgrade notes for 3.0](#upgrade-notes-for-30)
-- [Upgrade notes for 2.1.5](#upgrade-notes-for-215)
-- [Upgrade notes for 2.1.0](#upgrade-notes-for-210)
-- [Upgrade notes for 2.0.0](#upgrade-notes-for-200)
-- [Upgrade notes for 1.9.0](#upgrade-notes-for-190)
-- [Upgrade notes for 1.8.0](#upgrade-notes-for-180)
-- [Upgrade notes for 1.7.0](#upgrade-notes-for-170)
+- [Draft upgrade notes for 6.0](#upgrade-notes-for-60)
+
+* [Upgrade notes for 5.0](#upgrade-notes-for-50)
+* [Upgrade notes for 4.0](#upgrade-notes-for-40)
+* [Upgrade notes for 3.0](#upgrade-notes-for-30)
+* [Upgrade notes for 2.1.5](#upgrade-notes-for-215)
+* [Upgrade notes for 2.1.0](#upgrade-notes-for-210)
+* [Upgrade notes for 2.0.0](#upgrade-notes-for-200)
+* [Upgrade notes for 1.9.0](#upgrade-notes-for-190)
+* [Upgrade notes for 1.8.0](#upgrade-notes-for-180)
+* [Upgrade notes for 1.7.0](#upgrade-notes-for-170)
 
 [docker-zulip]: https://github.com/zulip/docker-zulip
 [commit-log]: https://github.com/zulip/zulip/commits/main

docs/production/install.md

@@ -15,7 +15,7 @@ Zulip's
   you can create a test Zulip Cloud organization at <https://zulip.com/new>.
 
 - If you are deciding between self-hosting Zulip and signing up for [Zulip Cloud](https://zulip.com/plans/),
-  our [self-hosing overview](https://zulip.com/self-hosting/) and [guide to
+  our [self-hosting overview](https://zulip.com/self-hosting/) and [guide to
   choosing between Zulip Cloud and
   self-hosting](https://zulip.com/help/getting-your-organization-started-with-zulip#choosing-between-zulip-cloud-and-self-hosting)
   are great places to start.

docs/translating/finnish.md

@@ -0,0 +1,140 @@
+# Finnish translation style guide
+
+## Guidelines
+
+Tervetuloa!
+
+Before you start, take a look these instructions we have gathered here
+for you to help on your translation journey.
+
+### Word order
+
+Consider translating the same thing with the easiest Finnish possible.
+It's not mandatory to follow the English text word by word, as long as
+the message is clear.
+
+Eg.
+
+- Click the button below to create the organization and register your
+  account. -> Luo organisaatio ja rekisteröi tilisi napsauttamalla
+  alla olevaa painiketta.
+
+- Sent! Your message is outside your current narrow. -> Lähetetty!
+  Viesti on nykyisen näkymäsi ulkopuolella.
+
+### Grammatical case (Sijamuodot)
+
+Translate using the UI to be sure what is the correct grammatical
+case. Basic form of a word might not always be suitable for the
+purpose.
+
+Eg.
+
+- Topics marked as resolved -> Ratkaistut aiheet (versus Aiheet, jotka on merkitty ratkaistuiksi)
+- View Shortcuts -> Katsel**un** pikanäppäimet
+
+### Loan word (Lainasanat)
+
+Even though it's common to use words formed directly from English, try
+to consider also users without IT background, people that don't speak
+English and accessibility. User interface should contain these with
+minimum amount, but for technical error messages could be preferred.
+There are some amount of software related words that are widely used
+as loan words.
+
+See section [Terms](#terms) for more details.
+
+### **_Please_**, in error messages
+
+As it might appeal to use correspondence _Ole hyvä ja_, it's not
+commonly used in Finnish. We are strict and used to more direct
+messaging. Let's not translate _please_ and use instruction format
+only. No Finn is going to be offended by this.
+
+Eg.
+
+- Please enter at most 10 emails. -> Lisää korkeintaan 10 sähköpostia.
+
+But
+
+- Yes, please! -> Kyllä, kiitos!
+
+### Zulip word inflection
+
+- in/from Zulip - **Zulipissa** / **Zulipista** / **Zulipin**
+- Zulip organization - **Zulip-organisaatio**
+- Zulip app - **Zulip-sovellus**
+
+### Your -expression
+
+Finnish language has _form of ownership_ so there shouldn't be need to
+thanslate _your_ as _sinun_ but rather use _si_ ending. It might be
+considered to leave out as well.
+
+Eg.
+
+- Your organization -> Organisaatio**si**
+- Your account -> Tili**si**
+
+But
+
+- You do have active accounts in the following organization(s). ->
+  Sinulla ei ole aktiivista tiliä seuraavissa organisaatio(i)ssa.
+
+### Comma
+
+Use commas in whole sentences where it is required. You can use these instructions as help.
+[Kotimaisten kielten keskus - pilkku.](http://www.kielitoimistonohjepankki.fi/haku/pilkku/ohje/86)
+
+## Terms
+
+- Administrator - **Järjestelmänvalvoja**
+- App - **Sovellus**
+- Authorization - **Valtuus**
+- Avatar - **Avatar**
+- Beta - **Beta**
+- Change - **Muuta**
+- Cheat sheet - **Lunttilappu**
+- Click - **Napsauta**
+- Configure - **Määritä**
+- Deactivate - **Poista käytöstä**
+- Domain - **Verkkotunnus**
+- Export - **Poiminta**
+- Filter - **Suodata**
+- Full member - **Täysivaltainen jäsen**
+- Host - **Isäntä**
+- Help Center - **Tukikeskus**
+- ID - **Tunnus**
+- Integraatio - **Integraatio**
+- Interactive - **Interaktiivinen**
+- Invalid - **Virheellinen**
+- Moderator - **Moderaattori**
+- Mute - **Mykistää**
+- Narrow - **Rajaa hakua**
+- Notification - **Ilmoitus**
+- Topic - **Aihe**
+- Organization - **Organisaatio**
+- Permission - **Lupa**
+- Pin - **Kiinnitä**
+- Picker - **Valitsin**
+- Plan - **Tilaus**
+- PM (private messages) - **YV (yksityisviesti)** - Short version is needed in mobile.
+- Reset - **Nollata**
+- Save - **Tallenna**
+- Stream - **Kanava**
+- Subscriber - **Tilaaja**
+- Subscription - **Tilaus**
+- Subscribe a stream - **Tilaa kanava**
+- Subdomain - **Aliverkkotunnus**
+- Shortcuts - **Pikanäppäimet**
+- Unsubscripe - **Peru tilaus**
+- Unsupported - **Ei-tuettu**
+- Unresolve - **Merkitse ratkaisemattomaksi**
+- Webhook - **Webhook**
+- Whoops - **Hupsista**
+- Widget - **Widgetti**
+
+## Other
+
+Some translations can be tricky, so please don't hesitate to ask the
+community or to contribute to this guide! Thanks for your effort!

docs/translating/index.md

@@ -8,6 +8,7 @@ maxdepth: 3
 translating
 internationalization
 chinese
+finnish
 french
 german
 hindi

docs/translating/translating.md

@@ -165,6 +165,7 @@ translate words like "stream" to), with reasoning, so that future
 translators can understand and preserve those decisions:
 
 - [Chinese](chinese.md)
+- [Finnish](finnish.md)
 - [French](french.md)
 - [German](german.md)
 - [Hindi](hindi.md)

frontend_tests/node_tests/composebox_typeahead.js

@@ -39,10 +39,10 @@ set_global("setTimeout", (f, time) => {
 });
 set_global("document", "document-stub");
 
-const emoji = zrequire("../shared/js/emoji");
 const typeahead = zrequire("../shared/js/typeahead");
 const compose_state = zrequire("compose_state");
 const compose_validate = zrequire("compose_validate");
+const emoji = zrequire("emoji");
 const typeahead_helper = zrequire("typeahead_helper");
 const muted_users = zrequire("muted_users");
 const people = zrequire("people");

frontend_tests/node_tests/dispatch.js

@@ -87,6 +87,7 @@ page_params.realm_description = "already set description";
 
 // For data-oriented modules, just use them, don't stub them.
 const alert_words = zrequire("alert_words");
+const emoji = zrequire("emoji");
 const stream_topic_history = zrequire("stream_topic_history");
 const stream_list = zrequire("stream_list");
 const message_helper = zrequire("message_helper");
@@ -96,8 +97,6 @@ const starred_messages = zrequire("starred_messages");
 const user_status = zrequire("user_status");
 const compose_pm_pill = zrequire("compose_pm_pill");
 
-const emoji = zrequire("../shared/js/emoji");
-
 const server_events_dispatch = zrequire("server_events_dispatch");
 
 function dispatch(ev) {

frontend_tests/node_tests/emoji.js

@@ -9,7 +9,7 @@ const emoji_codes = zrequire("../generated/emoji/emoji_codes.json");
 
 const events = require("./lib/events");
 
-const emoji = zrequire("../shared/js/emoji");
+const emoji = zrequire("emoji");
 
 const realm_emoji = events.test_realm_emojis;
 

frontend_tests/node_tests/emoji_picker.js

@@ -7,7 +7,7 @@ const _ = require("lodash");
 const {zrequire} = require("../zjsunit/namespace");
 const {run_test} = require("../zjsunit/test");
 
-const emoji = zrequire("../shared/js/emoji");
+const emoji = zrequire("emoji");
 const emoji_picker = zrequire("emoji_picker");
 
 const emoji_codes = zrequire("../generated/emoji/emoji_codes.json");

frontend_tests/node_tests/hotkey.js

@@ -110,9 +110,9 @@ message_lists.current = {
     },
 };
 
-const emoji_codes = zrequire("../generated/emoji/emoji_codes.json");
-const emoji = zrequire("../shared/js/emoji");
 const activity = zrequire("activity");
+const emoji = zrequire("emoji");
+const emoji_codes = zrequire("../generated/emoji/emoji_codes.json");
 const hotkey = zrequire("hotkey");
 
 emoji.initialize({

frontend_tests/node_tests/linkifiers.js

@@ -7,9 +7,11 @@ const {run_test} = require("../zjsunit/test");
 const blueslip = require("../zjsunit/zblueslip");
 
 const linkifiers = zrequire("linkifiers");
-const marked = zrequire("../third/marked/lib/marked");
+const markdown = zrequire("markdown");
+const markdown_config = zrequire("markdown_config");
 
 linkifiers.initialize([]);
+markdown.initialize(markdown_config.get_helpers());
 
 run_test("python_to_js_linkifier", () => {
     // The only way to reach python_to_js_linkifier is indirectly, hence the call
@@ -26,7 +28,7 @@ run_test("python_to_js_linkifier", () => {
             id: 20,
         },
     ]);
-    let actual_value = marked.InlineLexer.rules.zulip.linkifiers;
+    let actual_value = markdown.get_linkifier_regexes();
     let expected_value = [/\/aa\/g(?!\w)/gim, /\/aa\/g(?!\w)/g];
     assert.deepEqual(actual_value, expected_value);
     // Test case with multiple replacements.
@@ -37,7 +39,7 @@ run_test("python_to_js_linkifier", () => {
             id: 30,
         },
     ]);
-    actual_value = marked.InlineLexer.rules.zulip.linkifiers;
+    actual_value = markdown.get_linkifier_regexes();
     expected_value = [/#cf(\d+)([A-Z][\dA-Z]*)(?!\w)/g];
     assert.deepEqual(actual_value, expected_value);
     // Test incorrect syntax.
@@ -52,7 +54,7 @@ run_test("python_to_js_linkifier", () => {
             id: 40,
         },
     ]);
-    actual_value = marked.InlineLexer.rules.zulip.linkifiers;
+    actual_value = markdown.get_linkifier_regexes();
     expected_value = [];
     assert.deepEqual(actual_value, expected_value);
 });

frontend_tests/node_tests/markdown.js

@@ -35,7 +35,7 @@ set_global("Image", Image);
 
 set_global("document", {compatMode: "CSS1Compat"});
 
-const emoji = zrequire("../shared/js/emoji");
+const emoji = zrequire("emoji");
 const emoji_codes = zrequire("../generated/emoji/emoji_codes.json");
 const linkifiers = zrequire("linkifiers");
 const pygments_data = zrequire("../generated/pygments_data.json");
@@ -771,10 +771,16 @@ test("backend_only_linkifiers", () => {
 });
 
 test("translate_emoticons_to_names", () => {
+    const get_emoticon_translations = emoji.get_emoticon_translations;
+
+    function translate_emoticons_to_names(src) {
+        return markdown.translate_emoticons_to_names({src, get_emoticon_translations});
+    }
+
     // Simple test
     const test_text = "Testing :)";
     const expected = "Testing :smile:";
-    const result = markdown.translate_emoticons_to_names(test_text);
+    const result = translate_emoticons_to_names(test_text);
     assert.equal(result, expected);
 
     // Extensive tests.
@@ -813,12 +819,16 @@ test("translate_emoticons_to_names", () => {
                 expected: `Hello ${full_name}!`,
             },
         ]) {
-            const result = markdown.translate_emoticons_to_names(original);
+            const result = translate_emoticons_to_names(original);
             assert.equal(result, expected);
         }
     }
 });
 
+test("parse_non_message", () => {
+    assert.equal(markdown.parse_non_message("type `/day`"), "<p>type <code>/day</code></p>");
+});
+
 test("missing unicode emojis", ({override_rewire}) => {
     const message = {raw_content: "\u{1F6B2}"};
 
@@ -833,6 +843,8 @@ test("missing unicode emojis", ({override_rewire}) => {
         assert.equal(codepoint, "1f6b2");
         // return undefined
     });
+
+    markdown.initialize(markdown_config.get_helpers());
     markdown.apply_markdown(message);
     assert.equal(message.content, "<p>\u{1F6B2}</p>");
 });

frontend_tests/node_tests/markdown_parse.js

@@ -0,0 +1,230 @@
+"use strict";
+
+const {strict: assert} = require("assert");
+
+const {zrequire} = require("../zjsunit/namespace");
+const {run_test} = require("../zjsunit/test");
+
+const markdown = zrequire("markdown");
+
+const my_id = 101;
+
+const user_map = new Map();
+user_map.set(my_id, "Me Myself");
+user_map.set(105, "greg");
+
+function get_actual_name_from_user_id(user_id) {
+    return user_map.get(user_id);
+}
+
+function get_user_id_from_name(name) {
+    for (const [user_id, _name] of user_map.entries()) {
+        if (name === _name) {
+            return user_id;
+        }
+    }
+
+    return undefined;
+}
+
+function is_valid_full_name_and_user_id(name, user_id) {
+    return user_map.has(user_id) && user_map.get(user_id) === name;
+}
+
+function my_user_id() {
+    return my_id;
+}
+
+function is_valid_user_id(user_id) {
+    return user_map.has(user_id);
+}
+
+const staff_group = {
+    id: 201,
+    name: "Staff",
+};
+
+const user_group_map = new Map();
+user_group_map.set(staff_group.name, staff_group);
+
+function get_user_group_from_name(name) {
+    return user_group_map.get(name);
+}
+
+function is_member_of_user_group(user_group_id, user_id) {
+    assert.equal(user_group_id, staff_group.id);
+    assert.equal(user_id, my_id);
+    return true;
+}
+
+const social = {
+    stream_id: 301,
+    name: "social",
+};
+
+const sub_map = new Map();
+sub_map.set(social.name, social);
+
+function get_stream_by_name(name) {
+    return sub_map.get(name);
+}
+
+function stream_hash(stream_id) {
+    return `stream-${stream_id}`;
+}
+
+function stream_topic_hash(stream_id, topic) {
+    return `stream-${stream_id}-topic-${topic}`;
+}
+
+function get_emoticon_translations() {
+    return [
+        {regex: /(:\))/g, replacement_text: ":smile:"},
+        {regex: /(<3)/g, replacement_text: ":heart:"},
+    ];
+}
+
+const emoji_map = new Map();
+emoji_map.set("smile", "1f642");
+emoji_map.set("alien", "1f47d");
+
+function get_emoji_codepoint(emoji_name) {
+    return emoji_map.get(emoji_name);
+}
+
+function get_emoji_name(codepoint) {
+    for (const [emoji_name, _codepoint] of emoji_map.entries()) {
+        if (codepoint === _codepoint) {
+            return emoji_name;
+        }
+    }
+
+    return undefined;
+}
+
+const realm_emoji_map = new Map();
+realm_emoji_map.set("heart", "/images/emoji/heart.bmp");
+
+function get_realm_emoji_url(emoji_name) {
+    return realm_emoji_map.get(emoji_name);
+}
+
+const regex = /#foo(\d+)(?!\w)/g;
+const linkifier_map = new Map();
+linkifier_map.set(regex, "http://foo.com/\\1");
+
+function get_linkifier_map() {
+    return linkifier_map;
+}
+
+const helper_config = {
+    // user stuff
+    get_actual_name_from_user_id,
+    get_user_id_from_name,
+    is_valid_full_name_and_user_id,
+    is_valid_user_id,
+    my_user_id,
+
+    // user groups
+    get_user_group_from_name,
+    is_member_of_user_group,
+
+    // stream hashes
+    get_stream_by_name,
+    stream_hash,
+    stream_topic_hash,
+
+    // settings
+    should_translate_emoticons: () => true,
+
+    // emojis
+    get_emoji_codepoint,
+    get_emoji_name,
+    get_emoticon_translations,
+    get_realm_emoji_url,
+
+    // linkifiers
+    get_linkifier_map,
+};
+
+function assert_parse(raw_content, expected_content) {
+    const {content} = markdown.parse({raw_content, helper_config});
+    assert.equal(content, expected_content);
+}
+
+run_test("basics", () => {
+    assert_parse("boring", "<p>boring</p>");
+    assert_parse("**bold**", "<p><strong>bold</strong></p>");
+});
+
+run_test("user mentions", () => {
+    assert_parse("@**greg**", '<p><span class="user-mention" data-user-id="105">@greg</span></p>');
+
+    assert_parse("@**|105**", '<p><span class="user-mention" data-user-id="105">@greg</span></p>');
+
+    assert_parse(
+        "@**greg|105**",
+        '<p><span class="user-mention" data-user-id="105">@greg</span></p>',
+    );
+
+    assert_parse(
+        "@**Me Myself|101**",
+        '<p><span class="user-mention" data-user-id="101">@Me Myself</span></p>',
+    );
+});
+
+run_test("user group mentions", () => {
+    assert_parse(
+        "@*Staff*",
+        '<p><span class="user-group-mention" data-user-group-id="201">@Staff</span></p>',
+    );
+});
+
+run_test("stream links", () => {
+    assert_parse(
+        "#**social**",
+        '<p><a class="stream" data-stream-id="301" href="/stream-301">#social</a></p>',
+    );
+
+    assert_parse(
+        "#**social>lunch**",
+        '<p><a class="stream-topic" data-stream-id="301" href="/stream-301-topic-lunch">#social &gt; lunch</a></p>',
+    );
+});
+
+run_test("emojis", () => {
+    assert_parse(
+        "yup :)",
+        '<p>yup <span aria-label="smile" class="emoji emoji-1f642" role="img" title="smile">:smile:</span></p>',
+    );
+    assert_parse(
+        "I <3 JavaScript",
+        '<p>I <img alt=":heart:" class="emoji" src="/images/emoji/heart.bmp" title="heart"> JavaScript</p>',
+    );
+    assert_parse(
+        "Mars Attacks! \uD83D\uDC7D",
+        '<p>Mars Attacks! <span aria-label="alien" class="emoji emoji-1f47d" role="img" title="alien">:alien:</span></p>',
+    );
+});
+
+run_test("linkifiers", () => {
+    assert_parse(
+        "see #foo12345 for details",
+        '<p>see <a href="http://foo.com/12345" title="http://foo.com/12345">#foo12345</a> for details</p>',
+    );
+});
+
+run_test("topic links", () => {
+    const topic = "progress on #foo101 and #foo102";
+    const topic_links = markdown.get_topic_links({topic, get_linkifier_map});
+    assert.deepEqual(topic_links, [
+        {
+            text: "#foo101",
+            url: "http://foo.com/101",
+        },
+        {
+            text: "#foo102",
+            url: "http://foo.com/102",
+        },
+    ]);
+});

frontend_tests/node_tests/reactions.js

@@ -61,8 +61,8 @@ message_lists.current = {
 };
 set_global("document", "document-stub");
 
+const emoji = zrequire("emoji");
 const emoji_codes = zrequire("../generated/emoji/emoji_codes.json");
-const emoji = zrequire("../shared/js/emoji");
 const people = zrequire("people");
 const reactions = zrequire("reactions");
 

frontend_tests/node_tests/typeahead_helper.js

@@ -14,7 +14,7 @@ const peer_data = zrequire("peer_data");
 const people = zrequire("people");
 const stream_data = zrequire("stream_data");
 
-const emoji = zrequire("../shared/js/emoji");
+const emoji = zrequire("emoji");
 const pygments_data = zrequire("../generated/pygments_data.json");
 const actual_pygments_data = {...pygments_data};
 const ct = zrequire("composebox_typeahead");

frontend_tests/node_tests/user_status.js

@@ -10,7 +10,7 @@ const channel = mock_esm("../../static/js/channel");
 
 const user_status = zrequire("user_status");
 const emoji_codes = zrequire("../generated/emoji/emoji_codes.json");
-const emoji = zrequire("../shared/js/emoji");
+const emoji = zrequire("emoji");
 
 const emoji_params = {
     realm_emoji: {

puppet/zulip/manifests/profile/base.pp

@@ -61,7 +61,7 @@ class zulip::profile::base {
     }
   }
   package { 'ntp': ensure => 'purged', before => Package['chrony'] }
-  service { 'chrony': ensure => 'running', require => Package['chrony'] }
+  service { 'chrony': require => Package['chrony'] }
   package { $base_packages: ensure => 'installed' }
 
   group { 'zulip':

scripts/lib/check-database-compatibility.py

@@ -28,6 +28,45 @@ missing.discard(("guardian", "0001_initial"))
 missing.discard(("sites", "0001_initial"))
 missing.discard(("sites", "0002_alter_domain_unique"))
 
+# These migrations were squashed into 0001, in 6fbddf578a6e through
+# a21f2d771553, 1.7.0~3135.
+missing.difference_update(
+    [
+        ("zerver", "0002_django_1_8"),
+        ("zerver", "0003_custom_indexes"),
+        ("zerver", "0004_userprofile_left_side_userlist"),
+        ("zerver", "0005_auto_20150920_1340"),
+        ("zerver", "0006_zerver_userprofile_email_upper_idx"),
+        ("zerver", "0007_userprofile_is_bot_active_indexes"),
+        ("zerver", "0008_preregistrationuser_upper_email_idx"),
+        ("zerver", "0009_add_missing_migrations"),
+        ("zerver", "0010_delete_streamcolor"),
+        ("zerver", "0011_remove_guardian"),
+        ("zerver", "0012_remove_appledevicetoken"),
+        ("zerver", "0013_realmemoji"),
+        ("zerver", "0014_realm_emoji_url_length"),
+        ("zerver", "0015_attachment"),
+        ("zerver", "0016_realm_create_stream_by_admins_only"),
+        ("zerver", "0017_userprofile_bot_type"),
+        ("zerver", "0018_realm_emoji_message"),
+        ("zerver", "0019_preregistrationuser_realm_creation"),
+        ("zerver", "0020_add_tracking_attachment"),
+        ("zerver", "0021_migrate_attachment_data"),
+        ("zerver", "0022_subscription_pin_to_top"),
+        ("zerver", "0023_userprofile_default_language"),
+        ("zerver", "0024_realm_allow_message_editing"),
+        ("zerver", "0025_realm_message_content_edit_limit"),
+        ("zerver", "0026_delete_mituser"),
+        ("zerver", "0027_realm_default_language"),
+        ("zerver", "0028_userprofile_tos_version"),
+    ]
+)
+
+# This migration was in python-social-auth, and was mistakenly removed
+# from its `replaces` in
+# https://github.com/python-social-auth/social-app-django/pull/25
+missing.discard(("default", "0005_auto_20160727_2333"))
+
 for key, migration in loader.disk_migrations.items():
     missing.discard(key)
     missing.difference_update(migration.replaces)

scripts/lib/supervisor.py

@@ -42,7 +42,10 @@ def list_supervisor_processes(
         if filter_names:
             match = False
             for filter_name in filter_names:
-                if filter_name.endswith(":*") and name.startswith(filter_name[:-1]):
+                # zulip-tornado:* matches zulip-tornado:9800 and zulip-tornado
+                if filter_name.endswith(":*") and (
+                    name.startswith(filter_name[:-1]) or name == filter_name[:-2]
+                ):
                     match = True
                     break
                 if name == filter_name:

scripts/lib/upgrade-zulip-stage-2

@@ -309,11 +309,10 @@ if not args.skip_migrations:
         if line_str.startswith("[ ]"):
             migrations_needed = True
 
-if (not args.skip_puppet or migrations_needed) and IS_SERVER_UP:
-    # By default, we shut down the service to apply migrations and
-    # Puppet changes, to minimize risk of issues due to inconsistent
-    # state.
-    shutdown_server()
+# NOTE: Here begins the most likely critical period, where we may be
+# shutting down the server; we should strive to minimize the number of
+# steps that happen between here and the "Restarting Zulip" line
+# below.
 
 if rabbitmq_dist_listen:
     shutdown_server()
@@ -327,6 +326,7 @@ if cookie_size is not None and cookie_size == 20:
     # characters long by a good randomizer, it would be 96 bits and
     # more than sufficient.  We generate, using good randomness, a
     # 255-character cookie, the max allowed length.
+    shutdown_server()
     logging.info("Generating a secure erlang cookie...")
     subprocess.check_call(["./scripts/setup/generate-rabbitmq-cookie"])
 
@@ -357,19 +357,25 @@ if classes != new_classes:
 
 
 if not args.skip_puppet:
+    # Puppet may adjust random services; to minimize risk of issues
+    # due to inconsistent state, we shut down the server first.
+    shutdown_server()
     logging.info("Applying Puppet changes...")
     subprocess.check_call(["./scripts/zulip-puppet-apply", "--force"])
     subprocess.check_call(["apt-get", "-y", "--allow-downgrades", "upgrade"])
+    # Puppet may have reloaded supervisor, and in so doing started
+    # services; mark as potentially needing to stop the server.
+    IS_SERVER_UP = True
 
 if migrations_needed:
+    # Database migrations assume that they run on a database in
+    # quiesced state.
+    shutdown_server()
     logging.info("Applying database migrations...")
     subprocess.check_call(["./manage.py", "migrate", "--noinput"], preexec_fn=su_to_zulip)
 
 logging.info("Restarting Zulip...")
-if IS_SERVER_UP or not args.skip_puppet:
-    # Even if the server wasn't up previously, puppet might have
-    # started it if there were supervisord configuration changes, so
-    # we need to use restart-server if puppet ran.
+if IS_SERVER_UP:
     restart_args = ["--fill-cache"]
     if args.skip_tornado:
         restart_args.append("--skip-tornado")

static/js/admin.js

@@ -161,6 +161,7 @@ export function build_page() {
             settings_config.create_web_public_stream_policy_values,
         disable_enable_spectator_access_setting: !page_params.server_web_public_streams_enabled,
         can_sort_by_email: settings_data.show_email(),
+        realm_push_notifications_enabled: page_params.realm_push_notifications_enabled,
     };
 
     if (options.realm_logo_source !== "D" && options.realm_night_logo_source === "D") {

static/js/composebox_typeahead.js

@@ -2,7 +2,6 @@ import $ from "jquery";
 import _ from "lodash";
 
 import pygments_data from "../generated/pygments_data.json";
-import * as emoji from "../shared/js/emoji";
 import * as typeahead from "../shared/js/typeahead";
 
 import * as compose from "./compose";
@@ -10,6 +9,7 @@ import * as compose_pm_pill from "./compose_pm_pill";
 import * as compose_state from "./compose_state";
 import * as compose_ui from "./compose_ui";
 import * as compose_validate from "./compose_validate";
+import * as emoji from "./emoji";
 import * as flatpickr from "./flatpickr";
 import {$t} from "./i18n";
 import * as message_store from "./message_store";

static/js/emoji.js

@@ -6,13 +6,13 @@ let emoji_codes = {};
 // `emojis_by_name` is the central data source that is supposed to be
 // used by every widget in the web app for gathering data for displaying
 // emojis. Emoji picker uses this data to derive data for its own use.
-export const emojis_by_name = new Map();
+export let emojis_by_name = new Map();
 
 export const all_realm_emojis = new Map();
 export const active_realm_emojis = new Map();
 export const deactivated_emoji_name_to_code = new Map();
 
-const default_emoji_aliases = new Map();
+let default_emoji_aliases = new Map();
 
 // For legacy reasons we track server_realm_emoji_data,
 // since our settings code builds off that format.  We
@@ -28,9 +28,12 @@ export function get_server_realm_emoji_data() {
 
 let emoticon_translations = [];
 
-function build_emoticon_translations() {
+function build_emoticon_translations({emoticon_conversions}) {
     /*
 
+    Please keep this as a pure function so that we can
+    eventually share this code with the mobile codebase.
+
     Build a data structure that looks like something
     like this:
 
@@ -54,7 +57,7 @@ function build_emoticon_translations() {
     */
 
     const translations = [];
-    for (const [emoticon, replacement_text] of Object.entries(emoji_codes.emoticon_conversions)) {
+    for (const [emoticon, replacement_text] of Object.entries(emoticon_conversions)) {
         const regex = new RegExp("(" + _.escapeRegExp(emoticon) + ")", "g");
 
         translations.push({
@@ -63,7 +66,7 @@ function build_emoticon_translations() {
         });
     }
 
-    emoticon_translations = translations;
+    return translations;
 }
 
 const zulip_emoji = {
@@ -110,8 +113,34 @@ export function get_realm_emoji_url(emoji_name) {
     return data.emoji_url;
 }
 
-export function build_emoji_data(realm_emojis) {
-    emojis_by_name.clear();
+function build_emojis_by_name({
+    realm_emojis,
+    emoji_catalog,
+    get_emoji_name,
+    default_emoji_aliases,
+}) {
+    // Please keep this as a pure function so that we can
+    // eventually share this code with the mobile codebase.
+    const map = new Map();
+
+    for (const codepoints of Object.values(emoji_catalog)) {
+        for (const codepoint of codepoints) {
+            const emoji_name = get_emoji_name(codepoint);
+            if (emoji_name !== undefined) {
+                const emoji_dict = {
+                    name: emoji_name,
+                    display_name: emoji_name,
+                    aliases: default_emoji_aliases.get(codepoint),
+                    is_realm_emoji: false,
+                    emoji_code: codepoint,
+                    has_reacted: false,
+                };
+                // We may later get overridden by a realm emoji.
+                map.set(emoji_name, emoji_dict);
+            }
+        }
+    }
+
     for (const [realm_emoji_name, realm_emoji] of realm_emojis) {
         const emoji_dict = {
             name: realm_emoji_name,
@@ -121,25 +150,12 @@ export function build_emoji_data(realm_emojis) {
             url: realm_emoji.emoji_url,
             has_reacted: false,
         };
-        emojis_by_name.set(realm_emoji_name, emoji_dict);
+
+        // We want the realm emoji to overwrite any existing entry in this map.
+        map.set(realm_emoji_name, emoji_dict);
     }
 
-    for (const codepoints of Object.values(emoji_codes.emoji_catalog)) {
-        for (const codepoint of codepoints) {
-            const emoji_name = get_emoji_name(codepoint);
-            if (emoji_name !== undefined && !emojis_by_name.has(emoji_name)) {
-                const emoji_dict = {
-                    name: emoji_name,
-                    display_name: emoji_name,
-                    aliases: default_emoji_aliases.get(codepoint),
-                    is_realm_emoji: false,
-                    emoji_code: codepoint,
-                    has_reacted: false,
-                };
-                emojis_by_name.set(emoji_name, emoji_dict);
-            }
-        }
-    }
+    return map;
 }
 
 export function update_emojis(realm_emojis) {
@@ -182,7 +198,12 @@ export function update_emojis(realm_emojis) {
     // here "zulip" is an emoji name, which is fine.
     active_realm_emojis.set("zulip", zulip_emoji);
 
-    build_emoji_data(active_realm_emojis);
+    emojis_by_name = build_emojis_by_name({
+        realm_emojis: active_realm_emojis,
+        emoji_catalog: emoji_codes.emoji_catalog,
+        get_emoji_name,
+        default_emoji_aliases,
+    });
 }
 
 // This function will provide required parameters that would
@@ -244,20 +265,37 @@ export function get_emoji_details_for_rendering(opts) {
     };
 }
 
+function build_default_emoji_aliases({names, get_emoji_codepoint}) {
+    // Please keep this as a pure function so that we can
+    // eventually share this code with the mobile codebase.
+
+    // Create a map of codepoint -> names
+    const map = new Map();
+
+    for (const name of names) {
+        const base_name = get_emoji_codepoint(name);
+
+        if (map.has(base_name)) {
+            map.get(base_name).push(name);
+        } else {
+            map.set(base_name, [name]);
+        }
+    }
+
+    return map;
+}
+
 export function initialize(params) {
     emoji_codes = params.emoji_codes;
 
-    build_emoticon_translations();
+    emoticon_translations = build_emoticon_translations({
+        emoticon_conversions: emoji_codes.emoticon_conversions,
+    });
 
-    for (const value of emoji_codes.names) {
-        const base_name = get_emoji_codepoint(value);
-
-        if (default_emoji_aliases.has(base_name)) {
-            default_emoji_aliases.get(base_name).push(value);
-        } else {
-            default_emoji_aliases.set(base_name, [value]);
-        }
-    }
+    default_emoji_aliases = build_default_emoji_aliases({
+        names: emoji_codes.names,
+        get_emoji_codepoint,
+    });
 
     update_emojis(params.realm_emoji);
 }

static/js/emoji_picker.js

@@ -1,7 +1,6 @@
 import $ from "jquery";
 
 import emoji_codes from "../generated/emoji/emoji_codes.json";
-import * as emoji from "../shared/js/emoji";
 import * as typeahead from "../shared/js/typeahead";
 import render_emoji_popover from "../templates/emoji_popover.hbs";
 import render_emoji_popover_content from "../templates/emoji_popover_content.hbs";
@@ -10,6 +9,7 @@ import render_emoji_showcase from "../templates/emoji_showcase.hbs";
 
 import * as blueslip from "./blueslip";
 import * as compose_ui from "./compose_ui";
+import * as emoji from "./emoji";
 import * as message_lists from "./message_lists";
 import * as message_store from "./message_store";
 import * as popovers from "./popovers";

static/js/hotkey.js

@@ -1,7 +1,5 @@
 import $ from "jquery";
 
-import * as emoji from "../shared/js/emoji";
-
 import * as activity from "./activity";
 import * as browser_history from "./browser_history";
 import * as common from "./common";
@@ -12,6 +10,7 @@ import * as condense from "./condense";
 import * as copy_and_paste from "./copy_and_paste";
 import * as deprecated_feature_notice from "./deprecated_feature_notice";
 import * as drafts from "./drafts";
+import * as emoji from "./emoji";
 import * as emoji_picker from "./emoji_picker";
 import * as feedback_widget from "./feedback_widget";
 import * as gear_menu from "./gear_menu";

static/js/linkifiers.js

@@ -1,22 +1,9 @@
-import marked from "../third/marked/lib/marked";
-
 import * as blueslip from "./blueslip";
 
-const linkifier_map = new Map();
-export let linkifier_list = [];
+const linkifier_map = new Map(); // regex -> url
 
-function handleLinkifier(pattern, matches) {
-    let url = linkifier_map.get(pattern);
-
-    let current_group = 1;
-
-    for (const match of matches) {
-        const back_ref = "\\" + current_group;
-        url = url.replace(back_ref, match);
-        current_group += 1;
-    }
-
-    return url;
+export function get_linkifier_map() {
+    return linkifier_map;
 }
 
 function python_to_js_linkifier(pattern, url) {
@@ -78,11 +65,7 @@ function python_to_js_linkifier(pattern, url) {
 }
 
 export function update_linkifier_rules(linkifiers) {
-    // Update the marked parser with our particular set of linkifiers
     linkifier_map.clear();
-    linkifier_list = [];
-
-    const marked_rules = [];
 
     for (const linkifier of linkifiers) {
         const [regex, final_url] = python_to_js_linkifier(linkifier.pattern, linkifier.url_format);
@@ -92,18 +75,9 @@ export function update_linkifier_rules(linkifiers) {
         }
 
         linkifier_map.set(regex, final_url);
-        linkifier_list.push({
-            pattern: regex,
-            url_format: final_url,
-        });
-        marked_rules.push(regex);
     }
-
-    marked.InlineLexer.rules.zulip.linkifiers = marked_rules;
 }
 
 export function initialize(linkifiers) {
     update_linkifier_rules(linkifiers);
-
-    marked.setOptions({linkifierHandler: handleLinkifier});
 }

static/js/markdown.js

@@ -2,12 +2,10 @@ import {isValid} from "date-fns";
 import katex from "katex"; // eslint-disable-line import/no-unresolved
 import _ from "lodash";
 
-import * as emoji from "../shared/js/emoji";
 import * as fenced_code from "../shared/js/fenced_code";
 import marked from "../third/marked/lib/marked";
 
 import * as blueslip from "./blueslip";
-import * as linkifiers from "./linkifiers";
 
 // This contains zulip's frontend Markdown implementation; see
 // docs/subsystems/markdown.md for docs on our Markdown syntax.  The other
@@ -23,8 +21,9 @@ import * as linkifiers from "./linkifiers";
 // for example usage.
 let helpers;
 
-// Regexes that match some of our common backend-only Markdown syntax
-const backend_only_markdown_re = [
+// If we see preview-related syntax in our content, we will need the
+// backend to render it.
+const preview_regexes = [
     // Inline image previews, check for contiguous chars ending in image suffix
     // To keep the below regexes simple, split them out for the end-of-message case
 
@@ -33,12 +32,16 @@ const backend_only_markdown_re = [
 
     // Twitter and youtube links are given previews
 
-    /\S*(?:twitter|youtube).com\/\S*/,
+    /\S*(?:twitter|youtube)\.com\/\S*/,
 ];
 
-export function translate_emoticons_to_names(text) {
+function contains_preview_link(content) {
+    return preview_regexes.some((re) => re.test(content));
+}
+
+export function translate_emoticons_to_names({src, get_emoticon_translations}) {
     // Translates emoticons in a string to their colon syntax.
-    let translated = text;
+    let translated = src;
     let replacement_text;
     const terminal_symbols = ",.;?!()[] \"'\n\t"; // From composebox_typeahead
     const symbols_except_space = terminal_symbols.replace(" ", "");
@@ -64,7 +67,7 @@ export function translate_emoticons_to_names(text) {
         return match;
     };
 
-    for (const translation of emoji.get_emoticon_translations()) {
+    for (const translation of get_emoticon_translations()) {
         // We can't pass replacement_text directly into
         // emoticon_replacer, because emoticon_replacer is
         // a callback for `replace()`.  Instead we just mutate
@@ -76,30 +79,43 @@ export function translate_emoticons_to_names(text) {
     return translated;
 }
 
+function contains_problematic_linkifier(content) {
+    // If a linkifier doesn't start with some specified characters
+    // then don't render it locally. It is workaround for the fact that
+    // javascript regex doesn't support lookbehind.
+    for (const re of helpers.get_linkifier_map().keys()) {
+        const pattern = /[^\s"'(,:<]/.source + re.source + /(?!\w)/.source;
+        const regex = new RegExp(pattern);
+        if (regex.test(content)) {
+            return true;
+        }
+    }
+
+    return false;
+}
+
 export function contains_backend_only_syntax(content) {
     // Try to guess whether or not a message contains syntax that only the
     // backend Markdown processor can correctly handle.
     // If it doesn't, we can immediately render it client-side for local echo.
-    const markedup = backend_only_markdown_re.find((re) => re.test(content));
-
-    // If a linkifier doesn't start with some specified characters
-    // then don't render it locally. It is workaround for the fact that
-    // javascript regex doesn't support lookbehind.
-    const linkifier_list = linkifiers.linkifier_list;
-    const false_linkifier_match = linkifier_list.find((re) => {
-        const pattern = /[^\s"'(,:<]/.source + re.pattern.source + /(?!\w)/.source;
-        const regex = new RegExp(pattern);
-        return regex.test(content);
-    });
-    return markedup !== undefined || false_linkifier_match !== undefined;
+    return contains_preview_link(content) || contains_problematic_linkifier(content);
 }
 
-export function apply_markdown(message) {
+function parse_with_options({raw_content, helper_config, options}) {
+    // Given the raw markdown content of a message (raw_content)
+    // we return the HTML content (content) and flags.
+    // Our caller passes a helper_config object that has several
+    // helper functions for getting info about users, streams, etc.
+    // And it also passes in options for the marked processor.
+
+    helpers = helper_config;
+
     let mentioned = false;
     let mentioned_group = false;
     let mentioned_wildcard = false;
 
-    const options = {
+    const marked_options = {
+        ...options,
         userMentionHandler(mention, silently) {
             if (mention === "all" || mention === "everyone" || mention === "stream") {
                 let classes;
@@ -144,15 +160,15 @@ export function apply_markdown(message) {
 
                 if (full_name === undefined) {
                     // For @**|id** syntax
-                    if (!helpers.is_valid_user_id(user_id)) {
+                    if (!helper_config.is_valid_user_id(user_id)) {
                         // silently ignore invalid user id.
                         user_id = undefined;
                     } else {
-                        full_name = helpers.get_actual_name_from_user_id(user_id);
+                        full_name = helper_config.get_actual_name_from_user_id(user_id);
                     }
                 } else {
                     // For @**user|id** syntax
-                    if (!helpers.is_valid_full_name_and_user_id(full_name, user_id)) {
+                    if (!helper_config.is_valid_full_name_and_user_id(full_name, user_id)) {
                         user_id = undefined;
                         full_name = undefined;
                     }
@@ -162,7 +178,7 @@ export function apply_markdown(message) {
             if (user_id === undefined) {
                 // Handle normal syntax
                 full_name = mention;
-                user_id = helpers.get_user_id_from_name(full_name);
+                user_id = helper_config.get_user_id_from_name(full_name);
             }
 
             if (user_id === undefined) {
@@ -179,12 +195,12 @@ export function apply_markdown(message) {
 
             // If I mention "@aLiCe sMITH", I still want "Alice Smith" to
             // show in the pill.
-            let display_text = helpers.get_actual_name_from_user_id(user_id);
+            let display_text = helper_config.get_actual_name_from_user_id(user_id);
             let classes;
             if (silently) {
                 classes = "user-mention silent";
             } else {
-                if (helpers.my_user_id() === user_id) {
+                if (helper_config.my_user_id() === user_id) {
                     // Personal mention of current user.
                     mentioned = true;
                 }
@@ -197,7 +213,7 @@ export function apply_markdown(message) {
             )}</span>`;
         },
         groupMentionHandler(name, silently) {
-            const group = helpers.get_user_group_from_name(name);
+            const group = helper_config.get_user_group_from_name(name);
             if (group !== undefined) {
                 let display_text;
                 let classes;
@@ -207,7 +223,9 @@ export function apply_markdown(message) {
                 } else {
                     display_text = "@" + group.name;
                     classes = "user-group-mention";
-                    if (helpers.is_member_of_user_group(group.id, helpers.my_user_id())) {
+                    if (
+                        helper_config.is_member_of_user_group(group.id, helper_config.my_user_id())
+                    ) {
                         // Mentioned the current user's group.
                         mentioned_group = true;
                     }
@@ -249,34 +267,28 @@ export function apply_markdown(message) {
     };
 
     // Our Python-Markdown processor appends two \n\n to input
-    message.content = marked(message.raw_content + "\n\n", options).trim();
+    const content = marked(raw_content + "\n\n", marked_options).trim();
 
     // Simulate message flags for our locally rendered
     // message. Messages the user themselves sent via the browser are
     // always marked as read.
-    message.flags = ["read"];
+    const flags = ["read"];
     if (mentioned || mentioned_group) {
-        message.flags.push("mentioned");
+        flags.push("mentioned");
     }
     if (mentioned_wildcard) {
-        message.flags.push("wildcard_mentioned");
+        flags.push("wildcard_mentioned");
     }
 
-    message.is_me_message = is_status_message(message.raw_content);
+    return {content, flags};
 }
 
-export function add_topic_links(message) {
-    if (message.type !== "stream") {
-        message.topic_links = [];
-        return;
-    }
-    const topic = message.topic;
+export function get_topic_links({topic, get_linkifier_map}) {
+    // We export this for testing purposes, and mobile may want to
+    // use this as well in the future.
     const links = [];
-    const linkifier_list = linkifiers.linkifier_list;
 
-    for (const linkifier of linkifier_list) {
-        const pattern = linkifier.pattern;
-        const url = linkifier.url_format;
+    for (const [pattern, url] of get_linkifier_map().entries()) {
         let match;
         while ((match = pattern.exec(topic)) !== null) {
             let link_url = url;
@@ -307,7 +319,8 @@ export function add_topic_links(message) {
     for (const match of links) {
         delete match.index;
     }
-    message.topic_links = links;
+
+    return links;
 }
 
 export function is_status_message(raw_content) {
@@ -320,9 +333,9 @@ function make_emoji_span(codepoint, title, alt_text) {
     )}" role="img" title="${_.escape(title)}">${_.escape(alt_text)}</span>`;
 }
 
-function handleUnicodeEmoji(unicode_emoji) {
+function handleUnicodeEmoji({unicode_emoji, get_emoji_name}) {
     const codepoint = unicode_emoji.codePointAt(0).toString(16);
-    const emoji_name = emoji.get_emoji_name(codepoint);
+    const emoji_name = get_emoji_name(codepoint);
 
     if (emoji_name) {
         const alt_text = ":" + emoji_name + ":";
@@ -333,7 +346,7 @@ function handleUnicodeEmoji(unicode_emoji) {
     return unicode_emoji;
 }
 
-function handleEmoji(emoji_name) {
+function handleEmoji({emoji_name, get_realm_emoji_url, get_emoji_codepoint}) {
     const alt_text = ":" + emoji_name + ":";
     const title = emoji_name.replace(/_/g, " ");
 
@@ -344,7 +357,7 @@ function handleEmoji(emoji_name) {
     // Otherwise we'll look at Unicode emoji to render with an emoji
     // span using the spritesheet; and if it isn't one of those
     // either, we pass through the plain text syntax unmodified.
-    const emoji_url = emoji.get_realm_emoji_url(emoji_name);
+    const emoji_url = get_realm_emoji_url(emoji_name);
 
     if (emoji_url) {
         return `<img alt="${_.escape(alt_text)}" class="emoji" src="${_.escape(
@@ -352,7 +365,7 @@ function handleEmoji(emoji_name) {
         )}" title="${_.escape(title)}">`;
     }
 
-    const codepoint = emoji.get_emoji_codepoint(emoji_name);
+    const codepoint = get_emoji_codepoint(emoji_name);
     if (codepoint) {
         return make_emoji_span(codepoint, title, alt_text);
     }
@@ -360,6 +373,20 @@ function handleEmoji(emoji_name) {
     return alt_text;
 }
 
+function handleLinkifier({pattern, matches, get_linkifier_map}) {
+    let url = get_linkifier_map().get(pattern);
+
+    let current_group = 1;
+
+    for (const match of matches) {
+        const back_ref = "\\" + current_group;
+        url = url.replace(back_ref, match);
+        current_group += 1;
+    }
+
+    return url;
+}
+
 function handleTimestamp(time) {
     let timeobject;
     if (Number.isNaN(Number(time))) {
@@ -386,23 +413,23 @@ function handleTimestamp(time) {
     return `<time datetime="${escaped_isotime}">${escaped_time}</time>`;
 }
 
-function handleStream(stream_name) {
-    const stream = helpers.get_stream_by_name(stream_name);
+function handleStream({stream_name, get_stream_by_name, stream_hash}) {
+    const stream = get_stream_by_name(stream_name);
     if (stream === undefined) {
         return undefined;
     }
-    const href = helpers.stream_hash(stream.stream_id);
+    const href = stream_hash(stream.stream_id);
     return `<a class="stream" data-stream-id="${_.escape(stream.stream_id)}" href="/${_.escape(
         href,
     )}">#${_.escape(stream.name)}</a>`;
 }
 
-function handleStreamTopic(stream_name, topic) {
-    const stream = helpers.get_stream_by_name(stream_name);
+function handleStreamTopic({stream_name, topic, get_stream_by_name, stream_topic_hash}) {
+    const stream = get_stream_by_name(stream_name);
     if (stream === undefined || !topic) {
         return undefined;
     }
-    const href = helpers.stream_topic_hash(stream.stream_id, topic);
+    const href = stream_topic_hash(stream.stream_id, topic);
     const text = `#${stream.name} > ${topic}`;
     return `<a class="stream-topic" data-stream-id="${_.escape(
         stream.stream_id,
@@ -422,9 +449,11 @@ function handleTex(tex, fullmatch) {
     }
 }
 
-export function initialize(helper_config) {
-    helpers = helper_config;
+export function get_linkifier_regexes() {
+    return Array.from(helpers.get_linkifier_map().keys());
+}
 
+export function parse({raw_content, helper_config}) {
     function disable_markdown_regex(rules, name) {
         rules[name] = {
             exec() {
@@ -434,18 +463,19 @@ export function initialize(helper_config) {
     }
 
     // Configure the marked Markdown parser for our usage
-    const r = new marked.Renderer();
+    const renderer = new marked.Renderer();
 
     // No <code> around our code blocks instead a codehilite <div> and disable
     // class-specific highlighting.
-    r.code = (code) => fenced_code.wrap_code(code) + "\n\n";
+    renderer.code = (code) => fenced_code.wrap_code(code) + "\n\n";
 
     // Prohibit empty links for some reason.
-    const old_link = r.link;
-    r.link = (href, title, text) => old_link.call(r, href, title, text.trim() ? text : href);
+    const old_link = renderer.link;
+    renderer.link = (href, title, text) =>
+        old_link.call(renderer, href, title, text.trim() ? text : href);
 
     // Put a newline after a <br> in the generated HTML to match Markdown
-    r.br = function () {
+    renderer.br = function () {
         return "<br>\n";
     };
 
@@ -454,13 +484,16 @@ export function initialize(helper_config) {
     }
 
     function preprocess_translate_emoticons(src) {
-        if (!helpers.should_translate_emoticons()) {
+        if (!helper_config.should_translate_emoticons()) {
             return src;
         }
 
         // In this scenario, the message has to be from the user, so the only
         // requirement should be that they have the setting on.
-        return translate_emoticons_to_names(src);
+        return translate_emoticons_to_names({
+            src,
+            get_emoticon_translations: helper_config.get_emoticon_translations,
+        });
     }
 
     // Disable headings
@@ -485,7 +518,55 @@ export function initialize(helper_config) {
     // HTML into the output. This generated HTML is safe to not escape
     fenced_code.set_stash_func((html) => marked.stashHtml(html, true));
 
-    marked.setOptions({
+    function streamHandler(stream_name) {
+        return handleStream({
+            stream_name,
+            get_stream_by_name: helper_config.get_stream_by_name,
+            stream_hash: helper_config.stream_hash,
+        });
+    }
+
+    function streamTopicHandler(stream_name, topic) {
+        return handleStreamTopic({
+            stream_name,
+            topic,
+            get_stream_by_name: helper_config.get_stream_by_name,
+            stream_topic_hash: helper_config.stream_topic_hash,
+        });
+    }
+
+    function emojiHandler(emoji_name) {
+        return handleEmoji({
+            emoji_name,
+            get_realm_emoji_url: helper_config.get_realm_emoji_url,
+            get_emoji_codepoint: helper_config.get_emoji_codepoint,
+        });
+    }
+
+    function unicodeEmojiHandler(unicode_emoji) {
+        return handleUnicodeEmoji({
+            unicode_emoji,
+            get_emoji_name: helper_config.get_emoji_name,
+        });
+    }
+
+    function linkifierHandler(pattern, matches) {
+        return handleLinkifier({
+            pattern,
+            matches,
+            get_linkifier_map: helper_config.get_linkifier_map,
+        });
+    }
+
+    const options = {
+        get_linkifier_regexes,
+        linkifierHandler,
+        emojiHandler,
+        unicodeEmojiHandler,
+        streamHandler,
+        streamTopicHandler,
+        texHandler: handleTex,
+        timestampHandler: handleTimestamp,
         gfm: true,
         tables: true,
         breaks: true,
@@ -494,13 +575,52 @@ export function initialize(helper_config) {
         smartLists: true,
         smartypants: false,
         zulip: true,
-        emojiHandler: handleEmoji,
-        unicodeEmojiHandler: handleUnicodeEmoji,
-        streamHandler: handleStream,
-        streamTopicHandler: handleStreamTopic,
-        texHandler: handleTex,
-        timestampHandler: handleTimestamp,
-        renderer: r,
+        renderer,
         preprocessors: [preprocess_code_blocks, preprocess_translate_emoticons],
+    };
+
+    return parse_with_options({raw_content, helper_config, options});
+}
+
+// NOTE: Everything below this line is likely to be webapp-specific
+//       and won't be used by future platforms such as mobile.
+//       We may eventually move this code to a new file, but we want
+//       to wait till the dust settles a bit on some other changes first.
+
+let webapp_helpers;
+
+export function initialize(helper_config) {
+    // This is generally only intended to be called by the webapp. Most
+    // other platforms should call setup().
+    webapp_helpers = helper_config;
+    helpers = helper_config;
+}
+
+export function apply_markdown(message) {
+    // This is generally only intended to be called by the webapp. Most
+    // other platforms should call parse().
+    const raw_content = message.raw_content;
+    const {content, flags} = parse({raw_content, helper_config: webapp_helpers});
+    message.content = content;
+    message.flags = flags;
+    message.is_me_message = is_status_message(raw_content);
+}
+
+export function add_topic_links(message) {
+    if (message.type !== "stream") {
+        message.topic_links = [];
+        return;
+    }
+    message.topic_links = get_topic_links({
+        topic: message.topic,
+        get_linkifier_map: webapp_helpers.get_linkifier_map,
     });
 }
+
+export function parse_non_message(raw_content) {
+    // Occasionally we get markdown from the server that is not technically
+    // a message, but we want to convert it to HTML. Note that we parse
+    // raw_content exactly as if it were a Zulip message, so we will
+    // handle things like mentions, stream links, and linkifiers.
+    return parse({raw_content, helper_config: webapp_helpers}).content;
+}

static/js/markdown_config.js

@@ -1,4 +1,6 @@
+import * as emoji from "./emoji";
 import * as hash_util from "./hash_util";
+import * as linkifiers from "./linkifiers";
 import * as people from "./people";
 import * as stream_data from "./stream_data";
 import * as user_groups from "./user_groups";
@@ -26,6 +28,36 @@ import {user_settings} from "./user_settings";
     when the lookups fail.
 */
 
+function abstract_map(map) {
+    return {
+        keys: () => map.keys(),
+        entries: () => map.entries(),
+        get: (k) => map.get(k),
+    };
+}
+
+function stream(obj) {
+    if (obj === undefined) {
+        return undefined;
+    }
+
+    return {
+        stream_id: obj.stream_id,
+        name: obj.name,
+    };
+}
+
+function user_group(obj) {
+    if (obj === undefined) {
+        return undefined;
+    }
+
+    return {
+        id: obj.id,
+        name: obj.name,
+    };
+}
+
 export const get_helpers = () => ({
     // user stuff
     get_actual_name_from_user_id: people.get_actual_name_from_user_id,
@@ -35,14 +67,23 @@ export const get_helpers = () => ({
     is_valid_user_id: people.is_known_user_id,
 
     // user groups
-    get_user_group_from_name: user_groups.get_user_group_from_name,
+    get_user_group_from_name: (name) => user_group(user_groups.get_user_group_from_name(name)),
     is_member_of_user_group: user_groups.is_member_of,
 
     // stream hashes
-    get_stream_by_name: stream_data.get_sub,
+    get_stream_by_name: (name) => stream(stream_data.get_sub(name)),
     stream_hash: hash_util.by_stream_url,
     stream_topic_hash: hash_util.by_stream_topic_url,
 
     // settings
     should_translate_emoticons: () => user_settings.translate_emoticons,
+
+    // emojis
+    get_emoji_name: emoji.get_emoji_name,
+    get_emoji_codepoint: emoji.get_emoji_codepoint,
+    get_emoticon_translations: emoji.get_emoticon_translations,
+    get_realm_emoji_url: emoji.get_realm_emoji_url,
+
+    // linkifiers
+    get_linkifier_map: () => abstract_map(linkifiers.get_linkifier_map()),
 });

static/js/navbar_alerts.js

@@ -23,6 +23,7 @@ import * as util from "./util";
 export function resize_app() {
     const navbar_alerts_wrapper_height = $("#navbar_alerts_wrapper").height();
     $("body > .app").height("calc(100% - " + navbar_alerts_wrapper_height + "px)");
+    $(".recent_topics_container").height("calc(100vh - " + navbar_alerts_wrapper_height + "px)");
 
     // the floating recipient bar is usually positioned right below
     // the `.header` element (including padding).

static/js/reactions.js

@@ -1,10 +1,10 @@
 import $ from "jquery";
 
-import * as emoji from "../shared/js/emoji";
 import render_message_reaction from "../templates/message_reaction.hbs";
 
 import * as blueslip from "./blueslip";
 import * as channel from "./channel";
+import * as emoji from "./emoji";
 import * as emoji_picker from "./emoji_picker";
 import {$t} from "./i18n";
 import * as message_lists from "./message_lists";

static/js/server_events_dispatch.js

@@ -1,7 +1,5 @@
 import $ from "jquery";
 
-import * as emoji from "../shared/js/emoji";
-
 import * as activity from "./activity";
 import * as alert_words from "./alert_words";
 import * as alert_words_ui from "./alert_words_ui";
@@ -15,6 +13,7 @@ import * as compose_fade from "./compose_fade";
 import * as compose_pm_pill from "./compose_pm_pill";
 import * as composebox_typeahead from "./composebox_typeahead";
 import * as dark_theme from "./dark_theme";
+import * as emoji from "./emoji";
 import * as emoji_picker from "./emoji_picker";
 import * as giphy from "./giphy";
 import * as hotspots from "./hotspots";

static/js/settings.js

@@ -98,6 +98,7 @@ export function build_page() {
         user_can_change_avatar: settings_data.user_can_change_avatar(),
         user_role_text: people.get_user_type(page_params.user_id),
         default_language_name: settings_display.user_default_language_name,
+        realm_push_notifications_enabled: page_params.realm_push_notifications_enabled,
         settings_object: user_settings,
     });
 

static/js/settings_account.js

@@ -170,6 +170,7 @@ export function append_custom_profile_fields(element_id, user_id) {
             is_long_text_field: field.type === all_field_types.LONG_TEXT.id,
             is_user_field: field.type === all_field_types.USER.id,
             is_date_field: field.type === all_field_types.DATE.id,
+            is_url_field: field.type === all_field_types.URL.id,
             is_select_field,
             field_choices,
         });

static/js/settings_bots.js

@@ -362,7 +362,7 @@ export function set_up() {
         $(`[name*='${CSS.escape(selected_bot)}']`).show();
     });
 
-    $("#active_bots_list").on("click", "button.delete_bot", (e) => {
+    $("#active_bots_list").on("click", "button.deactivate_bot", (e) => {
         const bot_id = Number.parseInt($(e.currentTarget).attr("data-user-id"), 10);
 
         channel.del({

static/js/settings_emoji.js

@@ -1,13 +1,13 @@
 import $ from "jquery";
 
 import emoji_codes from "../generated/emoji/emoji_codes.json";
-import * as emoji from "../shared/js/emoji";
 import emoji_settings_warning_modal from "../templates/confirm_dialog/confirm_emoji_settings_warning.hbs";
 import render_admin_emoji_list from "../templates/settings/admin_emoji_list.hbs";
 import render_settings_emoji_settings_tip from "../templates/settings/emoji_settings_tip.hbs";
 
 import * as channel from "./channel";
 import * as confirm_dialog from "./confirm_dialog";
+import * as emoji from "./emoji";
 import {$t_html} from "./i18n";
 import * as ListWidget from "./list_widget";
 import * as loading from "./loading";

static/js/stream_settings_ui.js

@@ -318,6 +318,7 @@ export function update_settings_for_subscribed(slim_sub) {
 
     // Display the swatch and subscription stream_settings
     stream_ui_updates.update_regular_sub_settings(sub);
+    stream_ui_updates.update_permissions_banner(sub);
 }
 
 export function show_active_stream_in_left_panel() {
@@ -350,6 +351,7 @@ export function update_settings_for_unsubscribed(slim_sub) {
 
     // Remove private streams from subscribed streams list.
     stream_ui_updates.update_stream_row_in_settings_tab(sub);
+    stream_ui_updates.update_permissions_banner(sub);
 }
 
 function triage_stream(left_panel_params, sub) {

static/js/stream_ui_updates.js

@@ -1,6 +1,7 @@
 import $ from "jquery";
 
 import render_stream_permission_description from "../templates/stream_settings/stream_permission_description.hbs";
+import render_stream_settings_tip from "../templates/stream_settings/stream_settings_tip.hbs";
 
 import * as hash_util from "./hash_util";
 import {$t} from "./i18n";
@@ -122,6 +123,13 @@ export function update_change_stream_privacy_settings(sub) {
     }
 }
 
+export function update_permissions_banner(sub) {
+    const $settings = $(`.subscription_settings[data-stream-id='${CSS.escape(sub.stream_id)}']`);
+
+    const rendered_tip = render_stream_settings_tip(sub);
+    $settings.find(".stream-settings-tip-container").html(rendered_tip);
+}
+
 export function update_notification_setting_checkbox(notification_name) {
     // This is in the right panel (Personal settings).
     const $stream_row = $("#manage_streams_container .stream-row.active");

static/js/ui_init.js

@@ -3,7 +3,6 @@ import _ from "lodash";
 
 import generated_emoji_codes from "../generated/emoji/emoji_codes.json";
 import generated_pygments_data from "../generated/pygments_data.json";
-import * as emoji from "../shared/js/emoji";
 import * as fenced_code from "../shared/js/fenced_code";
 import render_compose from "../templates/compose.hbs";
 import render_edit_content_button from "../templates/edit_content_button.hbs";
@@ -28,6 +27,7 @@ import * as copy_and_paste from "./copy_and_paste";
 import * as dark_theme from "./dark_theme";
 import * as drafts from "./drafts";
 import * as echo from "./echo";
+import * as emoji from "./emoji";
 import * as emoji_picker from "./emoji_picker";
 import * as emojisets from "./emojisets";
 import * as gear_menu from "./gear_menu";

static/js/user_status.js

@@ -1,7 +1,6 @@
-import * as emoji from "../shared/js/emoji";
-
 import * as blueslip from "./blueslip";
 import * as channel from "./channel";
+import * as emoji from "./emoji";
 import {user_settings} from "./user_settings";
 
 const away_user_ids = new Set();

static/js/user_status_ui.js

@@ -1,10 +1,10 @@
 import $ from "jquery";
 
-import * as emoji from "../shared/js/emoji";
 import render_set_status_overlay from "../templates/set_status_overlay.hbs";
 import render_status_emoji_selector from "../templates/status_emoji_selector.hbs";
 
 import * as dialog_widget from "./dialog_widget";
+import * as emoji from "./emoji";
 import {$t, $t_html} from "./i18n";
 import * as people from "./people";
 import * as user_status from "./user_status";

static/js/zcommand.js

@@ -1,12 +1,11 @@
 import $ from "jquery";
 
-import marked from "../third/marked/lib/marked";
-
 import * as channel from "./channel";
 import * as common from "./common";
 import * as dark_theme from "./dark_theme";
 import * as feedback_widget from "./feedback_widget";
 import {$t} from "./i18n";
+import * as markdown from "./markdown";
 import * as scroll_bar from "./scroll_bar";
 
 /*
@@ -66,7 +65,7 @@ export function switch_to_light_theme() {
             dark_theme.disable();
             feedback_widget.show({
                 populate($container) {
-                    const rendered_msg = marked(data.msg).trim();
+                    const rendered_msg = markdown.parse_non_message(data.msg);
                     $container.html(rendered_msg);
                 },
                 on_undo() {
@@ -88,7 +87,7 @@ export function switch_to_dark_theme() {
             dark_theme.enable();
             feedback_widget.show({
                 populate($container) {
-                    const rendered_msg = marked(data.msg).trim();
+                    const rendered_msg = markdown.parse_non_message(data.msg);
                     $container.html(rendered_msg);
                 },
                 on_undo() {
@@ -110,7 +109,7 @@ export function enter_fluid_mode() {
             scroll_bar.set_layout_width();
             feedback_widget.show({
                 populate($container) {
-                    const rendered_msg = marked(data.msg).trim();
+                    const rendered_msg = markdown.parse_non_message(data.msg);
                     $container.html(rendered_msg);
                 },
                 on_undo() {
@@ -132,7 +131,7 @@ export function enter_fixed_mode() {
             scroll_bar.set_layout_width();
             feedback_widget.show({
                 populate($container) {
-                    const rendered_msg = marked(data.msg).trim();
+                    const rendered_msg = markdown.parse_non_message(data.msg);
                     $container.html(rendered_msg);
                 },
                 on_undo() {

static/shared/js/poll_data.js.flow

@@ -0,0 +1,46 @@
+// @flow strict
+
+/**
+ * The data encoded in a submessage that acts on a poll widget.
+ *
+ * In reality these are more specific than this type.  But they're currently
+ * completely undocumented in the API:
+ *   https://chat.zulip.org/#narrow/stream/412-api-documentation/topic/.60.2Esubmessages.60.20on.20message.20objects/near/1358493
+ * so we don't attempt to go any further here.
+ */
+type PollEvent = {...};
+
+declare export class PollData {
+    constructor({
+        message_sender_id: number,
+        current_user_id: number,
+        is_my_poll: boolean,
+        question: string,
+        options: interface {entries(): Iterable<[number, string]>},
+        comma_separated_names: (user_ids: number[]) => string,
+        report_error_function: (msg: string) => void,
+    }): void;
+
+    set_question(question: string): void;
+
+    get_question(): string;
+
+    set_input_mode(): void;
+
+    clear_input_mode(): void;
+
+    get_input_mode(): boolean;
+
+    get_widget_data(): {
+        question: string,
+        options: Array<{
+            option: string,
+            names: string,
+            count: number,
+            key: string,
+            current_user_vote: boolean,
+        }>,
+    };
+
+    handle_event(sender_id: number, data: PollEvent): void;
+}

static/shared/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zulip/shared",
-  "version": "0.0.9",
+  "version": "0.0.11",
   "license": "Apache-2.0",
   "dependencies": {
     "katex": "^0.15.3",

static/styles/subscriptions.css

@@ -857,6 +857,7 @@ h4.stream_setting_subsection_title {
 
     .stream-header {
         white-space: nowrap;
+        padding-top: 10px;
 
         .stream-name {
             display: inline-block;

static/templates/confirm_dialog/confirm_deactivate_stream.hbs

@@ -1,2 +1,5 @@
-Archiving stream <strong>{{stream_name}}</strong> <span>will immediately unsubscribe everyone. This action cannot be undone.</span>
+{{#tr}}
+    Archiving stream <z-stream></z-stream> will immediately unsubscribe everyone. This action cannot be undone.
+    {{#*inline "z-stream"}}<strong>{{stream_name}}</strong>{{/inline}}
+{{/tr}}
 <p><strong>{{t "Are you sure you want to archive this stream?" }}</strong></p>

static/templates/settings/bot_avatar_row.hbs

@@ -14,8 +14,8 @@
                 <button type="submit" id="copy_zuliprc" class="btn copy_zuliprc" title="{{t 'Copy zuliprc' }}">
                     <i class="fa fa-clipboard copy-gold"></i>
                 </button>
-                <button type="submit" class="btn delete_bot" title="{{t 'Delete bot' }}" data-user-id="{{user_id}}">
-                    <i class="fa fa-trash-o danger-red" aria-hidden="true"></i>
+                <button type="submit" class="btn deactivate_bot danger-red" title="{{t 'Deactivate bot' }}" data-user-id="{{user_id}}">
+                    <i class="fa fa-user-times" aria-hidden="true"></i>
                 </button>
             </div>
             {{/if}}

static/templates/settings/custom_user_profile_field.hbs

@@ -20,6 +20,8 @@
         <input class="custom_user_field_value datepicker" data-field-id="{{ field.id }}" type="text"
           value="{{ field_value.value }}" />
         <span class="remove_date"><i class="fa fa-close"></i></span>
+        {{else if is_url_field }}
+        <input class="custom_user_field_value" type="{{ field_type }}" value="{{ field_value.value }}" maxlength="2048" />
         {{else}}
         <input class="custom_user_field_value" type="{{ field_type }}" value="{{ field_value.value }}" maxlength="50" />
         {{/if}}

static/templates/settings/notification_settings.hbs

@@ -12,7 +12,7 @@
                     <th colspan="2" width="28%">{{t "Desktop"}}</th>
                     <th rowspan="2" width="14%" class="{{#if show_push_notifications_tooltip.push_notifications}}control-label-disabled{{/if}}">
                         {{t "Mobile"}}
-                        {{#if (not page_params.realm_push_notifications_enabled) }}
+                        {{#if (not realm_push_notifications_enabled) }}
                         <i class="fa fa-question-circle settings-info-icon tippy-zulip-tooltip" data-tippy-content="{{t 'Mobile push notifications are not configured on this server.' }}"></i>
                         {{/if}}
                     </th>

static/templates/stream_settings/new_stream_user.hbs

@@ -9,6 +9,6 @@
     {{/if}}
     <td>{{user_id}} </td>
     <td>
-        <button {{#if disabled}} disabled="disabled"{{/if}} data-user-id="{{user_id}}" class="remove_potential_subscriber button small rounded btn-danger">Remove</button>
+        <button {{#if disabled}} disabled="disabled"{{/if}} data-user-id="{{user_id}}" class="remove_potential_subscriber button small rounded btn-danger">{{t 'Remove' }}</button>
     </td>
 </tr>

static/templates/stream_settings/new_stream_users.hbs

@@ -7,7 +7,7 @@
 <button class="add_all_users_to_stream small button rounded sea-green">{{t 'Add all users'}}</button>
 
 <div class="create_stream_subscriber_list_header">
-    <h4 class="stream_setting_subsection_title">Subscribers</h4>
+    <h4 class="stream_setting_subsection_title">{{t 'Subscribers' }}</h4>
     <input class="add-user-list-filter" name="user_list_filter" type="text"
       autocomplete="off" placeholder="{{t 'Filter subscribers' }}" />
 </div>

static/templates/stream_settings/stream_settings.hbs

@@ -23,6 +23,9 @@
 
         <div class="general_settings stream_section">
             {{#with sub}}
+            <div class="stream-settings-tip-container">
+                {{> stream_settings_tip}}
+            </div>
             <div class="stream-header">
                 {{> stream_privacy_icon
                   invite_only=invite_only
@@ -31,7 +34,7 @@
                     <span class="sub-stream-name" title="{{name}}">{{name}}</span>
                 </div>
                 <div class="stream_change_property_info alert-notification"></div>
-                <div class="button-group">
+                <div class="button-group" {{#unless can_change_name_description}}style="display:none"{{/unless}}>
                     <button id="open_stream_info_modal" class="button rounded small btn-warning" title="{{t 'Change stream info' }}">
                         <i class="fa fa-pencil" aria-hidden="true"></i>
                     </button>

static/templates/stream_settings/stream_settings_tip.hbs

@@ -0,0 +1,7 @@
+{{#unless can_change_stream_permissions}}
+    {{#if can_change_name_description}}
+        <div class="tip">{{t "Only subscribers to this stream can edit stream permissions."}}</div>
+    {{else}}
+        <div class="tip">{{t "Only organization administrators can edit these settings."}}</div>
+    {{/if}}
+{{/unless}}

static/third/marked/lib/marked.js

@@ -484,7 +484,6 @@ var inline = {
   stream: noop,
   tex: noop,
   timestamp: noop,
-  linkifiers: [],
   text: /^[\s\S]+?(?=[\\<!\[_*`$]| {2,}\n|$)/
 };
 
@@ -550,7 +549,6 @@ inline.zulip = merge({}, inline.breaks, {
   stream: /^#\*\*([^\*]+)\*\*/,
   tex: /^(\$\$([^\n_$](\\\$|[^\n$])*)\$\$(?!\$))\B/,
   timestamp: /^<time:([^>]+)>/,
-  linkifiers: [],
   text: replace(inline.breaks.text)
     ('|', '|(\ud83c[\udd00-\udfff]|\ud83d[\udc00-\ude4f]|' +
           '\ud83d[\ude80-\udeff]|\ud83e[\udd00-\uddff]|' +
@@ -647,8 +645,10 @@ InlineLexer.prototype.output = function(src) {
 
     // linkifier (Zulip)
     var self = this;
-    this.rules.linkifiers.forEach(function (linkifier) {
-      var ret = self.inlineReplacement(linkifier, src, function(regex, groups, match) {
+
+    const regexes = this.options.get_linkifier_regexes ? this.options.get_linkifier_regexes() : [];
+    regexes.forEach(function (regex) {
+      var ret = self.inlineReplacement(regex, src, function(regex, groups, match) {
         // Insert the created URL
         href = self.linkifier(regex, groups, match);
         if (href !== undefined) {
@@ -1416,9 +1416,6 @@ function marked(src, opt, callback) {
 
     htmlStashCounter = 0;
     htmlStash = [];
-    for (var k = 0; k < opt.preprocessors.length; k++) {
-      src = opt.preprocessors[k](src);
-    }
 
     try {
       tokens = Lexer.lex(src, opt)
@@ -1480,8 +1477,8 @@ function marked(src, opt, callback) {
     if (opt) opt = merge({}, marked.defaults, opt);
     htmlStashCounter = 0;
     htmlStash = [];
-    for (var i = 0; i < marked.defaults.preprocessors.length; i++) {
-      src = marked.defaults.preprocessors[i](src);
+    for (var i = 0; i < opt.preprocessors.length; i++) {
+      src = opt.preprocessors[i](src);
     }
     return Parser.parse(Lexer.lex(src, opt), opt);
   } catch (e) {

templates/zerver/api/changelog.md

@@ -18,6 +18,11 @@ clients should check the `zulip_feature_level` field, present in the
 /register`](/api/register-queue) responses, to determine the API
 format used by the Zulip server that they are interacting with.
 
+## Changes in Zulip 6.0
+
+Feature levels 123-124 are reserved for future use in 5.x maintenance
+releases.
+
 ## Changes in Zulip 5.0
 
 **Feature level 122**

templates/zerver/footer.html

@@ -143,7 +143,7 @@
             <li><a href="/team/">{{ _("Team") }}</a> &amp; <a href="/history/">{{ _("History") }}</a></li>
             <li><a href="https://twitter.com/zulip/">Twitter</a></li>
             <li><a href="/jobs/">{{ _("Jobs") }}</a></li>
-            <li><a href="/attribution">Website attributions</a></li>
+            <li><a href="/attribution">{{ _("Website attributions") }}</a></li>
             <li><a href="https://github.com/sponsors/zulip">{{ _("Sponsor Zulip") }}</a></li>
         </ul>
     </div>

tools/semgrep.yml

@@ -39,7 +39,7 @@ rules:
         - zerver/migrations/0206_stream_rendered_description.py
         - zerver/migrations/0209_user_profile_no_empty_password.py
         - zerver/migrations/0260_missed_message_addresses_from_redis_to_db.py
-        - zerver/migrations/0376_set_realmemoji_author_and_reupload_realmemoji.py
+        - zerver/migrations/0387_reupload_realmemoji_again.py
         - pgroonga/migrations/0002_html_escape_subject.py
 
   - id: logging-format

tools/total-contributions

@@ -7,6 +7,8 @@ import sys
 from collections import defaultdict
 from typing import Dict, List
 
+bot_commits = 0
+
 
 def add_log(committer_dict: Dict[str, int], input: List[str]) -> None:
     for dataset in input:
@@ -15,6 +17,8 @@ def add_log(committer_dict: Dict[str, int], input: List[str]) -> None:
 
         if committer_name.endswith("[bot]"):
             # Exclude dependabot[bot] and other GitHub bots.
+            global bot_commits
+            bot_commits += commit_count
             continue
 
         committer_dict[committer_name] += commit_count
@@ -131,11 +135,21 @@ print(
     f"Commit range {lower_zulip_version}..{upper_zulip_version} corresponds to {lower_time} to {upper_time}"
 )
 
+repository_dict: Dict[str, int] = defaultdict(int)
 out_dict: Dict[str, int] = defaultdict(int)
 subprocess.check_call(["git", "fetch"], cwd=find_path("zulip"))
-zulip = retrieve_log("zulip", lower_zulip_version, upper_zulip_version)
-print(f"Commit range for zulip/zulip: {lower_zulip_version[0:12]}..{upper_zulip_version[0:12]}")
-add_log(out_dict, zulip)
+commit_count = len(
+    subprocess.check_output(
+        ["git", "log", "--pretty=oneline", f"{lower_zulip_version}..{upper_zulip_version}"],
+        cwd=find_path("zulip"),
+        text=True,
+    ).splitlines()
+)
+repo_log = retrieve_log("zulip", lower_zulip_version, upper_zulip_version)
+print(
+    f"{commit_count} commits from zulip/zulip: {lower_zulip_version[0:12]}..{upper_zulip_version[0:12]}"
+)
+add_log(out_dict, repo_log)
 
 # TODO: We should migrate the last couple repositories to use the
 # `main` default branch name and then simplify this.
@@ -163,9 +177,16 @@ for (full_repository, branch) in [
     subprocess.check_call(["git", "fetch"], cwd=find_path(repository))
     lower_repo_version = find_last_commit_before_time(repository, branch, lower_time)
     upper_repo_version = find_last_commit_before_time(repository, branch, upper_time)
+    commit_count = len(
+        subprocess.check_output(
+            ["git", "log", "--pretty=oneline", f"{lower_repo_version}..{upper_repo_version}"],
+            cwd=find_path(repository),
+            text=True,
+        ).splitlines()
+    )
     repo_log = retrieve_log(repository, lower_repo_version, upper_repo_version)
     print(
-        f"Commit range for {full_repository}: {lower_repo_version[0:12]}..{upper_repo_version[0:12]}"
+        f"{commit_count} commits from {full_repository}: {lower_repo_version[0:12]}..{upper_repo_version[0:12]}"
     )
     add_log(out_dict, repo_log)
 
@@ -177,7 +198,8 @@ for committer_name, commit_count in sorted(
     print(str(commit_count) + "\t" + committer_name)
     grand_total += commit_count
 
+print(f"Excluded {bot_commits} commits authored by bots.")
 print(
     f"{grand_total} total commits by {len(out_dict)} contributors between "
-    f"{lower_zulip_version} and {upper_repo_version}."
+    f"{lower_zulip_version} and {upper_zulip_version}."
 )

version.py

@@ -1,6 +1,6 @@
 import os
 
-ZULIP_VERSION = "5.0"
+ZULIP_VERSION = "6.0-dev+git"
 
 # Add information on number of commits and commit hash to version, if available
 zulip_git_version_file = os.path.join(
@@ -14,8 +14,8 @@ ZULIP_VERSION = lines.pop(0).strip()
 ZULIP_MERGE_BASE = lines.pop(0).strip()
 
 LATEST_MAJOR_VERSION = "5.0"
-LATEST_RELEASE_VERSION = "5.0"
-LATEST_RELEASE_ANNOUNCEMENT = "https://blog.zulip.com/2021/05/13/zulip-4-0-released/"
+LATEST_RELEASE_VERSION = "5.1"
+LATEST_RELEASE_ANNOUNCEMENT = "https://blog.zulip.com/2022/03/29/zulip-5-0-released/"
 
 # Versions of the desktop app below DESKTOP_MINIMUM_VERSION will be
 # prevented from connecting to the Zulip server.  Versions above

zerver/data_import/slack.py

@@ -78,14 +78,13 @@ class SlackBotEmail:
         else:
             raise AssertionError("Could not identify bot type")
 
-        email = slack_bot_name.replace("Bot", "").replace(" ", "") + f"-bot@{domain_name}"
+        email = slack_bot_name.replace("Bot", "").replace(" ", "").lower() + f"-bot@{domain_name}"
 
         if email in cls.duplicate_email_count:
-            email_prefix, email_suffix = email.split("@")
-            email_prefix += cls.duplicate_email_count[email]
-            email = "@".join([email_prefix, email_suffix])
-            # Increment the duplicate email count
             cls.duplicate_email_count[email] += 1
+            email_prefix, email_suffix = email.split("@")
+            email_prefix += "-" + str(cls.duplicate_email_count[email])
+            email = "@".join([email_prefix, email_suffix])
         else:
             cls.duplicate_email_count[email] = 1
 

zerver/migrations/0376_set_realmemoji_author_and_reupload_realmemoji.py

@@ -1,10 +1,7 @@
-from django.conf import settings
 from django.db import migrations
 from django.db.backends.postgresql.schema import DatabaseSchemaEditor
 from django.db.migrations.state import StateApps
 
-from zerver.lib.queue import queue_json_publish
-
 
 def set_emoji_author(apps: StateApps, schema_editor: DatabaseSchemaEditor) -> None:
     """
@@ -13,7 +10,6 @@ def set_emoji_author(apps: StateApps, schema_editor: DatabaseSchemaEditor) -> No
     """
 
     RealmEmoji = apps.get_model("zerver", "RealmEmoji")
-    Realm = apps.get_model("zerver", "Realm")
     UserProfile = apps.get_model("zerver", "UserProfile")
     ROLE_REALM_OWNER = 100
 
@@ -32,18 +28,12 @@ def set_emoji_author(apps: StateApps, schema_editor: DatabaseSchemaEditor) -> No
 
     RealmEmoji.objects.bulk_update(realm_emoji_to_update, ["author_id"])
 
-    if settings.TEST_SUITE:
-        # There are no custom emoji in the test suite data set, and
-        # the below code won't work because RabbitMQ isn't enabled for
-        # the test suite.
-        return
-
-    for realm_id in Realm.objects.order_by("id").values_list("id", flat=True):
-        event = {
-            "type": "reupload_realm_emoji",
-            "realm_id": realm_id,
-        }
-        queue_json_publish("deferred_work", event)
+    # Previously, this also pushed `reupload_realm_emoji` events onto
+    # the `deferred_work` queue; however,
+    # https://github.com/zulip/zulip/issues/21608 made those possibly
+    # run too early, and that work was repeated in migration 0387 to
+    # ensure it ran.  As such, the work has been removed from this
+    # migration, so it does not unnecessarily run twice.
 
 
 class Migration(migrations.Migration):

zerver/migrations/0387_reupload_realmemoji_again.py

@@ -0,0 +1,48 @@
+from django.conf import settings
+from django.db import migrations
+from django.db.backends.postgresql.schema import DatabaseSchemaEditor
+from django.db.migrations.state import StateApps
+
+from zerver.lib.queue import queue_json_publish
+
+
+def reupload_realm_emoji(apps: StateApps, schema_editor: DatabaseSchemaEditor) -> None:
+    """As detailed in https://github.com/zulip/zulip/issues/21608, it is
+    possible for the deferred_work queue from Zulip 4.x to have been
+    started up by puppet during the deployment before migrations were
+    run on Zulip 5.0.
+
+    This means that the deferred_work events originally produced by
+    migration 0376 might have been processed and discarded without
+    effect.
+
+    That code has been removed from the 0376 migration, and we run it
+    here, after the upgrade code has been fixed; servers which already
+    processed that migration might at worst do this work twice, which
+    is harmless aside from being a small waste of resources.
+    """
+
+    Realm = apps.get_model("zerver", "Realm")
+    if settings.TEST_SUITE:
+        # There are no custom emoji in the test suite data set, and
+        # the below code won't work because RabbitMQ isn't enabled for
+        # the test suite.
+        return
+
+    for realm_id in Realm.objects.order_by("id").values_list("id", flat=True):
+        event = {
+            "type": "reupload_realm_emoji",
+            "realm_id": realm_id,
+        }
+        queue_json_publish("deferred_work", event)
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("zerver", "0386_fix_attachment_caches"),
+    ]
+
+    operations = [
+        migrations.RunPython(reupload_realm_emoji, reverse_code=migrations.RunPython.noop),
+    ]

zerver/tests/test_message_edit.py

@@ -322,6 +322,7 @@ class EditMessageTest(EditMessageTestCase):
         self.assert_json_success(result)
         self.assertEqual(result.json()["raw_content"], "Personal message")
         self.assertEqual(result.json()["message"]["id"], msg_id)
+        self.assertEqual(result.json()["message"]["flags"], [])
 
         # Send message to web public stream where hamlet is not subscribed.
         # This will test case of user having no `UserMessage` but having access
@@ -335,6 +336,7 @@ class EditMessageTest(EditMessageTestCase):
         self.assert_json_success(result)
         self.assertEqual(result.json()["raw_content"], "web-public message")
         self.assertEqual(result.json()["message"]["id"], web_public_stream_msg_id)
+        self.assertEqual(result.json()["message"]["flags"], ["read", "historical"])
 
         # Spectator should be able to fetch message in web public stream.
         self.logout()
@@ -416,6 +418,7 @@ class EditMessageTest(EditMessageTestCase):
         result = self.client_get("/json/messages/" + str(web_public_stream_msg_id))
         self.assert_json_success(result)
         self.assertEqual(result.json()["raw_content"], "web-public message")
+        self.assertEqual(result.json()["message"]["flags"], ["read"])
 
         # Verify LIMITED plan type does not allow web-public access.
         do_change_realm_plan_type(user_profile.realm, Realm.PLAN_TYPE_LIMITED, acting_user=None)

zerver/tests/test_mirror_users.py

@@ -1,4 +1,3 @@
-from collections import namedtuple
 from typing import Any, List
 from unittest import mock
 
@@ -8,33 +7,23 @@ from django.utils.timezone import now as timezone_now
 from zerver.lib.actions import create_mirror_user_if_needed
 from zerver.lib.create_user import create_user_profile
 from zerver.lib.test_classes import ZulipTestCase
-from zerver.lib.test_helpers import HostRequestMock, reset_emails_in_zulip_realm
-from zerver.models import UserProfile, get_realm, get_user
+from zerver.lib.test_helpers import reset_emails_in_zulip_realm
+from zerver.models import UserProfile, get_client, get_realm, get_user
 from zerver.views.message_send import InvalidMirrorInput, create_mirrored_message_users
 
 
 class MirroredMessageUsersTest(ZulipTestCase):
-    def test_invalid_sender(self) -> None:
-        user = self.example_user("hamlet")
-        recipients: List[str] = []
-
-        Request = namedtuple("Request", ["POST"])
-        request = Request(POST={})  # no sender
-
-        with self.assertRaises(InvalidMirrorInput):
-            create_mirrored_message_users(request, user, recipients)
-
     def test_invalid_client(self) -> None:
         user = self.example_user("hamlet")
         sender = user
 
         recipients: List[str] = []
 
-        post_data = dict(sender=sender.email, type="private")
-        request = HostRequestMock(post_data=post_data, client_name="banned_mirror")
+        message_type = "private"
+        client = get_client("banned_mirror")
 
         with self.assertRaises(InvalidMirrorInput):
-            create_mirrored_message_users(request, user, recipients)
+            create_mirrored_message_users(client, user, recipients, sender.email, message_type)
 
     def test_invalid_email(self) -> None:
         invalid_email = "alice AT example.com"
@@ -44,13 +33,13 @@ class MirroredMessageUsersTest(ZulipTestCase):
         user = self.mit_user("starnine")
         sender = user
 
-        post_data = dict(sender=sender.email, type="private")
+        message_type = "private"
 
         for client_name in ["zephyr_mirror", "irc_mirror", "jabber_mirror"]:
-            request = HostRequestMock(post_data=post_data, client_name=client_name)
+            client = get_client(client_name)
 
             with self.assertRaises(InvalidMirrorInput):
-                create_mirrored_message_users(request, user, recipients)
+                create_mirrored_message_users(client, user, recipients, sender.email, message_type)
 
     @mock.patch(
         "DNS.dnslookup",
@@ -65,11 +54,12 @@ class MirroredMessageUsersTest(ZulipTestCase):
 
         recipients = [user.email, new_user_email]
 
-        # Now make the request.
-        post_data = dict(sender=sender.email, type="private")
-        request = HostRequestMock(post_data=post_data, client_name="zephyr_mirror")
+        message_type = "private"
+        client = get_client("zephyr_mirror")
 
-        mirror_sender = create_mirrored_message_users(request, user, recipients)
+        mirror_sender = create_mirrored_message_users(
+            client, user, recipients, sender.email, message_type
+        )
 
         self.assertEqual(mirror_sender, sender)
 
@@ -92,11 +82,12 @@ class MirroredMessageUsersTest(ZulipTestCase):
 
         recipients = ["stream_name"]
 
-        # Now make the request.
-        post_data = dict(sender=sender_email, type="stream")
-        request = HostRequestMock(post_data=post_data, client_name="zephyr_mirror")
+        message_type = "stream"
+        client = get_client("zephyr_mirror")
 
-        mirror_sender = create_mirrored_message_users(request, user, recipients)
+        mirror_sender = create_mirrored_message_users(
+            client, user, recipients, sender_email, message_type
+        )
 
         assert mirror_sender is not None
         self.assertEqual(mirror_sender.email, sender_email)
@@ -105,7 +96,8 @@ class MirroredMessageUsersTest(ZulipTestCase):
     def test_irc_mirror(self) -> None:
         reset_emails_in_zulip_realm()
 
-        sender = self.example_user("hamlet")
+        user = self.example_user("hamlet")
+        sender = user
 
         recipients = [
             self.nonreg_email("alice"),
@@ -113,11 +105,12 @@ class MirroredMessageUsersTest(ZulipTestCase):
             self.nonreg_email("cordelia"),
         ]
 
-        # Now make the request.
-        post_data = dict(sender=sender.email, type="private")
-        request = HostRequestMock(post_data=post_data, client_name="irc_mirror")
+        message_type = "private"
+        client = get_client("irc_mirror")
 
-        mirror_sender = create_mirrored_message_users(request, sender, recipients)
+        mirror_sender = create_mirrored_message_users(
+            client, user, recipients, sender.email, message_type
+        )
 
         self.assertEqual(mirror_sender, sender)
 
@@ -132,8 +125,8 @@ class MirroredMessageUsersTest(ZulipTestCase):
     def test_jabber_mirror(self) -> None:
         reset_emails_in_zulip_realm()
 
-        sender = self.example_user("hamlet")
-        user = sender
+        user = self.example_user("hamlet")
+        sender = user
 
         recipients = [
             self.nonreg_email("alice"),
@@ -141,11 +134,12 @@ class MirroredMessageUsersTest(ZulipTestCase):
             self.nonreg_email("cordelia"),
         ]
 
-        # Now make the request.
-        post_data = dict(sender=sender.email, type="private")
-        request = HostRequestMock(post_data=post_data, client_name="jabber_mirror")
+        message_type = "private"
+        client = get_client("jabber_mirror")
 
-        mirror_sender = create_mirrored_message_users(request, user, recipients)
+        mirror_sender = create_mirrored_message_users(
+            client, user, recipients, sender.email, message_type
+        )
 
         self.assertEqual(mirror_sender, sender)
 

zerver/tests/test_slack_importer.py

@@ -25,6 +25,7 @@ from zerver.data_import.slack import (
     AddedChannelsT,
     AddedMPIMsT,
     DMMembersT,
+    SlackBotEmail,
     channel_message_to_zerver_message,
     channels_to_zerver_stream,
     convert_slack_workspace_messages,
@@ -1213,3 +1214,38 @@ class SlackImporter(ZulipTestCase):
         self.assertEqual(uploads_list[0]["s3_path"], image_path)
         self.assertEqual(uploads_list[0]["realm_id"], realm_id)
         self.assertEqual(uploads_list[0]["user_profile_email"], "alice@example.com")
+
+    def test_bot_duplicates(self) -> None:
+        self.assertEqual(
+            SlackBotEmail.get_email(
+                {"real_name_normalized": "Real Bot", "bot_id": "foo"}, "example.com"
+            ),
+            "real-bot@example.com",
+        )
+
+        # SlackBotEmail keeps state -- doing it again appends a "2", "3", etc
+        self.assertEqual(
+            SlackBotEmail.get_email(
+                {"real_name_normalized": "Real Bot", "bot_id": "bar"}, "example.com"
+            ),
+            "real-bot-2@example.com",
+        )
+        self.assertEqual(
+            SlackBotEmail.get_email(
+                {"real_name_normalized": "Real Bot", "bot_id": "baz"}, "example.com"
+            ),
+            "real-bot-3@example.com",
+        )
+
+        # But caches based on the bot_id
+        self.assertEqual(
+            SlackBotEmail.get_email(
+                {"real_name_normalized": "Real Bot", "bot_id": "foo"}, "example.com"
+            ),
+            "real-bot@example.com",
+        )
+
+        self.assertEqual(
+            SlackBotEmail.get_email({"first_name": "Other Name", "bot_id": "other"}, "example.com"),
+            "othername-bot@example.com",
+        )

zerver/tests/test_subs.py

@@ -978,7 +978,6 @@ class StreamAdminTest(ZulipTestCase):
         self.assertTrue(attachment.is_realm_public)
 
         params = {
-            "stream_name": orjson.dumps("test_stream").decode(),
             "is_private": orjson.dumps(True).decode(),
             "history_public_to_subscribers": orjson.dumps(True).decode(),
         }
@@ -1000,7 +999,6 @@ class StreamAdminTest(ZulipTestCase):
         self.assertFalse(validate_attachment_request_for_spectator_access(realm, attachment))
 
         params = {
-            "stream_name": orjson.dumps("test_stream").decode(),
             "is_private": orjson.dumps(False).decode(),
             "is_web_public": orjson.dumps(True).decode(),
             "history_public_to_subscribers": orjson.dumps(True).decode(),
@@ -1025,7 +1023,6 @@ class StreamAdminTest(ZulipTestCase):
         self.assertTrue(attachment.is_realm_public)
 
         params = {
-            "stream_name": orjson.dumps("test_stream").decode(),
             "is_private": orjson.dumps(False).decode(),
             "is_web_public": orjson.dumps(False).decode(),
             "history_public_to_subscribers": orjson.dumps(True).decode(),

zerver/views/message_edit.py

@@ -205,6 +205,8 @@ def json_fetch_raw_message(
     else:
         if user_message:
             flags = user_message.flags_list()
+        else:
+            flags = ["read", "historical"]
         allow_edit_history = maybe_user_profile.realm.allow_edit_history
 
     # Security note: It's important that we call this only with a

zerver/views/message_send.py

@@ -42,20 +42,19 @@ class InvalidMirrorInput(Exception):
 
 
 def create_mirrored_message_users(
-    request: HttpRequest, user_profile: UserProfile, recipients: Iterable[str]
+    client: Client,
+    user_profile: UserProfile,
+    recipients: Iterable[str],
+    sender: str,
+    message_type: str,
 ) -> UserProfile:
-    if "sender" not in request.POST:
-        raise InvalidMirrorInput("No sender")
 
-    sender_email = request.POST["sender"].strip().lower()
+    sender_email = sender.strip().lower()
     referenced_users = {sender_email}
-    if request.POST["type"] == "private":
+    if message_type == "private":
         for email in recipients:
             referenced_users.add(email.lower())
 
-    client = RequestNotes.get_notes(request).client
-    assert client is not None
-
     if client.name == "zephyr_mirror":
         user_check = same_realm_zephyr_user
         fullname_function = compute_mit_user_fullname
@@ -188,6 +187,7 @@ def send_message_backend(
     user_profile: UserProfile,
     message_type_name: str = REQ("type"),
     req_to: Optional[str] = REQ("to", default=None),
+    req_sender: Optional[str] = REQ("sender", default=None, documentation_pending=True),
     forged_str: Optional[str] = REQ("forged", default=None, documentation_pending=True),
     topic_name: Optional[str] = REQ_topic(),
     message_content: str = REQ("content"),
@@ -252,7 +252,7 @@ def send_message_backend(
         # The most important security checks are in
         # `create_mirrored_message_users` below, which checks the
         # same-realm constraint.
-        if "sender" not in request.POST:
+        if req_sender is None:
             raise JsonableError(_("Missing sender"))
         if message_type_name != "private" and not can_forge_sender:
             raise JsonableError(_("User not authorized for this query"))
@@ -268,7 +268,9 @@ def send_message_backend(
         message_to = cast(Sequence[str], message_to)
 
         try:
-            mirror_sender = create_mirrored_message_users(request, user_profile, message_to)
+            mirror_sender = create_mirrored_message_users(
+                client, user_profile, message_to, req_sender, message_type_name
+            )
         except InvalidMirrorInput:
             raise JsonableError(_("Invalid mirrored message"))
 
@@ -276,7 +278,7 @@ def send_message_backend(
             raise JsonableError(_("Zephyr mirroring is not allowed in this organization"))
         sender = mirror_sender
     else:
-        if "sender" in request.POST:
+        if req_sender is not None:
             raise JsonableError(_("Invalid mirrored message"))
         sender = user_profile