diff --git a/docs/overview/architecture-overview.md b/docs/overview/architecture-overview.md index 569fe62bdb..95aa86408b 100644 --- a/docs/overview/architecture-overview.md +++ b/docs/overview/architecture-overview.md @@ -115,7 +115,7 @@ For more details on the frontend, see our documentation on [translation](../translating/translating.md), [templates](../subsystems/html-css.html#html-templates), [directory structure](../overview/directory-structure.md), and -[the static asset pipeline](../subsystems/front-end-build-process.md). +[the static asset pipeline](../subsystems/html-css.html#static-asset-pipeline). [Jinja2]: http://jinja.pocoo.org/ [Handlebars]: http://handlebarsjs.com/ diff --git a/docs/subsystems/dependencies.md b/docs/subsystems/dependencies.md index 7c9a89a97d..f8ea226c0d 100644 --- a/docs/subsystems/dependencies.md +++ b/docs/subsystems/dependencies.md @@ -248,8 +248,8 @@ reasoning here. dependencies in the `yarn.lock` file; `yarn install` updates the `yarn.lock` files. * `tools/update-prod-static`. This process is discussed in detail in - the [static asset pipeline](../subsystems/front-end-build-process.md) article, - but we don't use the `node_modules` directories directly in + the [static asset pipeline](../subsystems/html-css.html#static-asset-pipeline) + article, but we don't use the `node_modules` directories directly in production. Instead, static assets are compiled using our static asset pipeline and it is the compiled assets that are served directly to users. As a result, we don't ship the `node_modules` diff --git a/docs/subsystems/front-end-build-process.md b/docs/subsystems/front-end-build-process.md deleted file mode 100644 index fb2dc47323..0000000000 --- a/docs/subsystems/front-end-build-process.md +++ /dev/null @@ -1,131 +0,0 @@ -# Static asset pipeline - -This page documents additional information that may be useful when -developing new features for Zulip that require front-end changes, -especially those that involve adding new files. For a more general -overview, see the [new feature tutorial](../tutorials/new-feature-tutorial.md). - -Our [dependencies documentation](../subsystems/dependencies.md) has useful -relevant background as well. - -## Primary build process - -Most of the existing JS in Zulip is written in -[IIFE](https://www.google.com/#q=iife)-wrapped modules, one per file -in the `static/js` directory. We will over time migrate these to -Typescript modules. Stylesheets are written in the Sass extension of -CSS (with the scss syntax), they are converted from plain CSS and we -have yet to take full advantage of the features Sass offers. We use -Webpack to transpile and build JS and CSS bundles that the browser can -understand, one for each entry points specifed in -`tools/webpack.assets.json`; source maps are generated in the process -for better debugging experience. - -In development mode, bundles are built and served on the fly using -webpack-dev-server with live reloading. In production mode (and when creating a -release tarball using `tools/build-release-tarball`), the -`tools/update-prod-static` tool (called by both `tools/build-release-tarball` -and `tools/upgrade-zulip-from-git`) is responsible for orchestrating the -webpack build, JS minification and a host of other steps for getting the assets -ready for deployment. - -You can trace which source files are included in which HTML templates -by comparing the `render_bundle` calls in the HTML templates under -`templates/` with the bundles declared in `tools/webpack.assets.json`. - -## Adding static files - -To add a static file to the app (JavaScript, TypeScript, CSS/Sass, images, etc), -first add it to the appropriate place under `static/`. - -- Third-party packages from the NPM repository should be added to - `package.json` for management by yarn, this allows them to be upgraded easily - and not bloat our codebase. Run `./tools/provision` for yarn to install the - new packages and update its lock file. You should also update - `PROVISION_VERSION` in `version.py` in the same commit. When adding modules - to `package.json`, please pin specific versions of them (don't using carets - `^`, tildes `~`, etc). We prefer fixed versions so that when the upstream - providers release new versions with incompatible APIs, it can't break Zulip. - We update those versions periodically to ensure we're running a recent - version of third-party libraries. -- Third-party files that we have patched should all go in - `static/third/`. Tag the commit with "[third]" when adding or - modifying a third-party package. Our goal is to the extent possible - to eliminate patched third-party code from the project. -- Our own JavaScript and TypeScript files live under `static/js`. Ideally, - new modules should be written in TypeScript (details on this policy below). -- CSS/Sass files lives under `static/styles`. -- Portico JavaScript ("portico" means for logged-out pages) lives under - `static/js/portico`. -- Custom SVG graphics living under `static/assets/icons` are compiled into - custom icon webfonts by webfont-loader according to the - `static/assets/icons/template.hbs` template. - -For your asset to be included in a development/production bundle, it -needs to be accessible from one of the entry points defined in -`tools/webpack.assets.json`. - -* If you plan to only use the file within the app proper, and not on the login - page or other standalone pages, put it in the `app` bundle by importing it - in `static/js/bundles/app.js`. -* If it needs to be available both in the app and all - logged-out/portico pages, import it to - `static/js/bundles/common.js` which itself is imported to the - `app` and `common` bundles. -* If it's just used on a single standalone page (e.g. `/stats`), - create a new entry point in `tools/webpack.assets.json`. Use the - `bundle` macro (defined in `templates/zerver/base.html`) in the - relevant Jinja2 template to inject the compiled JS and CSS. - -If you want to test minified files in development, look for the -`DEBUG =` line in `zproject/settings.py` and set it to `False`. - -## How it works in production - -A few useful notes are: -* Zulip installs static assets in production in -`/home/zulip/prod-static`. When a new version is deployed, before the -server is restarted, files are copied into that directory. -* We use the VFL (Versioned File Layout) strategy, where each file in - the codebase (e.g. `favicon.ico`) gets a new name - (e.g. `favicon.c55d45ae8c58.ico`) that contains a hash in it. Each - deployment, has a manifest file - (e.g. `/home/zulip/deployments/current/staticfiles.json`) that maps - codebase filenames to serving filenames for that deployment. The - benefit of this VFL approach is that all the static files for past - deployments can coexist, which in turn eliminates most classes of - race condition bugs where browser windows opened just before a - deployment can't find their static assets. It also is necessary for - any incremental rollout strategy where different clients get - different versions of the site. -* Some paths for files (e.g. emoji) are stored in the - `rendered_content` of past messages, and thus cannot be removed - without breaking the rendering of old messages (or doing a - mass-rerender of old messages). - -## CommonJS/Typescript modules - -Webpack provides seemless interoperability between different module -systems such as CommonJS, AMD and ES6. Our JS files are written in the -CommonJS format, which specifies public functions and variables as -properties of the special `module.exports` object. We also currently -assign said object to the global `window` variable, which is a hack -allowing us to use modules without importing them with the `require()` -statement. - -New modules should ideally be written in TypeScript (though in cases -where one is moving code from an existing JavaScript module, the new -commit should just move the code, not translate it to TypeScript). - -TypeScript provides more accurate information to development tools, -allowing for better refactoring, auto-completion and static -analysis. TypeScript uses an ES6-like module system. Any declaration -can be made public by adding the `export` keyword. Consuming -variables, functions, etc exported from another module should be done -with the `import` statement as oppose to accessing them from the -global `window` scope. Internally our typescript compiler is -configured to transpile TS to the ES6 module system. - -Read more about these module systems here: -* [TypeScript modules](https://www.typescriptlang.org/docs/handbook/modules.html) -* [CommonJS](https://nodejs.org/api/modules.html#modules_modules) diff --git a/docs/subsystems/html-css.md b/docs/subsystems/html-css.md index de5e04ea11..f79a408da8 100644 --- a/docs/subsystems/html-css.md +++ b/docs/subsystems/html-css.md @@ -134,6 +134,138 @@ whenever a template is changed. All user-facing strings (excluding pages only visible to sysadmins or developers) should be tagged for [translation][]. +## Static asset pipeline + +This section documents additional information that may be useful when +developing new features for Zulip that require front-end changes, +especially those that involve adding new files. For a more general +overview, see the [new feature tutorial](../tutorials/new-feature-tutorial.md). + +Our [dependencies documentation](../subsystems/dependencies.md) has useful +relevant background as well. + +### Primary build process + +Most of the existing JS in Zulip is written in +[IIFE](https://www.google.com/#q=iife)-wrapped modules, one per file +in the `static/js` directory. We will over time migrate these to +Typescript modules. Stylesheets are written in the Sass extension of +CSS (with the scss syntax), they are converted from plain CSS and we +have yet to take full advantage of the features Sass offers. We use +Webpack to transpile and build JS and CSS bundles that the browser can +understand, one for each entry points specifed in +`tools/webpack.assets.json`; source maps are generated in the process +for better debugging experience. + +In development mode, bundles are built and served on the fly using +webpack-dev-server with live reloading. In production mode (and when creating a +release tarball using `tools/build-release-tarball`), the +`tools/update-prod-static` tool (called by both `tools/build-release-tarball` +and `tools/upgrade-zulip-from-git`) is responsible for orchestrating the +webpack build, JS minification and a host of other steps for getting the assets +ready for deployment. + +You can trace which source files are included in which HTML templates +by comparing the `render_bundle` calls in the HTML templates under +`templates/` with the bundles declared in `tools/webpack.assets.json`. + +### Adding static files + +To add a static file to the app (JavaScript, TypeScript, CSS/Sass, images, etc), +first add it to the appropriate place under `static/`. + +- Third-party packages from the NPM repository should be added to + `package.json` for management by yarn, this allows them to be upgraded easily + and not bloat our codebase. Run `./tools/provision` for yarn to install the + new packages and update its lock file. You should also update + `PROVISION_VERSION` in `version.py` in the same commit. When adding modules + to `package.json`, please pin specific versions of them (don't using carets + `^`, tildes `~`, etc). We prefer fixed versions so that when the upstream + providers release new versions with incompatible APIs, it can't break Zulip. + We update those versions periodically to ensure we're running a recent + version of third-party libraries. +- Third-party files that we have patched should all go in + `static/third/`. Tag the commit with "[third]" when adding or + modifying a third-party package. Our goal is to the extent possible + to eliminate patched third-party code from the project. +- Our own JavaScript and TypeScript files live under `static/js`. Ideally, + new modules should be written in TypeScript (details on this policy below). +- CSS/Sass files lives under `static/styles`. +- Portico JavaScript ("portico" means for logged-out pages) lives under + `static/js/portico`. +- Custom SVG graphics living under `static/assets/icons` are compiled into + custom icon webfonts by webfont-loader according to the + `static/assets/icons/template.hbs` template. + +For your asset to be included in a development/production bundle, it +needs to be accessible from one of the entry points defined in +`tools/webpack.assets.json`. + +* If you plan to only use the file within the app proper, and not on the login + page or other standalone pages, put it in the `app` bundle by importing it + in `static/js/bundles/app.js`. +* If it needs to be available both in the app and all + logged-out/portico pages, import it to + `static/js/bundles/common.js` which itself is imported to the + `app` and `common` bundles. +* If it's just used on a single standalone page (e.g. `/stats`), + create a new entry point in `tools/webpack.assets.json`. Use the + `bundle` macro (defined in `templates/zerver/base.html`) in the + relevant Jinja2 template to inject the compiled JS and CSS. + +If you want to test minified files in development, look for the +`DEBUG =` line in `zproject/settings.py` and set it to `False`. + +### How it works in production + +A few useful notes are: +* Zulip installs static assets in production in +`/home/zulip/prod-static`. When a new version is deployed, before the +server is restarted, files are copied into that directory. +* We use the VFL (Versioned File Layout) strategy, where each file in + the codebase (e.g. `favicon.ico`) gets a new name + (e.g. `favicon.c55d45ae8c58.ico`) that contains a hash in it. Each + deployment, has a manifest file + (e.g. `/home/zulip/deployments/current/staticfiles.json`) that maps + codebase filenames to serving filenames for that deployment. The + benefit of this VFL approach is that all the static files for past + deployments can coexist, which in turn eliminates most classes of + race condition bugs where browser windows opened just before a + deployment can't find their static assets. It also is necessary for + any incremental rollout strategy where different clients get + different versions of the site. +* Some paths for files (e.g. emoji) are stored in the + `rendered_content` of past messages, and thus cannot be removed + without breaking the rendering of old messages (or doing a + mass-rerender of old messages). + +### CommonJS/Typescript modules + +Webpack provides seemless interoperability between different module +systems such as CommonJS, AMD and ES6. Our JS files are written in the +CommonJS format, which specifies public functions and variables as +properties of the special `module.exports` object. We also currently +assign said object to the global `window` variable, which is a hack +allowing us to use modules without importing them with the `require()` +statement. + +New modules should ideally be written in TypeScript (though in cases +where one is moving code from an existing JavaScript module, the new +commit should just move the code, not translate it to TypeScript). + +TypeScript provides more accurate information to development tools, +allowing for better refactoring, auto-completion and static +analysis. TypeScript uses an ES6-like module system. Any declaration +can be made public by adding the `export` keyword. Consuming +variables, functions, etc exported from another module should be done +with the `import` statement as oppose to accessing them from the +global `window` scope. Internally our typescript compiler is +configured to transpile TS to the ES6 module system. + +Read more about these module systems here: +* [TypeScript modules](https://www.typescriptlang.org/docs/handbook/modules.html) +* [CommonJS](https://nodejs.org/api/modules.html#modules_modules) + [Jinja2]: http://jinja.pocoo.org/ [Handlebars]: http://handlebarsjs.com/ [trans]: http://jinja.pocoo.org/docs/dev/templates/#i18n diff --git a/docs/subsystems/index.rst b/docs/subsystems/index.rst index c64329fc64..0c9f560ffd 100644 --- a/docs/subsystems/index.rst +++ b/docs/subsystems/index.rst @@ -17,7 +17,6 @@ Subsystems Documentation caching realms management-commands - front-end-build-process schema-migrations migration-renumbering hashchange-system diff --git a/tools/linter_lib/custom_check.py b/tools/linter_lib/custom_check.py index a9fc0e74ef..c2848fc963 100644 --- a/tools/linter_lib/custom_check.py +++ b/tools/linter_lib/custom_check.py @@ -617,7 +617,7 @@ html_rules = whitespace_rules + prose_style_rules + [ 'good_lines': ['']}, {'pattern': 'script src="http', - 'description': "Don't directly load dependencies from CDNs. See docs/subsystems/front-end-build-process.md", + 'description': "Don't directly load dependencies from CDNs. See docs/subsystems/html-css.md", 'exclude': set(["templates/corporate/billing.html", "templates/zerver/hello.html", "templates/corporate/upgrade.html"]), 'good_lines': ["{{ render_bundle('landing-page') }}"], diff --git a/zerver/lib/storage.py b/zerver/lib/storage.py index 41f5a023b0..5a61ec17fe 100644 --- a/zerver/lib/storage.py +++ b/zerver/lib/storage.py @@ -1,4 +1,5 @@ -# Useful reading is https://zulip.readthedocs.io/en/latest/subsystems/front-end-build-process.html +# Useful reading is: +# https://zulip.readthedocs.io/en/latest/subsystems/html-css.html#front-end-build-process import os from typing import Optional