Lock the recursive requirements in common.txt via pip-compile.

The commit is composed of: (1) distill out top-level dependencies in
common.txt, (2) add -e flag to the vcs-based packages because
pip-compile can't do without, (3) pip-compile/generate the locked files
then remove the -e flags from the lockfile, (4) pin pathlib2 to dev.txt
because it turns out it is a direct requirement of
documentation_crawler, (5) document the structure and add an automation
script (6) remove cryptography==1.9 from requirements/scrapy.txt since
cryptography is automatically added from pyopenssl (7) add sed command
to remove future/futures from the generated lock file in python3 (this
should have been automatically handled by pip-compile, so pending for
the feature from pip-compile)

Tweaked by tabbott to update PROVISION_VERSION and add a missing
`first` dependency.
This commit is contained in:
rht
2017-05-12 07:30:38 +07:00
committed by Tim Abbott
parent 61e24cfbea
commit 8aaf34d06d
13 changed files with 284 additions and 33 deletions

32
requirements/README.md Normal file
View File

@@ -0,0 +1,32 @@
The dependency graph of the requirements is as follows:
```
py3_dev py3_prod py2_prod
+ + + + + + +
| | +->py3_common<-+ v v +>py2_common+----+
| | | + prod | + |
| | | v | v |
| | |py3_socialauth |py2_socialauth|
| v | | |
| >dev +--->common<------------+ |
| | + + |
| | v +-->emailmirror |
| |docs,moto,py3k,twisted |
| +-----------------------------------+py2_dev<+
v
mypy
```
Of the files, only py2_dev, py2_prod, py3_dev, py3_prod, and mypy have been
used in the install scripts directly. The rest are implicit dependencies.
py2_common and py3_common are locked.
Steps to update a lock file, e.g. to update ipython from 5.3.0 to 6.0.0 in
common.txt and propagate it py2_common_lock.txt and py3_common_lock:
0. Replace `ipython==5.4.1` with `ipython==6.0.0` in common.txt
1. Run './tools/update-locked-requirements'
The reason the steps to remove the `-e` is necessary is because `pip-compile`
doesn't support installing from vcs without `-e` yet.
You may track the ongoing progress here https://github.com/jazzband/pip-tools/issues/355.