Usage: pip-accel [ARGUMENTS TO PIP]
pip-accel program is a wrapper for
pip, the Python package manager.
It accelerates the usage of
pip to initialize
Python virtual environments
given one or more
files. It does so by combining the following two approaches:
Source distribution downloads are cached and used to generate a
local index of source distribution archives.
If all your dependencies are pinned to absolute versions whose
source distribution downloads were previously cached,
won’t need a network connection at all! This is one of the reasons
pip can be so slow: given absolute pinned dependencies
available in the download cache it will still scan
PyPi and distribution websites.
are used to speed up the process of installing dependencies with
binary components (like
LXML). Instead of recompiling
these dependencies again for every virtual environment we compile
them once and cache the result as a binary
pip-accel command supports all subcommands and options
pip, however it is of course only useful for the
pip install subcommand. So for example:
pip-accel install -r requirements.txt
If you pass a `-v` option then
pip-accel will both
use verbose output. To enable verbose output from
you can set the environment variable
Based on the user running
pip-accel the following file locations
are used by default:
Root user All other users Purpose
~/.pip/download-cache Assumed to be pip’s download cache
~/.pip-accel Used to store the source/binary indexes
These defaults can be overridden by defining the environment
To give you an idea of how effective
pip-accel is, below are the
results of a test to build a virtual environment for one of the
internal codebases of Paylogic. This
code base requires more than 40 dependencies including several
packages that need compilation with SWIG and a C compiler:
Program Description Duration Percentage
pip Default configuration 444 seconds 100% (baseline) pip With download cache (first run) 416 seconds 94% pip With download cache (second run) 318 seconds 72% pip-accel First run 397 seconds 89% pip-accel Second run 30 seconds 7%
pip-accel works is not very intuitive but it is very
effective. Below is an overview of the control flow. Once you take
a look at the code you’ll notice that the steps below are all
embedded in a loop that retries several times. This is mostly
because of step 2 (downloading the source distributions).
pip install --no-index --no-install -r requirements.txtto unpack source distributions available in the local source index. This is the first step because
requirements.txtfiles as input but it will manually install dependencies from cached binary distributions (without using
If the command succeeds it means all dependencies are already available as downloaded source distributions. We’ll parse the verbose pip output of step 1 to find the direct and transitive dependencies (names and versions) defined in
requirements.txtand use them as input for step 3. Go to step 3.
If the command fails it probably means not all dependencies are available as local source distributions yet so we should download them. Go to step 2.
pip install --no-install -r requirements.txtto download missing source distributions to the download cache:
If the command fails it means that pip encountered errors while scanning PyPi, scanning a distribution website, downloading a source distribution or unpacking a source distribution. Usually these kinds of errors are intermittent so retrying a few times is worth a shot. Go to step 2.
If the command succeeds it means all dependencies are now available as local source distributions; we don’t need the network anymore! Go to step 1.
python setup.py bdist_dumb --format=gztar for each
dependency that doesn’t have a cached binary distribution yet
(taking version numbers into account). Go to step 4.
Install all dependencies from binary distributions based on the
list of direct and transitive dependencies obtained in step 1. We
have to do these installations manually because
pip support binary
If you have questions, bug reports, suggestions, etc. please create
an issue on the
GitHub project page. The
latest version of
pip-accel will always be available on GitHub.
This software is licensed under the
MIT license just like
pip (on which
© 2013 Peter Odding and Paylogic International.
Last updated Sun May 19 19:07:46 UTC 2013.