Skip to content

Introduction to Package Managers and Environments

While many programming languages exist, Python has risen to the top amongst the scientific and data science community. Python is simple to learn and use, is platform independent, and most importantly has access to a wide range of open source packages and libraries that aid in data analysis. While many of these packages are open source and readily available, in order to leverage these libraries in our code, we must use package managers to install and manage the packages. There are two common package management systems used - pip and conda.


While it is easiest to work with pip and conda in the command line/terminal, for ease of demonstration, we will use ipython notebooks (in Co-calc) for the workshop.

Most of the commands (especially those starting with pip,conda, or mamba) used in this section are meant to be directly executed in a command-line. Within an ipython notebook, we use the ! to run a single line of code in the command-line. Additionally, if we wish to execute the entirety of a cell as command-line, we can use the %%bash magic command. We will see examples of these later in the notebook.

Introduction to pip

pip is a package management system for python that is used to install and manage additional libraries and dependencies. pip typically comes pre-installed by alongside python.

Let's verify that pip is installed:

!pip --version
pip 21.2.3 from /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages/pip (python 3.9)

*note that we are using the ! to execute a command on the system terminal/command-line

Let's see what we can do with pip using the help command

!pip --help

Usage:   
  pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible dependencies.
  config                      Manage local and global configuration.
  search                      Search PyPI for packages.
  cache                       Inspect and manage pip's wheel cache.
  index                       Inspect information available from package indexes.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  debug                       Show information useful for debugging.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring
                              environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be
                              used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be
                              used up to 3 times (corresponding to WARNING,
                              ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --no-input                  Disable prompting for input.
  --proxy <proxy>             Specify a proxy in the form
                              [user:[email protected]]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should
                              attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists:
                              (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host or host:port pair as trusted,
                              even though it does not have valid or any HTTPS.
  --cert <path>               Path to PEM-encoded CA certificate bundle. If
                              provided, overrides the default. See 'SSL
                              Certificate Verification' in pip documentation
                              for more information.
  --client-cert <path>        Path to SSL client certificate, a single file
                              containing the private key and the certificate
                              in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine
                              whether a new version of pip is available for
                              download. Implied with --no-index.
  --no-color                  Suppress colored output.
  --no-python-version-warning
                              Silence deprecation warnings for upcoming
                              unsupported Pythons.
  --use-feature <feature>     Enable new functionality, that may be backward
                              incompatible.
  --use-deprecated <feature>  Enable deprecated functionality, that will be
                              removed in the future.

pip list will show all installed packages, while pip freeze will show all user installed packages

!pip list
Package                    Version             Location
-------------------------- ------------------- -------------------------------------------------
aioitertools               0.8.0
anyio                      3.3.0
argon2-cffi                20.1.0
async-generator            1.10
atomate                    0.9.9
attrs                      21.2.0
Automat                    20.2.0
Babel                      2.9.1
backcall                   0.2.0
bcrypt                     3.2.0
bleach                     4.0.0
boltons                    21.0.0
bravado                    11.0.3
bravado-core               5.17.0
brotlipy                   0.7.0
certifi                    2021.5.30
cffi                       1.14.6
chardet                    4.0.0
charset-normalizer         2.0.0
click                      8.0.1
colorama                   0.4.4
conda                      4.10.3
conda-package-handling     1.7.3
constantly                 15.1.0
crochet                    2.0.0
cryptography               3.4.7
crystal-toolkit            2021.4.29
crystaltoolkit-extension   0.3.0
custodian                  2021.2.8
cycler                     0.10.0
debugpy                    1.4.1
decorator                  5.0.9
defusedxml                 0.7.1
dnspython                  2.1.0
email-validator            1.1.3
emmet-core                 0.9.3
entrypoints                0.3
fastapi                    0.68.0
fido                       4.2.2
filetype                   1.0.7
FireWorks                  1.9.7
Flask                      2.0.1
flask-paginate             0.8.1
flatten-dict               0.4.2
future                     0.18.2
ghp-import                 2.0.1
gitdb                      4.0.7
GitPython                  3.1.18
graphviz                   0.17
gunicorn                   20.1.0
h5py                       3.3.0
htmlmin                    0.1.12
hyperlink                  21.0.0
idna                       3.1
imageio                    2.9.0
importlib-metadata         4.6.3
incremental                21.3.0
inflect                    5.3.0
iniconfig                  1.1.1
ipykernel                  6.0.3
ipython                    7.26.0
ipython-genutils           0.2.0
ipywidgets                 7.6.3
itsdangerous               2.0.1
jedi                       0.18.0
Jinja2                     3.0.1
joblib                     1.0.1
jsmin                      2.2.2
json2html                  1.3.0
json5                      0.9.6
jsonpointer                2.1
jsonref                    0.2
jsonschema                 3.2.0
jupyter-client             6.1.12
jupyter-core               4.7.1
jupyter-server             1.10.2
jupyterlab                 3.1.4
jupyterlab-pygments        0.1.2
jupyterlab-server          2.6.2
jupyterlab-widgets         1.0.0
kiwisolver                 1.3.1
latexcodec                 2.0.1
llvmlite                   0.36.0
maggma                     0.30.4
mamba                      0.15.2
Markdown                   3.3.4
MarkupSafe                 2.0.1
matminer                   0.7.4
matplotlib                 3.4.2
matplotlib-inline          0.1.2
mergedeep                  1.3.4
mistune                    0.8.4
mkdocs                     1.2.2
mkdocs-material            7.2.2
mkdocs-material-extensions 1.0.1
mkdocs-minify-plugin       0.4.0
mknotebooks                0.7.0.post0.dev2
mongogrant                 0.3.3
mongomock                  3.23.0
monotonic                  1.6
monty                      2021.7.8
mp-api                     0.7.1
mp-pyrho                   0.0.14
mp-workshop                0.1.dev1+g8a2d3aa   /home/runner/work/workshop/workshop/python_module
mpcontribs-client          3.13.0
mpmath                     1.2.1
msgpack                    1.0.2
nbclassic                  0.3.1
nbclient                   0.5.3
nbconvert                  6.1.0
nbformat                   5.1.3
nbgitpuller                0.10.1
nbzip                      0.1.0
nest-asyncio               1.5.1
networkx                   2.6.2
notebook                   6.4.2
numba                      0.53.1
numpy                      1.21.1
olefile                    0.46
orjson                     3.6.1
packaging                  21.0
palettable                 3.3.0
pandas                     1.3.1
pandocfilters              1.4.3
paramiko                   2.7.2
parso                      0.8.2
pexpect                    4.8.0
pickleshare                0.7.5
Pillow                     8.3.1
Pint                       0.17
pip                        21.2.3
plotly                     4.14.3
pluggy                     0.13.1
prometheus-client          0.11.0
prompt-toolkit             3.0.19
ptyprocess                 0.7.0
PubChemPy                  1.0.4
py                         1.10.0
pyasn1                     0.4.8
pyasn1-modules             0.2.8
pybtex                     0.24.0
pycairo                    1.20.1
pycosat                    0.6.3
pycparser                  2.20
pydantic                   1.8.2
pydash                     5.0.2
Pygments                   2.9.0
PyGObject                  3.40.1
pyIsEmail                  1.3.2
pymatgen                   2022.0.11
pymatgen-diffusion         2020.10.8
pymdown-extensions         8.2
pymongo                    3.12.0
PyNaCl                     1.4.0
pynng                      0.7.1
pyOpenSSL                  20.0.1
pyparsing                  2.4.7
pyrsistent                 0.18.0
PySocks                    1.7.1
pytest                     6.2.4
python-dateutil            2.8.2
pytz                       2021.1
PyWavelets                 1.1.1
PyYAML                     5.4.1
pyyaml_env_tag             0.1
pyzmq                      22.2.1
ratelimit                  2.2.1
reaction-network           2.0.3
requests                   2.26.0
requests-futures           1.0.0
requests-unixsocket        0.2.0
retrying                   1.3.3
rfc3987                    1.3.8
robocrys                   0.2.7
ruamel.yaml                0.17.10
ruamel.yaml.clib           0.2.6
ruamel-yaml-conda          0.15.80
scikit-image               0.18.2
scikit-learn               0.24.2
scipy                      1.7.1
Send2Trash                 1.8.0
sentinels                  1.0.0
sentry-sdk                 1.3.1
service-identity           21.1.0
setuptools                 49.6.0.post20210108
simplejson                 3.17.3
six                        1.16.0
smmap                      4.0.0
sniffio                    1.2.0
spglib                     1.16.1
sshtunnel                  0.4.0
starlette                  0.14.2
strict-rfc3339             0.7
swagger-spec-validator     2.7.3
sympy                      1.8
tabulate                   0.8.9
terminado                  0.10.1
testpath                   0.5.0
threadpoolctl              2.2.0
tifffile                   2021.8.8
toml                       0.10.2
tornado                    6.1
tqdm                       4.62.0
traitlets                  5.0.5
Twisted                    21.7.0
typing-extensions          3.10.0.0
uncertainties              3.1.6
urllib3                    1.26.6
watchdog                   2.1.3
wcwidth                    0.2.5
webcolors                  1.11.1
webencodings               0.5.1
websocket-client           1.1.1
Werkzeug                   2.0.1
wheel                      0.36.2
widgetsnbextension         3.5.1
wrapt                      1.12.1
yelp-bytes                 0.3.0
yelp-encodings             1.0.0
zipp                       3.5.0
zope.interface             5.4.0
zstandard                  0.15.2

!pip freeze
aioitertools==0.8.0
anyio==3.3.0
argon2-cffi==20.1.0
async-generator==1.10
atomate==0.9.9
attrs==21.2.0
Automat==20.2.0
Babel==2.9.1
backcall==0.2.0
bcrypt==3.2.0
bleach==4.0.0
boltons==21.0.0
bravado==11.0.3
bravado-core==5.17.0
brotlipy==0.7.0
certifi==2021.5.30
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1625835307225/work
chardet @ file:///home/conda/feedstock_root/build_artifacts/chardet_1610093492116/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1626371162869/work
click==8.0.1
colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1602866480661/work
conda==4.10.3
conda-package-handling @ file:///home/conda/feedstock_root/build_artifacts/conda-package-handling_1618231390031/work
constantly==15.1.0
crochet==2.0.0
cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography_1616851485986/work
crystal-toolkit==2021.4.29
crystaltoolkit-extension==0.3.0
custodian==2021.2.8
cycler==0.10.0
debugpy==1.4.1
decorator==5.0.9
defusedxml==0.7.1
dnspython==2.1.0
email-validator==1.1.3
emmet-core==0.9.3
entrypoints==0.3
fastapi==0.68.0
fido==4.2.2
filetype==1.0.7
FireWorks==1.9.7
Flask==2.0.1
flask-paginate==0.8.1
flatten-dict==0.4.2
future==0.18.2
ghp-import==2.0.1
gitdb==4.0.7
GitPython==3.1.18
graphviz==0.17
gunicorn==20.1.0
h5py==3.3.0
htmlmin==0.1.12
hyperlink==21.0.0
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1609836280497/work
imageio==2.9.0
importlib-metadata==4.6.3
incremental==21.3.0
inflect==5.3.0
iniconfig==1.1.1
ipykernel==6.0.3
ipython==7.26.0
ipython-genutils==0.2.0
ipywidgets==7.6.3
itsdangerous==2.0.1
jedi==0.18.0
Jinja2==3.0.1
joblib==1.0.1
jsmin==2.2.2
json2html==1.3.0
json5==0.9.6
jsonpointer==2.1
jsonref==0.2
jsonschema==3.2.0
jupyter-client==6.1.12
jupyter-core==4.7.1
jupyter-server==1.10.2
jupyterlab==3.1.4
jupyterlab-pygments==0.1.2
jupyterlab-server==2.6.2
jupyterlab-widgets==1.0.0
kiwisolver @ file:///home/conda/feedstock_root/build_artifacts/kiwisolver_1610099772044/work
latexcodec==2.0.1
llvmlite==0.36.0
maggma==0.30.4
mamba @ file:///home/conda/feedstock_root/build_artifacts/mamba_1626467930043/work
Markdown==3.3.4
MarkupSafe==2.0.1
matminer==0.7.4
matplotlib @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-suite_1620506038354/work
matplotlib-inline==0.1.2
mergedeep==1.3.4
mistune==0.8.4
mkdocs==1.2.2
mkdocs-material==7.2.2
mkdocs-material-extensions==1.0.1
mkdocs-minify-plugin==0.4.0
mknotebooks==0.7.0.post0.dev2
mongogrant==0.3.3
mongomock==3.23.0
monotonic==1.6
monty==2021.7.8
mp-api==0.7.1
mp-pyrho==0.0.14
-e git+https://github.com/materialsproject/[email protected]#egg=mp_workshop
mpcontribs-client==3.13.0
mpmath==1.2.1
msgpack==1.0.2
nbclassic==0.3.1
nbclient==0.5.3
nbconvert==6.1.0
nbformat==5.1.3
nbgitpuller==0.10.1
nbzip==0.1.0
nest-asyncio==1.5.1
networkx==2.6.2
notebook==6.4.2
numba==0.53.1
numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1626648378336/work
olefile @ file:///home/conda/feedstock_root/build_artifacts/olefile_1602866521163/work
orjson==3.6.1
packaging==21.0
palettable==3.3.0
pandas==1.3.1
pandocfilters==1.4.3
paramiko==2.7.2
parso==0.8.2
pexpect==4.8.0
pickleshare==0.7.5
Pillow @ file:///home/conda/feedstock_root/build_artifacts/pillow_1625677815963/work
Pint==0.17
plotly==4.14.3
pluggy==0.13.1
prometheus-client==0.11.0
prompt-toolkit==3.0.19
ptyprocess==0.7.0
PubChemPy==1.0.4
py==1.10.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybtex==0.24.0
pycairo==1.20.1
pycosat @ file:///home/conda/feedstock_root/build_artifacts/pycosat_1610094799048/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1593275161868/work
pydantic==1.8.2
pydash==5.0.2
Pygments==2.9.0
PyGObject==3.40.1
pyIsEmail==1.3.2
pymatgen==2022.0.11
pymatgen-diffusion==2020.10.8
pymdown-extensions==8.2
pymongo==3.12.0
PyNaCl==1.4.0
pynng==0.7.1
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1608055815057/work
pyparsing==2.4.7
pyrsistent==0.18.0
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1610291451001/work
pytest==6.2.4
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
pytz==2021.1
PyWavelets==1.1.1
PyYAML==5.4.1
pyyaml_env_tag==0.1
pyzmq==22.2.1
ratelimit==2.2.1
reaction-network==2.0.3
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1626393743643/work
requests-futures==1.0.0
requests-unixsocket==0.2.0
retrying==1.3.3
rfc3987==1.3.8
robocrys==0.2.7
ruamel-yaml-conda @ file:///home/conda/feedstock_root/build_artifacts/ruamel_yaml_1611943432947/work
ruamel.yaml==0.17.10
ruamel.yaml.clib==0.2.6
scikit-image==0.18.2
scikit-learn==0.24.2
scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy_1628206395139/work
Send2Trash==1.8.0
sentinels==1.0.0
sentry-sdk==1.3.1
service-identity==21.1.0
simplejson==3.17.3
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
smmap==4.0.0
sniffio==1.2.0
spglib==1.16.1
sshtunnel==0.4.0
starlette==0.14.2
strict-rfc3339==0.7
swagger-spec-validator==2.7.3
sympy==1.8
tabulate==0.8.9
terminado==0.10.1
testpath==0.5.0
threadpoolctl==2.2.0
tifffile==2021.8.8
toml==0.10.2
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1610094708661/work
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1627689047698/work
traitlets==5.0.5
Twisted==21.7.0
typing-extensions==3.10.0.0
uncertainties==3.1.6
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1624634538755/work
watchdog==2.1.3
wcwidth==0.2.5
webcolors==1.11.1
webencodings==0.5.1
websocket-client==1.1.1
Werkzeug==2.0.1
widgetsnbextension==3.5.1
wrapt==1.12.1
yelp-bytes==0.3.0
yelp-encodings==1.0.0
zipp==3.5.0
zope.interface==5.4.0
zstandard @ file:///home/conda/feedstock_root/build_artifacts/zstandard_1614954803539/work

Searching for python packages

Python packages for (nearly) any task can be found on the Python Package Index (PyPI). PyPI is an online repository of 3rd party packages. Additionally, most python packages can be found on github, gitlab or other code repositories. Google search is also very handy when searching for packages.

  • For those familiar with pip, pip search has recently been deprecated. See this link and this github issue for more information.

For demonstration purposes, we will examine the PyPI entry for MDAnalysis. The landing page for MDAnalysis provides us with details on the package in addition to installation instructions, examples, and release history.

Installing python packages

To install the latest version of a package, use pip install

!pip show MDAnalysis
WARNING: Package(s) not found: MDAnalysis

!pip install MDAnalysis
Collecting MDAnalysis
  Downloading MDAnalysis-1.0.0.tar.gz (19.6 MB)
     |████████████████████████████████| 19.6 MB 16.8 MB/s 
Requirement already satisfied: numpy>=1.13.3 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis) (1.21.1)
Collecting biopython>=1.71
  Downloading biopython-1.79-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (2.3 MB)
     |████████████████████████████████| 2.3 MB 79.5 MB/s 
Requirement already satisfied: networkx>=1.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis) (2.6.2)
Collecting GridDataFormats>=0.4.0
  Downloading GridDataFormats-0.5.0-py2.py3-none-any.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 112.1 MB/s 
Requirement already satisfied: six>=1.4.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis) (1.16.0)
Collecting mmtf-python>=1.0.0
  Downloading mmtf_python-1.1.2-py2.py3-none-any.whl (21 kB)
Requirement already satisfied: joblib>=0.12 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis) (1.0.1)
Requirement already satisfied: scipy>=1.0.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis) (1.7.1)
Requirement already satisfied: matplotlib>=1.5.1 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis) (3.4.2)
Collecting mock
  Downloading mock-4.0.3-py3-none-any.whl (28 kB)
Requirement already satisfied: tqdm>=4.43.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis) (4.62.0)
Collecting gsd>=1.4.0
  Downloading gsd-2.4.2-cp39-cp39-manylinux2010_x86_64.whl (354 kB)
     |████████████████████████████████| 354 kB 78.3 MB/s 
Requirement already satisfied: python-dateutil>=2.7 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from matplotlib>=1.5.1->MDAnalysis) (2.8.2)
Requirement already satisfied: pyparsing>=2.2.1 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from matplotlib>=1.5.1->MDAnalysis) (2.4.7)
Requirement already satisfied: cycler>=0.10 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from matplotlib>=1.5.1->MDAnalysis) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from matplotlib>=1.5.1->MDAnalysis) (1.3.1)
Requirement already satisfied: pillow>=6.2.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from matplotlib>=1.5.1->MDAnalysis) (8.3.1)
Requirement already satisfied: msgpack>=0.5.6 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from mmtf-python>=1.0.0->MDAnalysis) (1.0.2)
Building wheels for collected packages: MDAnalysis
  Building wheel for MDAnalysis (setup.py) ... - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done
  Created wheel for MDAnalysis: filename=MDAnalysis-1.0.0-cp39-cp39-linux_x86_64.whl size=2122190 sha256=332acdb1b0611aba1a75f990e4af6707461f2e88a345bde92ae160a4f74c51b6
  Stored in directory: /home/runner/.cache/pip/wheels/50/9f/9e/803a0fafc253694711c79fb2940e717d32cf5114719a444cc5
Successfully built MDAnalysis
Installing collected packages: mock, mmtf-python, gsd, GridDataFormats, biopython, MDAnalysis
Successfully installed GridDataFormats-0.5.0 MDAnalysis-1.0.0 biopython-1.79 gsd-2.4.2 mmtf-python-1.1.2 mock-4.0.3

Now, we confirm the package is installed using the pip show command again.

!pip show MDAnalysis
Name: MDAnalysis
Version: 1.0.0
Summary: An object-oriented toolkit to analyze molecular dynamics trajectories generated by CHARMM, Gromacs, NAMD, LAMMPS, or Amber.
Home-page: https://www.mdanalysis.org
Author: Naveen Michaud-Agrawal
Author-email: [email protected]
License: GPL 2
Location: /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages
Requires: mmtf-python, GridDataFormats, biopython, scipy, joblib, gsd, networkx, numpy, matplotlib, six, tqdm, mock
Required-by: 

After installing the package, we must reload the notebook kernel to load the newly installed package. To do this, click Kernel -> Restart Kernel

import MDAnalysis
MDAnalysis.__version__
'1.0.0'

Package versions can be found on the PyPI page for each package. We can specify the version we want installed using ==.

!pip install --upgrade MDAnalysis==2.0.0b0
Collecting MDAnalysis==2.0.0b0
  Downloading MDAnalysis-2.0.0b0.tar.gz (3.4 MB)
     |████████████████████████████████| 3.4 MB 7.1 MB/s 
Requirement already satisfied: numpy>=1.16.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis==2.0.0b0) (1.21.1)
Requirement already satisfied: biopython>=1.71 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis==2.0.0b0) (1.79)
Requirement already satisfied: networkx>=1.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis==2.0.0b0) (2.6.2)
Requirement already satisfied: GridDataFormats>=0.4.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis==2.0.0b0) (0.5.0)
Requirement already satisfied: mmtf-python>=1.0.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis==2.0.0b0) (1.1.2)
Requirement already satisfied: joblib>=0.12 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis==2.0.0b0) (1.0.1)
Requirement already satisfied: scipy>=1.0.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis==2.0.0b0) (1.7.1)
Requirement already satisfied: matplotlib>=1.5.1 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis==2.0.0b0) (3.4.2)
Requirement already satisfied: tqdm>=4.43.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis==2.0.0b0) (4.62.0)
Requirement already satisfied: threadpoolctl in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis==2.0.0b0) (2.2.0)
Requirement already satisfied: gsd>=1.4.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from MDAnalysis==2.0.0b0) (2.4.2)
Requirement already satisfied: six in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from GridDataFormats>=0.4.0->MDAnalysis==2.0.0b0) (1.16.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from matplotlib>=1.5.1->MDAnalysis==2.0.0b0) (1.3.1)
Requirement already satisfied: pyparsing>=2.2.1 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from matplotlib>=1.5.1->MDAnalysis==2.0.0b0) (2.4.7)
Requirement already satisfied: python-dateutil>=2.7 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from matplotlib>=1.5.1->MDAnalysis==2.0.0b0) (2.8.2)
Requirement already satisfied: cycler>=0.10 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from matplotlib>=1.5.1->MDAnalysis==2.0.0b0) (0.10.0)
Requirement already satisfied: pillow>=6.2.0 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from matplotlib>=1.5.1->MDAnalysis==2.0.0b0) (8.3.1)
Requirement already satisfied: msgpack>=0.5.6 in /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages (from mmtf-python>=1.0.0->MDAnalysis==2.0.0b0) (1.0.2)
Building wheels for collected packages: MDAnalysis
  Building wheel for MDAnalysis (setup.py) ... - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done
  Created wheel for MDAnalysis: filename=MDAnalysis-2.0.0b0-cp39-cp39-linux_x86_64.whl size=2128678 sha256=286c3f3999cc0048338213524891883889f117d6a17e633426590fc6b3ac3832
  Stored in directory: /home/runner/.cache/pip/wheels/5e/a7/28/1c392640be72c44d40adeefd5ea798b0e67d685eca30c253e9
Successfully built MDAnalysis
Installing collected packages: MDAnalysis
  Attempting uninstall: MDAnalysis
    Found existing installation: MDAnalysis 1.0.0
    Uninstalling MDAnalysis-1.0.0:
      Successfully uninstalled MDAnalysis-1.0.0
Successfully installed MDAnalysis-2.0.0b0

!pip show MDAnalysis
Name: MDAnalysis
Version: 2.0.0b0
Summary: An object-oriented toolkit to analyze molecular dynamics trajectories generated by CHARMM, Gromacs, NAMD, LAMMPS, or Amber.
Home-page: https://www.mdanalysis.org
Author: MDAnalysis Development Team
Author-email: [email protected]
License: GPL 2
Location: /usr/share/miniconda/envs/workshop/lib/python3.9/site-packages
Requires: joblib, tqdm, numpy, threadpoolctl, mmtf-python, gsd, networkx, matplotlib, scipy, biopython, GridDataFormats
Required-by: 

Again, we must restart our kernel to reset our ipython runtime environment.

import MDAnalysis
MDAnalysis.__version__
'1.0.0'

Installing from packages from Github

In some cases, especially for development or when using alpha or beta code, we must install the package from the source code. As an example, we will install mpmorph which is a package for running AIMD calculations in vasp using pymatgen, fireworks, and atomate.

!git clone https://github.com/materialsproject/mpmorph.git

!pip install mpmorph/

!pip show mpmorph

Removing Python Packages

Use pip uninstall to remove a python package. Use the -y flag to pre-confirm the uninstall

!pip uninstall -y MDAnalysis
Found existing installation: MDAnalysis 2.0.0b0
Uninstalling MDAnalysis-2.0.0b0:
  Successfully uninstalled MDAnalysis-2.0.0b0

!pip show MDAnalysis
WARNING: Package(s) not found: MDAnalysis

Now when we attempt to show the details, we see that the MDAnalysis package is uninstalled and cannot be found

!pip uninstall -y mpmorph
WARNING: Skipping mpmorph as it is not installed.

Introduction to Conda (and Mamba)

Much like pip, conda is a program writen in python for the purpose of managing packages. However, using conda, managed packages and software are not limited to python. Conda is a general-purpose package manager and enables installation of C libraries or R packages. Additionally, conda installs pre-built binaries, although it is possible to build packages from the source.

Conda also allows for the creation of virtual environments. Environments are very powerful and allow us to install a local collection of packages isolated from system packages or other environments. Unlike pip, conda can also install or manage different versions of python. Conda and pip are usually used in conjunction.

While conda is written in python, a mamba has recently been released and reimplements the functionality of conda using C++. Mamba offers parallel downloading of repository data and package files and a faster dependency solver. We will be using mamba in the workshop, however the two are interchangeable - that is, all mamba commands we use can be directly replaced with conda (i.e. mamba install -y pymatgen and conda install -y pymatgen both work) unless otherwise noted.

!mamba -h
usage: mamba [-h] [-V] command ...

conda is a tool for managing and deploying applications, environments and packages.

Options:

positional arguments:
  command
    clean        Remove unused packages and caches.
    compare      Compare packages between conda environments.
    config       Modify configuration values in .condarc. This is modeled
                 after the git config command. Writes to the user .condarc
                 file (/home/runner/.condarc) by default.
    create       Create a new conda environment from a list of specified
                 packages.
    help         Displays a list of available conda commands and their help
                 strings.
    info         Display information about current conda install.
    init         Initialize conda for shell interaction. [Experimental]
    install      Installs a list of packages into a specified conda
                 environment.
    list         List linked packages in a conda environment.
    package      Low-level conda package utility. (EXPERIMENTAL)
    remove       Remove a list of packages from a specified conda environment.
    uninstall    Alias for conda remove.
    run          Run an executable in a conda environment. [Experimental]
    search       Search for packages and display associated information. The
                 input is a MatchSpec, a query language for conda packages.
                 See examples below.
    update       Updates conda packages to the latest compatible version.
    upgrade      Alias for conda update.
    repoquery    Query repositories using mamba.

optional arguments:
  -h, --help     Show this help message and exit.
  -V, --version  Show the conda version number and exit.

conda commands available from other packages:
  env

We can get a list of the available conda environments using the conda env list command

!mamba env --help

!mamba env list

Creating Conda Environments

Create a new conda environment using the conda create command. We use the -y flag, since we cannot provide confirmation to the running code. We specify the environment to be created with a installation of python 3.9

!mamba create -y --name temp_env python=3.9

Our new environment shows up under our list of conda environments. However it is not the active environment (does not have a * next to it).

!mamba env list

Using and interacting with Conda environments

We must activate our environment to install python packages and run python code using the new environment. Since we are interacting with our environment from jupyter notebook, we must activate our environment within each cell before we use any bash commands.

To do this, use the %%bash command to create a multi-line bash cell, then activate the environment using source activate <env_name>

%%bash source activate temp_env

mamba env list

Now, using the conda env list command, we see our new environment is active, as denoted by the *

%%bash source activate temp_env

mamba list

Installing from conda-forge

When using conda to install or search for packages, the standard behaviour is to search the default repositories. These are packages provided by Anaconda, Inc. for free. To access third-party software, we can use the conda-forge channel. Conda-Forge is a community effort to provide conda packages for a wide range of software.

To install software from the conda-forge channel, simply add -c conda-forge to the conda install command. This will tell conda to include the packages in conda-forge when searching for packages and dependencies.

Many Materials Project codes (such as pymatgen, fireworks, custodian, atomate, and matminer) are available through the conda-forge channel.

* Note: repoquery is not available in conda. In conda, you must use conda search numpy and its equivalent command in mamba is mamba search numpy. We use the repoquery command, as it offers improved speed over the default search.

%%bash source activate temp_env

mamba repoquery search numpy

%%bash source activate temp_env

mamba install -y numpy -c conda-forge

We can add conda-forge to the list of channels (to avoid adding -c conda-forge to every installation line)

* mamba does not currently support changing configuration files. We use conda for this, although the config changes also apply to mamba

%%bash source activate temp_env

conda config --add channels conda-forge conda config --set channel_priority strict

Now we can install packages using less arguments

%%bash source activate temp_env

mamba install -y scipy

Creating a Custom Kernel for Jupyter/IPython

While our environment is ready for command-line use, there are additional steps to use it with jupyter.

  1. First, we will install "ipykernel"

%%bash source activate temp_env

mamba install -y ipykernel

  1. We will create a jupyter kernelspec for our environment

!jupyter kernelspec list

We will create a kernelspec (a pointer to our environment) using the python in our environment and add it to the list of kernels for our base environment.

!mamba env list
# conda environments:
#
                         /usr/share/miniconda
base                  *  /usr/share/miniconda/envs/workshop


!/home/user/.conda/envs/temp_env/bin/python -m ipykernel install --user --name 'temp_env_kernel'

!jupyter kernelspec list

import numpy, scipy

Remove/Delete Conda environment

When a conda environment is no longer useful or needs to be uninstalled, remove it with conda remove.

!mamba remove -y --name temp_env --all

!mamba env list

If we created a kernelspec, we'll want to remove it (not done by default)

!jupyter kernelspec remove -f temp_env_kernel