Installing Emacs on Ubuntu and Python in 2017

  • The aim is to use python3 and virtualenv for python development.
  • sudo apt install emacs25-lucid python-pip python3-dev
  • Install elpy
  • Now install Python packages required by elpy. The idea is to deliberately keep these “elpy requirements” separate from the packages required by each python project. This way, we can update the “elpy packages” across the entire system in one go; and we can list just the packages required by each project using pip freeze --local.
    • pip3 install --user jedi flake8 importmagic autopep8 virtualenvwrapper
    • pip3 install --user 'ipython<5' (ipython >= 5 does not work well on emacs. See here and here)

Add this to the end of ~/.profile:

# Python virtualenvwrapper
export PATH="$PATH:$HOME/.local/bin"
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/workingcopies
export VIRTUALENVWRAPPER_VIRTUALENV_ARGS='--system-site-packages'

Add this to the end of ~/.bashrc:

# Python virtualenvwrapper
source $HOME/.local/bin/virtualenvwrapper.sh

Logout. Log back in.

~/.emacs should contain this:

(require 'package)
(add-to-list 'package-archives
             '("elpy" . "http://jorgenschaefer.github.io/packages/"))

(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(package-selected-packages (quote (elpy)))
 )

;; eply
(package-initialize)
(elpy-enable)
(elpy-use-ipython)

Documenting code

Python libraries

  • Monary - need to install these Ubuntu packages, then install mongo-c-driver, then create /etc/ld.so.conf.d/libmongoc.conf and write /usr/local/lib in that file. Then run sudo ldconfig. Then run make test. Then do pip install pkgconfig monary.
  • Theano - “Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently.”
  • Pandas - “an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language… Time series-functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging. Even create domain-specific time offsets and join time series without losing data;”
  • GPUStats - “gpustats is a PyCUDA-based library implementing functionality similar to that present in scipy.stats. It implements a simple framework for specifying new CUDA kernels and extending existing ones. Here is a (partial) list of target functionality: Probability density functions (pdfs). These are intended to speed up likelihood calculations in particular in Bayesian inference applications, such as in PyMC, Random variable generation using CURAND”
  • PyOpenCL
  • PyMC - “PyMC is a python module that implements Bayesian statistical models and fitting algorithms, including Markov chain Monte Carlo. Its flexibility and extensibility make it applicable to a large suite of problems. Along with core sampling functionality, PyMC includes methods for summarizing output, plotting, goodness-of-fit and convergence diagnostics.”

  • SciKits - “Welcome to SciKits! Here you’ll find a searchable index of add-on toolkits that complement SciPy, a library of scientific computing routines. The SciKits cover a broad spectrum of application domains, including financial computation, audio processing, geosciences, computer vision, engineering, machine learning, medical computing and bioinformatics.”

  • NetworkX - “NetworkX is a Python language software package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.”

Python 2 versus 3

GUIs

  • From some very superficial searching, it looks like wxPython is the prefered Python GUI for use with matplotlib (I could be wrong though). One big disadvantage is that wxPython isn’t packaged as standard with Python, whilst tkInter is.
  • wxPython screenshots
  • list of Python GUI toolkits

Plotting

Tutorials & videos

Optimised compilers

Development tools

Statistics and graphical models

  • pebl - Python Environment for Bayesian Learning

Installing an up-to-date scientific Python stack on Ubuntu when you do have root permissions

sudo apt-get install python-dev python-pip python-sphinx libzmq-dev python-matplotlib python-scipy
sudo pip install cython pandas pyzmq jinja2  ipython sphinx

libzmq-dev, pyzmq and jinja2 are all required for iPython notebook.

For scipy:

sudo apt-get install libatlas-base-dev gfortran python-pip
sudo pip install scipy

If you get the following error when trying to import scipy libatlas.so.3: cannot open shared object file: No such file or directory then run

sudo update-alternatives --config libblas.so.3

and select /usr/lib/atlas-base/atlas/libblas.so.3 (this tip taken from Daniel Nouri’s blog post on libblas)

Installing Python when you don’t have root permissions

  • ./configure --prefix=/data/usr
  • Then edit Makefile and add -fPIC to the end of the line that starts CC= (as per this SO answer) (-fPIC is required so xmllib2 compiles correctly)
  • make -j8
  • make install

Install stuff for GTK+ development (when you don’t have root permissions)

  • Compile Python as above
  • Install libxml2:
    • download libxml2-2.X.Y.tar.gz from xmlsoft.org/libxml2
    • ./configure --prefix=/data/usr
    • make -j8
    • make install
    • cd python
    • setenv LD_LIBRARY_PATH "/data/usr/lib"
    • python setup.py build
    • python setup.py install
  • ​Make sure LD_LIBRARY_PATH is set as above
  • Follow “Installing from Source” instructions from here (install jhbuild, then install pygobject using jhbuild). Some notes on that process:
    • I added the following two lines to ~/.config/jhbuildrc:
      • prefix = "/homes/dk3810/.local/opt"
      • modulesets_dir = "/homes/dk3810/.local/modulesets"
    • I copied the *.modules files from releng to /homes/dk3810/.local/modulesets

Profiling

  • add the following to ~/.bash_aliases: alias profile='python -m cProfile -s time' (from SO)

Packaging

Integrating git workflow with the Python package publishing process

Notes for creating a package

Aims & Overview:

  • Upload just description of project to pypi using python setup.py register.
  • Don’t upload code to pypi. Instead use download_url in setup.py to point to github. e.g.: download_url = "https://github.com/JackKelly/rfm_ecomanager_logger/tarball/master#egg=rfm_ecomanager_logger-dev"
  • Use git tags to track version numbers.
  • Automatically suck these version numbers into Python’s packaging system and also into the project’s __version__ attribute.

Details: