- sphinx-apidoc "is a tool for automatic generation of Sphinx sources that, using the autodoc extension, document a whole package in the style of other automatic API documentation tools."
- Math support in Sphinx
- Example documentation markup from An Example PyPi project
- Monary - need to install these Ubuntu packages, then install mongo-c-driver, then create
/usr/local/libin that file. Then run
sudo ldconfig. Then run
make test. Then do
pip install pkgconfig monary.
- Theano - “Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently.”
- Pandas - “an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language… Time series-functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging. Even create domain-specific time offsets and join time series without losing data;”
- GPUStats - "gpustats is a PyCUDA-based library implementing functionality similar to that present in scipy.stats. It implements a simple framework for specifying new CUDA kernels and extending existing ones. Here is a (partial) list of target functionality: Probability density functions (pdfs). These are intended to speed up likelihood calculations in particular in Bayesian inference applications, such as in PyMC, Random variable generation using CURAND"
- PyMC - “PyMC is a python module that implements Bayesian statistical models and fitting algorithms, including Markov chain Monte Carlo. Its flexibility and extensibility make it applicable to a large suite of problems. Along with core sampling functionality, PyMC includes methods for summarizing output, plotting, goodness-of-fit and convergence diagnostics.”
SciKits - “Welcome to SciKits! Here you’ll find a searchable index of add-on toolkits that complement SciPy, a library of scientific computing routines. The SciKits cover a broad spectrum of application domains, including financial computation, audio processing, geosciences, computer vision, engineering, machine learning, medical computing and bioinformatics.”
- sckikit-learn: machine learning in Python
NetworkX - “NetworkX is a Python language software package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.”
Python 2 versus 3
- From some very superficial searching, it looks like wxPython is the prefered Python GUI for use with matplotlib (I could be wrong though). One big disadvantage is that wxPython isn’t packaged as standard with Python, whilst tkInter is.
- wxPython screenshots
- list of Python GUI toolkits
- Nice examples by Eli Bendersky (including live data and interactivity) of using matplotlib with wxPython GUIs
- embedding matplotlib plots in GUI apps
Tutorials & videos
- Advanced Statistical Computing at Vanderbilt University's Department of Biostatistics by Chris Fonnesbeck (lead dev. PyMC)
- Python Scientific Lecture Notes
- goatbar’s iPython “research tools” videos
- Data analysis in Python with Pandas (3hr video)
- Ubuntu tutorial videos on using Python with GTK and GObject and Glade
- Numba: a NumPy-aware optimised compiler for Python (and here's a Numba vs Cython blog post by jakevdp)
- Using iPython from within PyDev (Eclipse)
Statistics and graphical models
- pebl - Python Environment for Bayesian Learning
Installing an up-to-date scientific Python stack on Ubuntu when you do have root permissions
sudo apt-get install python-dev python-pip python-sphinx libzmq-dev python-matplotlib python-scipy sudo pip install cython pandas pyzmq jinja2 ipython sphinx
libzmq-dev, pyzmq and jinja2 are all required for iPython notebook.
sudo apt-get install libatlas-base-dev gfortran python-pip sudo pip install scipy
If you get the following error when trying to import scipy
libatlas.so.3: cannot open shared object file: No such file or directory then run
sudo update-alternatives --config libblas.so.3
/usr/lib/atlas-base/atlas/libblas.so.3 (this tip taken from Daniel Nouri's blog post on libblas)
Installing Python when you don’t have root permissions
- Then edit
-fPICto the end of the line that starts
CC=(as per this SO answer) (
-fPICis required so
Install stuff for GTK+ development (when you don’t have root permissions)
- Compile Python as above
- Install libxml2:
setenv LD_LIBRARY_PATH "/data/usr/lib"
python setup.py build
python setup.py install
- Make sure
LD_LIBRARY_PATHis set as above
- Follow “Installing from Source” instructions from here (install
jhbuild, then install pygobject using jhbuild). Some notes on that
- I added the following two lines to
prefix = "/homes/dk3810/.local/opt"
modulesets_dir = "/homes/dk3810/.local/modulesets"
- I copied the
*.modulesfiles from releng to
- I added the following two lines to
- add the following to
alias profile='python -m cProfile -s time'(from SO)
distributeaims to supercede
distributeis compatible with Python 3,
setup.pyfiles are sufficiently simple to mean that I don't need to modify anything to allow users to use either
- Building and Distributing Packages with Distribute
- See here for details of where files are installed by
- Official Python documentation on modules and packages and directory layout
- pip documentation
- Non-recursive upgrades using pip
Integrating git workflow with the Python package publishing process
- SO: How to configure setup.py to have pip install from GitHub master?
- SO: Automatic version number both in setup.py (setuptools) AND source code?
- Blog post on cberner.com on Git revision numbers for setuptools packages using a simple bash script.
- Blog post on dcreager.net on Extracting setuptools version numbers from your git using a small Python script
- setuptools manual on using "tagging" (but this doesn't integrate directly with git)
Notes for creating a package
Aims & Overview:
- Upload just description of project to
python setup.py register.
- Don't upload code to
pypi. Instead use
setup.pyto point to github. e.g.:
download_url = "https://github.com/JackKelly/rfm_ecomanager_logger/tarball/master#egg=rfm_ecomanager_logger-dev"
- Use git tags to track version numbers.
- Automatically suck these version numbers into Python's packaging system and also into the project's
- Setup directory structure etc. as described in The Hitchhiker's Guide to Packaging and "Dive Into Python 3: Chapter 16, Packaging Python Libraries"
git -s tag VERSION.NUMBERfor version numbers and push these tags to github with
git push --tags(read tagging to learn how to use git tagging)
- Read blog post on dcreager.net on Extracting setuptools version numbers from your git using a small Python script
- Figure out how to use these version numbers for version as well as for setuptools. See SO: Automatic version number both in setup.py (setuptools) AND source code?
- Might need to use ConfigParse to parse setup.cfg (see this example) to extract the version number from
- Figure out how to point
download_urlto the correct tag
- It appears that two things are necessary to get upgrading to work correctly:
setup.py) needs to incremement and
download_urlneeds to point to a URL with
#egg=PROJECT-VERSIONappended to it (or upload all the files to
pypiinstead of downloading from
github, but that feels rather ugly to duplicate lots of files)