Developer Guide¶
Do not hang around in Pythran code base without your developer guide! It is the compass that will guide you in the code jungle!
Disclaimer¶
This document is a never ending work-in-progress draft. Please contribute!
Additional Packages¶
Additional packages may be needed to use pythran to its fullest: colored logs, tests on several cores, doc generation
$> sudo apt-get install python-setuptools python-pytest-xdist python-docutils
$> sudo pip install colorlog flake8
Configuration¶
Pythran can be configured with a rc file. An example is found in pythran/pythran.cfg. Look at it! To customize it:
$> cp pythran/pythran.cfg ~/.pythranrc
In particular, you may want to add -g -O0
to the cxxflags
.
NT2 is required to compile the C++ code generated by Pythran. It is automatically installed along Pythran. If you don’t have NT2 installed in a system directory and you try to run Pythran directly from the source directory, the NT2 includes won’t be found.
A trick to ease the development and to enable running directly from the source
is to install Pythran once and to add
-I<install_prefix>/lib/<python-version>/site-packages/pythran
to the
cppflags
in order to point to the NT2 headers.
Coding Style¶
All Python code must be conform to the PEP 8, and the flake8
command must not
yield any message when run on our database. Additionally, avoid backslashes,
and try to make your code as concise as possible.
$> flake8 pythran/.py pythran//.py –exclude=”pythran/tests/test.py,__init__.py”
C++ code use spaces (no tabs) and a tab width of 4.
File Hierarchy¶
Listing the top level directory yields the following entries:
- setup.py
The files that describes what gets installed, that holds
PyPI
entries and such.- doc/
If you’re reading this document, you know what it’s all about!
MANUAL
is the user documentation andDEVGUIDE
is the developer documentation.- LICENSE
Boring but important stuff.
- MANIFEST.in
Describe additional stuff to package there.
- README
Quick introduction and description of _pythran_. The
README.rst
file is just a symbolic link that pleasesgithub
andPyPI
.- pythran/
The source of all things.
- pythran/tests/
The source of all issues.
- pythran/pythonic/
Where C++ back-end lies.
- scripts/
Where Python scripts calling the
pythran
module lies.
Validation¶
pythran
uses the unittest
module to manage test cases. The whole
validation suite is run through the command:
$> python setup.py test
It is tested using the pytest <http://pytest.org/latest/> package:
$> apt-get install python-pytest
To run it faster we use the pytest extension xdist from debian package python-pytest-xdist , the test suite will run using all available cores. Otherwise it might run very slowly, something like four hours on a decent laptop :’(.
Note that it is still possible to use the pytest
module directly, for
instance to pass a subset of the test suite:
$> PYTHONPATH=. py.test -n 8 pythran/tests/test_list.py
runs all the tests found in pythran/tests/test_list.py
.
Only compiler tests can be check using test filtering:
$> PYTHONPATH=. py.test -n 8 pythran/tests -m "not module"
There are two kinds of tests in pythran
:
unit tests that test a specific feature of the implementation. Such tests are listed as method of a class deriving from
test_env.TestEnv
and must call therun_test(function_to_translate, *effective_parameters, **name_to_signature)
method 1. It translatesfunction_to_translate
into a native function using the type annotations given in thename_to_signature
dictionary, runs both the python and the native version witheffective_parameters
as arguments and asserts the results are the same.- 1
See examples in
pythran/tests/test_base.py
for more details.
test cases that are just plain python modules to be converted in native module by
pythran
. It is used to test complex situations, codes or benchmarks found on the web etc. They are just translated, not run. These test cases lie inpythran/tests/cases/
and are listed inpythran/tests/test_cases.py
.
C++ runtime¶
The C++ code generated by pythran
relies on a specific back-end,
pythonic
. It is a set of headers that mimics Python’s intrinsics and
collections behavior in C++. It lies in pythran/pythonic/
. There is one
directory per module, e.g. pythran/pythonic/numpy
for the numpy
module,
and one file per function, e.g. pythran/pythonic/numpy/ones.hpp
for the
numpy.ones
function. Type definitions are stored in the seperate
pythran/pythonic/types
directory, one header per type. Each function header
must be #includ
-able independently, i.e. it itself includes all the type
and function definition it needs. This helps keeping compilation time low.
All Pythran functions and types live in the pythonic
namespace. Each extra
module defines a new namespace, like pythonic::math
or
pythonic::random
, and each type is defined in the pythonic::types
namespace. The DECLARE_FUNCTOR
and DEFINE_FUNCTOR
macros from
pythonic/utils/functor.hpp
is commonly used to convert functions into
functors and put them into the mandatory functor
namespace.
The pythonic runtime can be used without Python support, so it is important to
protect all Python-specific stuff inside ENABLE_PYTHON_MODULE
guard.
All methods are represented by functions in Pythran. The associated
pseudo-modules are prefixed and suffixed by a double underscore __
, as in
pythran/pythonic/__list__
.
Benchmarking and Testing¶
Stand-alone algorithms are put into pythran/tests/cases
. They must be valid
Pythran input (including spec annotations). To be taken into account by the
validation suite, they must be listed in pythran/tests/test_cases.py
. To be
taken into account by the benchmarking suite, they must have a line starting
with the #runas
directive. Check pythran/tests/matmul.py
for a complete
example.
To run the benchmark suite, one can rely on:
$> python setup.py bench --mode=<mode>
where <mode> is one among:
- python
Uses the interpreter used to run
setup.py
.- pythran
Uses the Pythran compiler.
- pythran+omp
Uses the Pythran compiler in OpenMP mode.
All measurements are made using the timeit
module. The number of iterations
is customizable through the --nb-iter
switch.
How to¶
- Add support for a new module
- Provide its C++ implementation in
pythran/pythonic++/<mymodule>
. pythran/pythonic++/math/*.hpp
andpythran/pythonic++/__list__/*.hpp
are good example to referer to.
- Provide its C++ implementation in
- Provide its description in
pythran/tables.py
. Each function, method or variable must be listed there with the appropriate description.
- Provide its description in
- Provide its test suite in
pythran/tests/
under the name test_my_module.py
. One test case per function, method or variable is great.
- Provide its test suite in
- Add a new analysis
Subclass one of
ModuleAnalysis
,FunctionAnalysis
orNodeAnalysis
.List analysis required by yours in the parent constructor, they will be built automatically and stored in the attribute with the corresponding uncameled name.
Write your analysis as a regular
ast.NodeVisitor
. The analysis result must be stored inself.result
.Use it either from another pass’s constructor, or through the
passmanager.gather
function.
- Push changes into the holy trunk
Use the
github
interface and the pull/push requests features- Make your dev available on the web and asks for a merge on the IRC
channel
#pythran