Distribution build pipeline

The Kolibri Package build pipeline looks like this:

                       Git release branch
                               |
                               |
                              / \
                             /   \
Python dist, online dependencies  \
   `python setup.py bdist_wheel`   \
               /                    \
              /                Python dist, bundled dependencies
       Upload to PyPi        `python setup.py bdist_wheel --static`
      Installable with                 \
    `pip install kolibri`               \
                                   Upload to PyPi
                                   Installable with
                             `pip install kolibri-static`
                               /            |          \
                              /             |           \
                        Windows          Android        Debian
                       installer           APK         installer

Make targets

  • To build a wheel file, run make dist

  • To build a pex file, run make pex after make dist

  • Builds for additional platforms are triggered from buildkite based on .buildkite/pipeline.yml

More on version numbers

Note

The content below is pulled from the docstring of the kolibri.utils.version module.

We follow semantic versioning 2.0.0 according to semver.org but for Python distributions and in the internal string representation in Python, you will find a PEP-440 flavor.

  • 1.1.0 (Semver) = 1.1.0 (PEP-440).

  • 1.0.0-alpha1 (Semver) = 1.0.0a1 (PEP-440).

Here’s how version numbers are generated:

  • kolibri.__version__ is automatically set, runtime environments use it to decide the version of Kolibri as a string. This is especially something that PyPi and setuptools use.

  • kolibri.VERSION is a tuple containing major, minor, and patch version information, it’s set in kolibri/__init__.py

  • kolibri/VERSION is a file containing the exact version of Kolibri for a distributed environment - when it exists, as long as its major, minor, and patch versions are compatible with kolibri.VERSION then it is used as the version. If these versions do not match, an AssertionError will be thrown.

  • git describe --tags is a command run to fetch tag information from a git checkout with the Kolibri code. The information is used to validate the major components of kolibri.VERSION and to add a suffix (if needed). This information is stored permanently in kolibri/VERSION before shipping any built asset by calling make writeversion during make dist etc.

This table shows examples of kolibri.VERSION and git data used to generate a specific version:

Release type

kolibri.VERSION

Git data

Examples

Final

(1, 2, 3)

Final tag: e.g. v1.2.3

1.2.3

dev release (alpha0)

(1, 2, 3)

timestamp of latest commit + hash

1.2.3.dev0+git.123.f1234567

alpha1+

(1, 2, 3)

Alpha tag: e.g. v1.2.3a1

Clean head: 1.2.3a1, 4 changes since tag: 1.2.3a1.dev0+git.4.f1234567

beta1+

(1, 2, 3)

Beta tag: e.g. v1.2.3b1

Clean head: 1.2.3b1, 5 changes since tag: 1.2.3b1.dev0+git.5.f1234567

rc1+ (release candidate)

(1, 2, 3)

RC tag: e.g. v1.2.3rc1

Clean head: 1.2.3rc1, Changes since tag: 1.2.3rc1.dev0+git.f1234567

Built assets: kolibri/VERSION is auto-generated with make writeversion during the build process. The file is read in preference to git data in order to prioritize swift version resolution in an installed environment.

Release order example 1.2.3 release:

  • VERSION = (1, 2, 3) throughout the development phase, this results in a lot of 1.2.3.dev0+git1234abcd with no need for git tags.

  • VERSION = (1, 2, 3) for the first alpha release, a git tag v1.2.3a0 is made.

Warning

Do not import anything from the rest of Kolibri in this module, it’s crucial that it can be loaded without the settings/configuration/django stack.

If you wish to use version.py in another project, raw-copy the contents of this file. You cannot import this module in other distributed package’s __init__, because setup.py cannot depend on the import of other packages at install-time (which is when the version is generated and stored).

Warning

Tagging is known to break after rebasing, so in case you rebase a branch after tagging it, delete the tag and add it again. Basically, git describe --tags detects the closest tag, but after a rebase, its concept of distance is misguided.