Distribution build pipeline

The Kolibri Package build pipeline looks like this:

                       Git release branch
                               |
                               |
                              / \
                             /   \
Python dist, online dependencies  \
   `python setup.py bdist_wheel`   \
               /                    \
              /                Python dist, bundled dependencies
       Upload to PyPi        `python setup.py bdist_wheel --static`
      Installable with                 \
    `pip install kolibri`               \
                                   Upload to PyPi
                                   Installable with
                             `pip install kolibri-static`
                               /            |          \
                              /             |           \
                        Windows          Android        Debian
                       installer           APK         installer

Make targets

  • To build a wheel file, run make dist

  • To build a pex file, run make pex after make dist

  • Builds for additional platforms are triggered from buildkite based on .buildkite/pipeline.yml

More on version numbers

Note

The content below is pulled from the docstring of the kolibri.utils.version module.

We follow semantic versioning 2.0.0 according to semver.org but for Python distributions and in the internal string representation in Python, you will find a PEP-440 flavor.

  • 1.1.0 (Semver) = 1.1.0 (PEP-440).

  • 1.0.0-alpha1 (Semver) = 1.0.0a1 (PEP-440).

Here’s how version numbers are generated:

  • kolibri.__version__ is automatically set, runtime environments use it to decide the version of Kolibri as a string. This is especially something that PyPi and setuptools use.

  • kolibri.VERSION is a tuple containing version information, it’s set in kolibri/__init__.py is automatically suffixed in pre-releases by a number of rules defined below. For a final release (not a pre-release), it will be used exactly as it appears.

  • kolibri/VERSION is a file containing the exact version of Kolibri for a distributed environment (pre-releases only!)

  • git describe --tags is a command run to fetch tag information from a git checkout with the Kolibri code. The information is used to validate the major components of kolibri.VERSION and to suffix the final version of prereleases. This information is stored permanently in kolibri/VERSION before shipping a pre-release by calling make writeversion during make dist etc.

Confused? Here’s a table:

Release type

kolibri.VERSION

kolibri/VERSION

Git data

Examples

Final

Canonical, only information used

N/A

N/A

0.1.0, 0.2.2, 0.2.post1

dev release (alpha0)

(1, 2, 3, ‘alpha’, 0), 0th alpha = a dev release! Never used as a canonical

Fallback

timestamp of latest commit + hash

1.2.3.dev0+git.123.f1234567

alpha1+

(1, 2, 3, ‘alpha’, 1)

Fallback

git describe --tags

Clean head: 1.2.3a1, 4 changes since tag: 1.2.3a1.dev0+git.4.f1234567

beta1+

(1, 2, 3, ‘alpha’, 1)

Fallback

git describe --tags

Clean head: 1.2.3b1, 5 changes since tag: 1.2.3b1.dev0+git.5.f1234567

rc1+ (release candidate)

(1, 2, 3, ‘alpha’, 1)

Fallback

git describe --tags

Clean head: 1.2.3rc1, Changes since tag: 1.2.3rc1.dev0+git.f1234567

beta0, rc0, post0, x.y.0

DO NOT USE

Fallback

timestamp of latest commit + hash

1.2.3b0.dev0+git.123.f1234567

Fallback: kolibri/VERSION is auto-generated with make writeversion during the build process. The file is read as a fallback when there’s no git data available in a pre-release (which is the case in an installed environment).

Release order example 1.2.3 release:

  • VERSION = (1, 2, 3, 'alpha', 0) throughout the development phase, this results in a lot of 1.2.3.dev0+git1234abcd with no need for git tags.

  • VERSION = (1, 2, 3, 'alpha', 1) for the first alpha release.

Warning

Do not import anything from the rest of Kolibri in this module, it’s crucial that it can be loaded without the settings/configuration/django stack.

If you wish to use version.py in another project, raw-copy the contents of this file. You cannot import this module in other distributed package’s __init__, because setup.py cannot depend on the import of other packages at install-time (which is when the version is generated and stored).

Warning

Tagging is known to break after rebasing, so in case you rebase a branch after tagging it, delete the tag and add it again. Basically, git describe --tags detects the closest tag, but after a rebase, its concept of distance is misguided.