Getting started

First of all, thank you for your interest in contributing to Kolibri! The project was founded by volunteers dedicated to helping make educational materials more accessible to those in need, and every contribution makes a difference. The instructions below should get you up and running the code in no time!

Setting up Kolibri for development

Most of the steps below require entering commands into your Terminal, so you should expect to become comfortable with this if you’re not already.

If you encounter issues:

  • Searching online is often effective: chances are high that someone else encountered similar issues in the past

  • Please let us know if our docs can be improved, either by filing an issue or submitting a PR!

Note

Theoretically, Windows can be used to develop Kolibri, but we haven’t tested this lately. If you’re running Windows, you are likely to encounter issues with this guide. That said, we’d appreciate any help improving these docs for Windows developers!

Git and GitHub

  1. Install and set up Git on your computer. Try this tutorial if you need more practice with Git!

  2. Sign up and configure your GitHub account if you don’t have one already.

  3. Fork the main Kolibri repository. This will make it easier to submit pull requests. Read more details about forking from GitHub.

  4. Important: Install and set up the Git LFS extension.

Tip

Register your SSH keys on GitHub to avoid having to repeatedly enter your password

Checking out the code

First, clone your Kolibri fork to your local computer. In the command below, replace $USERNAME with your own GitHub username:

git clone git@github.com:$USERNAME/kolibri.git

Next, initialize Git LFS:

git lfs install

Finally, add the Learning Equality repo as a remote. That way you can keep your local checkout updated with the most recent changes:

cd kolibri  # Enter the Kolibri directory
git remote add upstream git@github.com:learningequality/kolibri.git
git fetch --all  # Check if there are changes upstream
git checkout develop # Checkout the development branch

Python and Pip

To develop on Kolibri, you’ll need:

  • Python 3.4+ or Python 2.7+

  • pip

Managing Python installations can be quite tricky. We highly recommend using package managers like Homebrew on Mac or apt on Debian for this.

Warning

Never modify your system’s built-in version of Python

Python virtual environment

You should use a Python virtual environment to isolate the dependencies of your Python projects from each other and to avoid corrupting your system’s Python installation.

There are many ways to set up Python virtual environments: You can use Pipenv as shown in the instructions below; you can also use Virtualenv, Python 3 venv, Poetry etc.

Note

Most virtual environments will require special setup for non-Bash shells such as Fish and ZSH.

Once Pipenv is installed, you can use the following commands to set up and use a virtual environment from within the Kolibri repo:

pipenv --python 3  # can also make a python 2 environment
pipenv shell  # activates the virtual environment

Now, any commands you run will target your virtual environment rather than the global Python installation. To deactivate the virtualenv, simply run:

exit

(Note that you’ll want to leave it activated for the remainder of the setup process)

Warning

Never install project dependencies using sudo pip install ...

Environment variables

Environment variables can be set in many ways, including:

  • adding them to a ~/.bash_profile file (for Bash) or a similar file in your shell of choice

  • using a .env file for this project, loaded with Pipenv

  • setting them temporarily in the current Bash session using EXPORT or similar (not recommended except for testing)

There are two environment variables you should plan to set:

  • KOLIBRI_RUN_MODE is required.

    This variable is sent to our pingback server (private repo), and you must set it to something besides an empty string. This allows us to filter development work out of our usage statistics. There are also some special testing behaviors that can be triggered for special strings, as described elsewhere in the developer docs and integration testing Gherkin scenarios.

    For example, you could add this line at the end of your ~/.bash_profile file:

    export KOLIBRI_RUN_MODE="dev"
    
  • KOLIBRI_HOME is optional.

    This variable determines where Kolibri will store its content and databases. It is useful to set if you want to have multiple versions of Kolibri running simultaneously.

Install Python dependencies

To install Kolibri project-specific dependencies make sure you’re in the kolibri directory and your Python virtual environment is active. Then run:

# required
pip install -r requirements.txt --upgrade
pip install -r requirements/dev.txt --upgrade
pip install -e .

# optional
pip install -r requirements/build.txt --upgrade
pip install -r requirements/test.txt --upgrade
pip install -r requirements/docs.txt --upgrade

Note that the --upgrade flags above can usually be omitted to speed up the process.

Install Node.js, Yarn and other dependencies

  1. Install Node.js (version 10 is required)

  2. Install Yarn

  3. Install non-python project-specific dependencies

The Python project-specific dependencies installed above will install nodeenv, which is a useful tool for using specific versions of Node.js and other Node.js tools in Python environments. To setup Node.js and Yarn within the Kolibri project environment, ensure your Python virtual environment is active, then run:

# node.js, npm, and yarn
nodeenv -p --node=10.15.3
npm install -g yarn

# other required project dependencies
yarn install

Running the Kolibri server

Database setup

To initialize the database run the following command:

kolibri manage migrate

Development server

To start up the development server and build the client-side dependencies, use the following command:

yarn run devserver

This will take some time to build the front-end assets, after which you should be able to access the server at http://127.0.0.1:8000/.

Alternatively, you can run the devserver with hot reload enabled using:

yarn run devserver-hot

Note that the default devserver commands above will automatically watch your source files for changes as you edit them, and do formatting and linting fixes on them. If you would prefer to do these on demand (such as with IDE linting tools or using a tool like pre-commit), then it is best to use the following commands, whereby linting and formatting errors will generate warnings, but not be fixed on the fly:

yarn run devserver-warn

Or:

yarn run devserver-hot-warn

Warning

Some functionality, such as right-to-left language support, is broken when hot-reload is enabled

Development server - advanced

The commands above will start multiple concurrent processes: One for the Django web server, and at least one more for the webpack devserver. If you’d like to start these processes separately, you can do it in two separate terminal windows.

In the first terminal you can start the django development server with this command:

yarn run python-devserver

In the second terminal, you can start the webpack build process for frontend assets in ‘watch’ mode – meaning they will be automatically rebuilt if you modify them – with this command:

yarn run watch

If you need to make the development server available through the LAN, you need to do a production build of the assets; so use the following commands:

# first build the assets
yarn run build
# now, run the Django devserver
yarn run python-devserver

Now you can simply use your server’s IP from another device in the local network through the port 8000, for example http://192.168.1.38:8000/.

Tip

If you get an error similar to “Node Sass could not find a binding for your current environment”, try running npm rebuild node-sass

Production

In production, content is served through CherryPy. Static assets must be pre-built:

# first build the assets
yarn run build
# now, run the Django production server
kolibri start

Now you should be able to access the server at http://127.0.0.1:8080/.

Developing on Kolibri inside Docker

Note

The Docker workflows below have not been fully tested

Users who are familiar with Docker can spin up a Kolibri instance quickly without setting up the full JavaScript and Python development environments. We provide docker images that contain all the necessary prerequisites for running Kolibri.

The docker/ directory contains the docker files and startup scripts needed for various tasks.

  • docker/base.dockerfile: the base layer that installs JavaScript and Python dependencies (image tag learningequality/kolibri).

  • docker/build_whl.dockerfile: generates a .whl, tar.gz, and .pex files in dist/

  • docker/build_windows.dockerfile: used to generate the Windows installer.

  • docker/dev.dockerfile: container with full development setup, running devserver.

  • docker/demoserver.dockerfile: runs the pex from KOLIBRI_PEX_URL with production setup.

  • docker/entrypoint.py: startup script that configures Kolibri based on ENV variables:

    • Set KOLIBRI_PEX_URL to string default to run latest pex from Kolibri download page

    • Set KOLIBRI_PEX_URL to something like http://host.org/nameof.pex

    • Set DOCKERMNT_PEX_PATH to something like /docker/mnt/nameof.pex

    • KOLIBRI_RUN_MODE: set in Dockerfile

    • KOLIBRI_PROVISIONDEVICE_FACILITY: if this environment variable is set the entrypoint script will run the provision device an setup a facility with this name. The KOLIBRI_LANG environment variable and the following other environment variables will be used in the process:

      • KOLIBRI_PROVISIONDEVICE_PRESET: defaults to formal, with the other options being nonformal and informal

      • KOLIBRI_PROVISIONDEVICE_SUPERUSERNAME: default devowner

      • KOLIBRI_PROVISIONDEVICE_SUPERUSERPASSWORD: default admin123

    • KOLIBRI_HOME: default /kolibrihome

    • KOLIBRI_HTTP_PORT: default 8080

    • KOLIBRI_LANG: default en

    • CHANNELS_TO_IMPORT: comma-separated list of channel IDs (not set by default)

Building a pex file:

When simply testing things out or reviewing a pull request, the easiest way to obtain a pex file is to get the link from the buildkite assets link that is present for every git branch and every pull request. This is the approach we recommend in combination with the demoserver approach for running described in the next section.

However, if you want to build and run a pex from the Kolibri code in your current local source files without relying on the github and the buildkite integration, you can run the following commands to build a pex file:

make docker-whl

The pex file will be generated in the dist/ directory. You can run this pex file using the demoserver approach described below.

Starting a demo server:

You can start a Kolibri instance running any pex file by setting the appropriate environment variables in your local copy of docker/env.list then running the commands:

make docker-build-base      # only needed first time
make docker-demoserver

The choice of pex file can be controlled by setting environment variables in the file docker/env.list:

  • Set KOLIBRI_PEX_URL to string default to run the latest pex from Kolibri download page

  • Set KOLIBRI_PEX_URL to something like http://host.org/nameof.pex

  • Set DOCKERMNT_PEX_PATH to something like /docker/mnt/nameof.pex

Starting a devserver:

# start the Kolibri devserver running inside a container
make docker-build-base  # only needed first time
make docker-devserver   # takes a few mins to run pip install -e + webpack build

Development workflows

Design system

We have a large number of reusable patterns, conventions, and components built into the application. Review the Kolibri Design System to get a sense for the tools at your disposal, and to ensure that new changes stay consistent with established UI patterns.

Linting and auto-formatting

Linting and code auto-formatting provided by Prettier and Black are run in the background automatically by yarn run devserver (see Database setup). You can monitor for linting errors and warnings in the terminal outputs of the dev server while it is running.

A full set of linting and auto-formatting can also be applied by pre-commit hooks (instructions below). The pre-commit hooks are identical to the automated build check by Travis CI in Pull Requests.

Tip

As a convenience, many developers install linting and formatting plugins in their code editor (IDE). Installing ESLint, Prettier, Black, and Flake8 plugins in your editor will catch most (but not all) code-quality checks.

You can manually run the auto-formatters using:

yarn run lint-frontend:format
yarn run fmt-backend

Or to check the formatting without writing changes, run:

yarn run lint-frontend
yarn run fmt-backend:check

pre-commit is used to apply a full set of checks and formatting automatically each time that git commit runs. If there are errors, the Git commit is aborted and you are asked to fix the error and run git commit again.

Pre-commit is already installed as a development dependency, but you also need to enable it:

pre-commit install

Note

Pre-commit can have issues running from alternative Git clients like GitUp. If you encounter problems while committing changes, run pre-commit uninstall to disable pre-commit.

Warning

If you do not use any linting tools, your code is likely fail our server-side checks and you will need to update the PR in order to get it merged.

Automated testing

Kolibri comes with a Javascript test suite based on Jest. To run all front-end tests:

yarn run test

Kolibri comes with a Python test suite based on pytest. To run all back-end tests:

pytest

To run specific tests only, you can add the filepath of the file. To further filter either by TestClass name or test method name, you can add -k followed by a string to filter classes or methods by. For example, to only run a test named test_admin_can_delete_membership in kolibri/auth/test/test_permissions.py:

pytest kolibri/auth/test/test_permissions -k test_admin_can_delete_membership

To only run the whole class named MembershipPermissionsTestCase in kolibri/auth/test/test_permissions.py:

pytest kolibri/auth/test/test_permissions -k MembershipPermissionsTestCase

For more advanced usage, logical operators can also be used in wrapped strings, for example, the following will run only one test, named test_admin_can_delete_membership in the MembershipPermissionsTestCase class in kolibri/auth/test/test_permissions.py:

pytest kolibri/auth/test/test_permissions -k "MembershipPermissionsTestCase and test_admin_can_delete_membership"

You can also use tox to setup a clean and disposable environment:

tox -e py3.4  # Runs tests with Python 3.4

To run Python tests for all environments, use simply tox. This simulates what our CI also does on GitHub PRs.

Note

tox reuses its environment when it is run again. If you add anything to the requirements, you will want to either delete the .tox directory, or run tox with the -r argument to recreate the environment

Updating documentation

First, install some additional dependencies related to building documentation output:

pip install -r requirements/docs.txt
pip install -r requirements/build.txt

To make changes to documentation, edit the rst files in the kolibri/docs directory and then run:

make docs

You can also run the auto-build for faster editing from the docs directory:

cd docs
sphinx-autobuild --port 8888 . _build

Manual testing

All changes should be thoroughly tested and vetted before being merged in. Our primary considerations are:

  • Performance

  • Accessibility

  • Compatibility

  • Localization

  • Consistency

For more information, see the next section on Manual testing & QA.

Submitting a pull request

Here’s a very simple scenario. Below, your remote is called origin, and Learning Equality is le.

First, create a new local working branch:

# checkout the upstream develop branch
git checkout le/develop
# make a new feature branch
git checkout -b my-awesome-changes

After making changes to the code and committing them locally, push your working branch to your fork on GitHub:

git push origin my-awesome-changes

Go to Kolibri’s GitHub page, and create a the new pull request.

Note

Please fill in all the applicable sections in the PR template and DELETE unecessary headings

Another member of the team will review your code, and either ask for updates on your part or merge your PR to Kolibri codebase. Until the PR is merged you can push new commits to your branch and add updates to it.

Learn more about our Development workflow and Release process