Python Environment

This page contains getting started and how-to information for Bessy python developers (The github README.md files are public user documentation).

Setup

The Bessy Python repository is structured as a development environment (consumers of Bessy install it using pip). Use the following steps to set up for Bessy Python development:

1. Install VSCode for editing
2. Install Miniconda for controlling python environment (smaller than Conda)

• If you don’t like Conda automatically activating the “base” environment in every terminal window, disable it with conda config --set auto_activate_base False

3. Clone the Bessy repository
4. Set up the Conda environment for the repository:

cd path-to-repository
conda env create -f ./environment.yml 

5. Install dependencies into the Conda environment and make bci-essentials package in editable mode:

pip install -e .

6. Ensure VSCode is using the python interpreter from your conda environment (bci). For more information see: desired Python interpreter. Typically this is done with ctrl + shift + p, type ‘python select interpreter’ and then choose the correct path.
7. Run the smoke test to make sure that Bessy is working correctly:

python -m unittest

Code checks

We use Python black for code formatting, and flake8 for linting. These checks are run automatically prior to commits and PRs, and will halt the action if any errors are found. However, you should run these tools yourself before committing changes to the repo so that you can catch errors without waiting for a failed commit; this can be done automatically, manually, or using VS Code extensions as described below. For sake of simplicity, it is recommended to install the VS Code extensions.

Note: Black does not format comments, so be mindful of the length of your multiline comments.

Automatic

1. Update your Conda environment using the environment.yml file which includes black, flake8 and pre-commit.
2. In your terminal, navigate to the root folder of the project and run pre-commit install. This should install the contents in the.pre-commit-config.yaml file.
3. After doing a GIT commit call, the pre-commit routine will run black and flake8 for you, if there are any issues with the format of the files committed, black will modify these and throw a failed warning. Meanwhile flake8 will notify you of any errors, but will not automatically fix them.
4. After resolving any issues, re-stage and re-commit the files using GIT.

Manual

1. Install black using pip install black.
2. Install flake8 using pip install flake8.
3. Run both tools before doing a commit by running the commands black . and flake8 . in the root of the project.

Extension

1. Install the Black formatter and Flake8 VS Code extensions by MS.
2. Navigate to your settings.json file by searching settings json using the command palette (Ctrl + Shift + P in Windows or Shift + Command + P on Mac).
3. Add the following lines to the file:

"[python]": {
  "editor.defaultFormatter": "ms-python.black-formatter",
  "editor.formatOnSave": true
}

4. Your Python files will be autoformatted after saving the file, and any flake8 errors will be highlighted in the open file.

Why Conda?

Python is like any other scripting language that grew up in a Unix environment (perl, tcl, bash, etc). There’s an interpreter (python) that needs to live in your path. The interpreter can either run in an interactive shell mode, or read in a script file (ex: myprogram.py). If your script imports a dependency, the interpreter scans your whole system for it.

Out of the box, python doesn’t have a concept of project-level dependency management. When the package manager pip installs a dependency, it will install it for the whole system, not just your own project. When two projects need different versions of the same dependency (ex: numpy) one of them breaks - this is a classic source of “works for me” problems.

Tools like Conda (smaller sibling miniconda) provide project-level dependency management. These tools set up an “environment” which makes python or pip only look in certain places for dependencies. You can think of it as a simple form of containerization.

https://xkcd.com/1987/

Making Changes

Changes should be made in small incremental steps following a branch-per-feature model. If the feature is going to take several sprints, consider breaking it down into smaller features. This avoids big ugly merges and keeps the team synchronized to the same code base. When the change is complete, a Pull Request is used to merge the change back into main.

1. Work on a branch

Perform code changes on a branch. Commit changes to your branch in small logical steps. Make commit comments brief but descriptive (“changed code” is useless to anyone else). Push your branch to the remote (GitHub) so that others can follow your changes and help if need be.

2. Sync your branch with main and test

When your changes are done, check that they are compatible with the latest changes on main. This is done by merging main into your branch and then testing. Address any merge conflicts on your branch. For larger changes, sync with main regularly, perhaps every day.

Test that your changes haven’t broken anything. Test that your changes work as expected. When done, make sure all your changes have been pushed to the remote branch.

To quickly test data loading and classification for MI, P300, and SSVEP:

python -m unittest

TODO - call out the minimum tests to run for regression (we need a smoke test or maybe unit tests can do this)

3. Create pull request

Using GitHub, create a Pull Request (PR) to merge your branch into main. GitHub will warn you if the changes cannot be merged. If that happens, go back and sync your branch with main.

Assign someone from the team as a reviewer of the PR. Include in the PR notes about what changed (link to Jira task), how to test it, what to look for, etc. GitHub should notify the reviewer that they have been assigned by email. It’s a good practice to also send them a message on Slack.

The reviewer will independently test your changes, review code and provide feedback. When the PR is created, an automated build will also run a set of tests. To resolve issues, make changes on your local branch and commit, sync and push to the remote. GitHub will automatically detect the push and restart automated tests.

The reviewer, when satisfied, will approve the PR and merge changes into main (there’s a button for this).

TODO - we may care more about the git history given the repo is public. provide recommendations on how to make the history clean (ex: squashing merges / rebasing instead of simple merges)

4. Clean up

When the PR is complete delete the remote branch on GitHub. It is also good practice to clean up your local workspace by removing the remote tracking branches (git pull --prune) and deleting the local branch (git branch -d my_branch).

Publishing Packages

The official Bessy package is hosted on the Pypi package repository. Technical users shall obtain Bessy using pip install bci-essentials.

The Bessy package is a source distribution (sdist). The contents of the package and dependencies are declared in pyproject.toml. Setuptools builds the package and Twine publishes to Pypi. These development dependencies are declared in environment.yml (installed when the Conda environment is set up). For more information see Python packaging guide and Setup tools quickstart.

Automatic publishing

Packages are built and published automatically when a version tag is created. This is done so that we have a record of exactly what source was used to create the package. This is useful if we need to patch a past release, etc.

1. acceptance testing

TODO - insert acceptance testing requirements before publishing

TODO - insert release branch tips (probably can just go from main with a small team)

2. set package version number

Package version numbers shall follow a semantic versioning format: a.b.c, where: a.b indicate new features and .c indicates a defect fix / minor changes. We don’t bother with alpha and beta designations - users wanting the very latest Bessy can just clone the repo.

Set the version number of the package in pyproject.toml. Ensure this has been pushed up to GitHub.

3. tag the commit

Tags must use match the version in pyproject.toml. Use the format va.b.c to trigger the publish action, for example, v1.0.2.

Ensure your local repository is up to date and the last commit is the one you want to tag. To create the tag use the command below (or you can do this from GitHub):

git tag -a v1.0.2
...
(annotate tag with "bci-essentials-1.0.2")
...
git push --tags

The publishing action will fire as soon as the tag is pushed. You can watch the progress here: https://github.com/kirtonBCIlab/bci-essentials-python/actions. The process takes ~4 minutes to complete.

Note on credentials - GitHub secrets are used to pass credentials to the publish action (via environment variables). The secrets are defined here: https://github.com/kirtonBCIlab/bci-essentials-python/settings/secrets/actions, where the username is token and the password is a token string provided by Pypi.

4. confirm package was published

Check that the expected package is now available on Pypi.

Manual publishing (ex: for testing)

Although it is possible to build a package on your own computer and publish to pypi manually, it is better to use the automated build. GitHub runners provide a known, repeatable configuration from which to build packages.

To manually create a package in dist/

python -m build

To manually publish the package to the TestPypi repository. You’ll need to set up an account on TesyPypi to do this.

twine upload --repository testpypi dist/*

To install packages from TestPypi (note - some Bessy dependencies don’t live in the test repo, so they are pulled from the real Pypi):

pip install -i https://test.pypi.org/simple --extra-index-url https://pypi.org/simple bci-essentials