The aim of this document is to describe our CICD pipeline and our release process.

1. Pipeline Configuration

This section briefly describes the organization of our CICD pipeline. If you are looking for more detailed information, see this document describing the steps that were necessary to create our mock pipeline as well as the issues we ran into.

Most of the access and permissions required by our CICD plateform (Travis) are managed by tokens that are created on each of the required services (SonarCloud, DockerHub, GitHub). A technical user account (opfabtech) has been created for each of these services so that these tokens are not linked to the account of any member of the team.

1.1. CICD Pipeline

1.1.1. GitHub

Work in progress

1.1.2. Travis CI

We use Travis CI to manage our pipeline. As of today, it is composed of 7 stages:

test-sonar

Builds the commit, runs tests and sonar analysis

test

Similar to test-sonar but without sonar analysis

doc

Generates the documentation (from asciidoc sources and API documentation) and pushes it to the opfab.github.io repository to update the website.

doc-dry-run

Generates the documentation without pushing it

docker-push-version

Builds Docker images, tags them with the current version (either SNAPSHOT or X.X.X.RELEASE) and pushes them to DockerHub

docker-push-latest

Builds Docker images, tags them with latest and pushes them to DockerHub

docker-tag-version

Builds Docker images and tags them with the current version, without pushing them anywhere. This stage can be triggered when we just want to check that the images can be built without actually updating them on DockerHub.

Among these stages, four can be considered "sensitive" because they push public content that serves as a reference for the project (docker images, documentation and to a lesser extent, sonar analysis), meaning we don’t want it tampered with. These stages are test-sonar, doc and the two docker-push stages.

These stages are triggered depending on:

  • branch type

  • event type (CRON job, push or pull request)

  • commit message hooks

In the table below:

doc hook

stands for adding the keyword ci_documentation to the commit message

docker hook

stands for adding the keyword ci_docker to the commit message

latest hook

stands for adding the keyword ci_latest to the commit message

Summary of stage triggers

develop

release

master

feature or hotfix

pull request

Stage

CRON

push

doc hook

docker hook

push

push

latest hook

push

doc hook

docker hook

push

test-sonar

X

X

X

X

X

internal

test

external

doc

X

X

X

doc-dry-run

X

X

docker-push-version

X

X

X

docker-push-latest

X

docker-tag-version

X

X

  • The test-sonar phase is ran for every build except those triggered by external PRs (i.e. originating from a fork of the repository). This is because the sonar-scanner step it comprises requires access to an encrypted token (to be able to push the analysis to SonarCloud, see below for details) that is not shared with external PRs for security reasons, so this would cause the stage (and the build) to fail. This is why in the case of external PRs the test phase is ran instead (leaving out sonar-scanner).

1.1.3. SonarCloud

To be allowed to push results to SonarCloud, Travis needs to be authenticated. This is done by generating a token on SonarCloud with an account (opfabtech) that has admin rights to the organization, and then providing this token to Travis either through the .travis.yml file or as an environment variable through Travis settings.

1.1.4. GitHub (documentation)

To be allowed to push the generated documentation to the opfab.github.io, Travis needs write access to the repository. This is done by setting up a Personal Access Token in GitHub using the technical account. This token is then passed to Travis as an environment variable through Travis settings, and is used in the .travis.yml file. Right now the scope of this token is maximal, it can probably be reduced (see OC-755).

After new content is pushed to the opfab.github.io repository, it can take a few minutes before this content is visible on the website because it needs to be built by GitHub pages, and this can take a short while depending on how busy the service is.

1.1.5. DockerHub

To be allowed to push images to DockerHub, Travis needs to be authenticated. This is done by setting the DockerHub login and password of the technical account as environment variables through Travis settings and referencing them in the .travis.yml file.

2. Release process

2.1. Version numbers

We work with two types of versions:

  • X.Y.Z.RELEASE versions are stable versions

  • SNAPSHOT version represents the current state of merged developments

Version numbers for X.Y.Z.RELEASE should be understood like this:

  • X: Major version, a major version adds new features and breaks compatibility with previous major and minor versions.

  • Y: Minor version, a minor version adds new features and does not break compatibility with previous minor versions for the same major version.

  • Z: Patch, a patch version only contains bug fixes of current minor version

2.2. Releasing a Version

To release a version we use some Travis dedicated jobs. These jobs are triggered by specific commit keywords and rely on the VERSION file at the root of this repository to know which version is being produced. It is thus crucial to double-check the content of this file before any push (triggering the Travis jobs) is made.

Before releasing a version, you need to prepare the release.

2.2.1. Checking the release notes

  1. Click the Next Release from JIRA the release list to get the release notes (click "Release notes" under the version name at the top) listing new features, fixed bugs etc…​

    Release notes link
  2. Make sure that the release_notes.adoc file lists all the issues, bugs, tags or feature requests that are relevant for OperatorFabric users along with explanations if need be.

  3. Based on the content of this version and the rules listed above, determine the version number for next version.

2.2.2. Creating a release branch and preparing the release

  1. On the operatorfabric-core repository, create a branch off the develop branch named X.X.X.release (note the lowercase release to distinguish it from X.X.X.RELEASE tags).

    git checkout -b X.X.X.release
  2. Cut the contents from the release_notes.adoc file from the release-notes repository and paste it to the release_notes.adoc file found under src/docs/asciidoc/docs in the operatorfabric-core repository.

  3. In the operatorfabric-core repository release-notes.adoc file, replace the Version SNAPSHOT title by Version X.X.X.RELEASE

  4. In the releases page on JIRA, change the name from "Next Version" to "X.X.X.RELEASE"

  5. Use the ./CICD/prepare_release_version.sh script to automatically perform all the necessary changes:

    ./CICD/prepare_release_version.sh -v X.X.X.RELEASE

    You should get the following output:

    Current version is SNAPSHOT (based on VERSION file)
    Preparing X.X.X.RELEASE
    Updating version for pipeline in VERSION file
    Replacing SNAPSHOT with X.X.X.RELEASE in swagger.yaml files
    Using X.X.X.RELEASE for lfeoperatorfabric images in dev and docker environment docker-compose files
    The following files have been updated:
     M VERSION
     M config/dev/docker-compose.yml
     M config/docker/docker-compose.yml
     M services/core/cards-publication/src/main/modeling/swagger.yaml
     M services/core/businessconfig/src/main/modeling/swagger.yaml
     M services/core/users/src/main/modeling/swagger.yaml

    This script performs the following changes:

    • Replace SNAPSHOT with X.X.X.RELEASE in swagger.yaml files and the VERSION file at the root operator-fabric folder

    • Change the version from SNAPSHOT to X.X.X.RELEASE in the docker-compose files for dev and docker deployments

  6. Commit the changes with the template message:

    git add .
    git commit -m "[RELEASE] X.X.X.RELEASE"
  7. Push the commit

    git push --set-upstream origin X.X.X.release
  8. Check that the build is correctly triggered

    You can check the status of the build job triggered by the commit on Travis CI. The build job should have the following three stages:

    Running build for release branch screenshot

    Wait for the build to complete (around 20 minutes) and check that all stages have been successful. This ensures that the code builds, tests are OK and there is no error preventing documentation or Docker images generation.

2.2.3. Merging the release branch into master

Once the release branch build is passing, you should merge the release branch into master to bring the new developments into master and trigger the CICD tasks associated with a release (Docker images for DockerHub and documentation).

git checkout master (1)
git pull (2)
git merge X.X.X.release (3)
1 Check out the master branch
2 Make sure your local copy is up to date
3 Merge the X.X.X.release branch into master
If you also want the new docker images to be tagged latest (as should be the case for most release versions), you should add the keyword ci_latest to the merge commit message.
git tag X.X.X.RELEASE (1)
git push (2)
git push origin X.X.X.RELEASE (3)
1 Tag the commit with the X.X.X.RELEASE tag
2 Push the commits to update the remote master branch
3 Push the tag
  1. Check that the build is correctly triggered

    You can check the status of the build job triggered by the commit on Travis CI. The build job should have the following four stages (or three if the images aren’t tagged as latest)

    Running build for master branch screenshot

    Wait for the build to complete (around 20 minutes) and check that all stages have been successful.

  2. Check that the X.X.X.RELEASE images have been generated and pushed to DockerHub.

  3. Check that the latest images have been updated on DockerHub (if this has been triggered).

  4. Check that the documentation has been generated and pushed to the GitHub pages website

    1. Check the version and revision date at the top of the documents in the current documentation (for example the architecture documentation)

    2. Check that you see the X.X.X.RELEASE under the releases page and that the links work.

  5. Check that the tag was correctly pushed to GitHub and is visible under the releases page for the repository.

2.2.4. Checking the docker-compose files

While the docker-compose files should always point to the SNAPSHOT images while on the develop branch, on the master branch they should rely on the latest RELEASE version available on DockerHub. Once the CI pipeline triggered by the previous steps has completed successfully, and you can see X.X.X.RELEASE images for all services on DockerHub, you should:

  1. Remove your locally built X.X.X.RELEASE images if any

  2. Run the config/demo docker-compose file to make sure it pulls the images from DockerHub and behaves as intended.

People who want to experiment with OperatorFabric are pointed to this docker-compose so it’s important to make sure that it’s working correctly.

2.2.5. In Jira

In the "Releases" screen, release X.X.X.RELEASE.

2.3. Advertising the new release on the LFE mailing list

  1. Send an email to the opfab-announce@lists.lfenergy.org mailing list with a link to the release notes on the website.

Here is the link to the administration website for the LFE mailing lists in case there is an issue.

2.4. Preparing the next version

You should wait for all the tasks associated with creating the X.X.X.RELEASE version to finish and make sure that they’ve had the expected output before starting the preparation of the next version. This is because any committed/pushed changes preparing the new version will make rolling back or correcting any mistake on the release more complicated.

2.4.1. In Jira

In the "Releases" screen create a new release called Next Release.

2.4.2. On the release-notes repository

Remove the items listed in the release_notes.adoc file so it’s ready for the next version.

2.4.3. On the operatorfabric-core repository

Now that the release branch has served its purpose, it should be deleted so as not to clutter the repository and to avoid confusion with the actual release commit tagged on master.

git branch -d X.X.X.release (1)
​git push origin --delete X.X.X.release (2)
1 Delete the branch locally
2 Remove it from GitHub