The aim of this document is to describe our CICD pipeline and our release process.

1. Pipeline Configuration

This section briefly describes the organization of our CICD pipeline.

Most of the access and permissions required by our CICD plateform are managed by tokens that are created on each of the required services (SonarCloud, DockerHub, GitHub). A technical user account (opfabtech) has been created for each of these services so that these tokens are not linked to the account of any member of the team.

1.1. CICD Pipeline

1.1.1. Github Actions

We use github Actions to manage our pipeline (github.com/opfab/operatorfabric-core/actions).

1.1.2. SonarCloud

To be allowed to push results to SonarCloud, github needs to be authenticated. This is done by generating a token on SonarCloud with an account (opfabtech) that has admin rights to the organization, and then providing this token to github using actions secrets .

1.1.3. GitHub (documentation)

To be allowed to push the generated documentation to the opfab.github.io, Github needs write access to the repository. This is done by setting up a Personal Access Token in GitHub using the technical account. This token is then passed to Github using actions secrets .

After new content is pushed to the opfab.github.io repository, it can take a few minutes before this content is visible on the website because it needs to be built by GitHub pages, and this can take a short while depending on how busy the service is.

1.1.4. DockerHub

To be allowed to push images to DockerHub, Github needs to be authenticated. This is done again by generating a token in DockerHub using the technical account and provide it to github via actions secrets.

2. Release process

2.1. Version numbers

We work with two types of versions:

  • X.Y.Z.RELEASE versions are stable versions

  • SNAPSHOT version represents the current state of merged developments

Version numbers for X.Y.Z.RELEASE should be understood like this:

  • X: Major version, a major version adds new features and breaks compatibility with previous major and minor versions.

  • Y: Minor version, a minor version adds new features and does not break compatibility with previous minor versions for the same major version.

  • Z: Patch, a patch version only contains bug fixes of current minor version

2.2. Releasing a Major or Minor Version

To release a version we use some GitHub Actions dedicated jobs. These jobs are triggered when pushing a commit in branches containing the keyword release at the end (**release) and rely on the VERSION file at the root of this repository to know which version is being produced. It is thus crucial to double-check the content of this file before any push (triggering the GitHub Actions jobs) is made.

Before releasing a version, you need to prepare the release.

2.2.1. Checking the release notes

  1. On the Projects tab of the core repository, click on the project corresponding to the release that is about to be released.

  2. Make sure that the release_notes.md file lists all the issues, bugs, tags or feature requests that are relevant for OperatorFabric users along with explanations if need be.

  3. Based on the content of this version and the rules listed above, make sure that the version number is appropriate.

  4. Check if there is a migration guide for this version, if so, check if the corresponding file has been included in src/docs/asciidoc/resources/index.adoc and include a link to it at the top of the release notes.

2.2.2. Creating a release branch and preparing the release

  1. On the operatorfabric-core repository, create a branch off the develop branch named X.X.X.release (note the lowercase release to distinguish it from X.X.X.RELEASE tags).

    git checkout -b X.X.X.release
  2. Use the ./CICD/prepare_release_version.sh script to automatically perform all the necessary changes:

    ./CICD/prepare_release_version.sh -v X.X.X.RELEASE

    You should get the following output:

    Current version is SNAPSHOT (based on VERSION file)
    Preparing X.X.X.RELEASE
    Updating version for pipeline in VERSION file
    Replacing SNAPSHOT with X.X.X.RELEASE in swagger.yaml files
    Using X.X.X.RELEASE for lfeoperatorfabric images in dev and docker environment docker-compose files
    The following files have been updated:
     M VERSION
     M config/dev/docker-compose.yml
     M config/docker/docker-compose.yml
     M services/cards-publication/src/main/modeling/swagger.yaml
     M services/businessconfig/src/main/modeling/swagger.yaml
     M services/users/src/main/modeling/swagger.yaml

    This script performs the following changes:

    • Replace SNAPSHOT with X.X.X.RELEASE in swagger.yaml files and the VERSION file at the root operator-fabric folder

    • Change the version from SNAPSHOT to X.X.X.RELEASE in the docker-compose files for dev and docker deployments

  3. Commit the changes with the template message:

    git add .
    git commit -s -m "[RELEASE] X.X.X.RELEASE"
  4. Push the commit

    git push --set-upstream origin X.X.X.release
  5. Check that the build is correctly triggered

    You can check the status of the GitHub Actions triggered by the commit on Github Actions.

Wait for the build to complete (around 30 minutes) and check that all jobs have been successful. This ensures that the code builds, tests are OK and there is no error preventing documentation or Docker images generation.

2.2.3. Merging the release branch into master

Once the release branch build is passing, you should merge the release branch into master to bring the new developments into master and trigger the CICD tasks associated with a release (Docker images for DockerHub and documentation).

git checkout master (1)
git pull (2)
git merge X.X.X.release (3)
1 Check out the master branch
2 Make sure your local copy is up to date
3 Merge the X.X.X.release branch into master, accepting changes from X.X.X.release in case of conflicts.
git tag X.X.X.RELEASE (1)
git push (2)
git push origin X.X.X.RELEASE (3)
1 Tag the commit with the X.X.X.RELEASE tag
2 Push the commits to update the remote master branch
3 Push the tag
  1. Check that the build is correctly triggered

    You can check the status of the GitHub Action workflow triggered by the commit on Github Actions. The Github Actions workflow should have the following six jobs executed : - Build - Karate tests - Cypress tests - Publish Documentation (latest) - Push images to dockerhub - Push images latest to dockerhub

Wait for the build to complete (around 40 minutes) and check that all jobs have been successful.

  1. Check that the X.X.X.RELEASE images have been generated and pushed to DockerHub.

  2. Check that the latest images have been updated on DockerHub (if this has been triggered).

  3. Check that the documentation has been generated and pushed to the GitHub pages website : check the version and revision date at the top of the documents in the current documentation (for example the architecture documentation)

  4. Check that the tag was correctly pushed to GitHub and is visible under the tags page for the repository.

2.2.4. Updating the version list on the website

On the website repository, edit the /_data/versions.yml file to:

  1. Add the version being released to the list with the current badge

  2. Remove the current badge from the previous version

For example:

Before
- id: SNAPSHOT
  type: SNAPSHOT
  external_devices_api: true
- id: D.E.F.RELEASE
  badge: current
  external_devices_api: true
- id: A.B.C.RELEASE
  #... end of file omitted
After
- id: SNAPSHOT
  type: SNAPSHOT
  external_devices_api: true
- id: X.X.X.RELEASE
  badge: current
  external_devices_api: true
- id: D.E.F.RELEASE
  external_devices_api: true
- id: A.B.C.RELEASE
  #... end of file omitted

This file determines which versions (and in which order) are displayed on the release page of the website.

Check that you see the X.X.X.RELEASE under the releases page and that the links work (It may need a few minutes for the website to be updated).

The external_devices_api property should be set to true for all new versions, so the API documentation for the External Devices API is displayed on the website.

2.2.5. Checking the docker-compose files

While the docker-compose files should always point to the SNAPSHOT images while on the develop branch, on the master branch they should rely on the latest RELEASE version available on DockerHub. Once the CI pipeline triggered by the previous steps has completed successfully, and you can see X.X.X.RELEASE images for all services on DockerHub, you should:

  1. Remove your locally built X.X.X.RELEASE images if any

  2. Run the config/docker docker-compose file to make sure it pulls the images from DockerHub and behaves as intended.

People who want to experiment with OperatorFabric are pointed to this docker-compose so it’s important to make sure that it’s working correctly.

2.2.6. Publishing the release on GitHub

  1. On the releases screen for the core repository, draft a new release.

    1. Select the existing X.X.X.RELEASE tag

    2. The title should be X.X.X.RELEASE

    3. In the description field, paste the content from the release_notes.md file from the release-notes repository.

    4. Replace any "TODO" comments with the appropriate links to the documentation.

    5. Click "Publish release"

2.2.7. Publishing the jars for the client library to Maven Central

Once everything else looks ok, you can publish the jars for the client library to MavenCentral. This is done as a last step once we are pretty sure we won’t need to go back and change things on the release because jars are not meant to be removed from Maven Central once they are published (even briefly), and it’s not something that could be managed by the project.

To do so:

  1. Set the appropriate properties (credentials and GPG key information) as described in the documentation for the publishing task

  2. Run the following command from the project root:

    ./gradlew publish
  3. After a while you should be prompted to enter the passphrase for the GPG key.

  4. Once the task has completed, log in to the OSSRH Repository using the same credentials as for the Sonatype JIRA.

    Welcome page for the OSSRH repository manager
  5. Click on Staging repositories link on the left. After a while (and maybe after clicking the refresh button), you should see a repository with the name orgopfab-XXXX (where XXXX is a Sonatype-generated id, not related to the release number).

    Staging repositories
  6. Click on the repository then on the "content" tab below to check its content and metadata.

    Check staging repository
  7. If there is an issue with the repository, click on the "Drop" button and start the process again after making the necessary changes. If everything looks in order, click on the "Close" button and add a small comment when prompted to confirm.

    Close staging repository
  8. This will trigger validation of the Sonatype requirements (for example, making sure that the pom file contains the required information), as you can see from the Activity tab below (Refresh might be needed).

    Closing and validation of the staging repository
  9. If all the validations pass, the "Release" button will become available. Click it to send the jars to Maven Central. When prompted, write a comment then confirm (keeping the "Automatically Drop" option checked).

    Release to Maven Central
  10. The jars for the release should then be available on the project space in the Maven repository within 10 minutes.

  11. It can take up to two hours for them to appear on the Maven Central Repository Search.

2.2.8. Advertising the new release on the LFE mailing list

  1. Send an email to the opfab-announce@lists.lfenergy.org mailing list with a link to the release notes on GitHub.

Here is the link to the administration website for the LFE mailing lists in case there is an issue.

2.2.9. Preparing the next version

2.2.9.1. On the release-notes repository

Remove the items listed in the release_notes.md file so it’s ready for the next version.

2.2.9.2. On the operatorfabric-core repository

Now that the release branch has served its purpose, it should be deleted so as not to clutter the repository and to avoid confusion with the actual release commit tagged on master.

git branch -d X.X.X.release (1)
1 Delete the branch locally
You should also delete the branch on GitHub.

You should also close the project for this version, and create one for the next version if it doesn’t already exist (use the "Automated Kanban" template).

2.3. Releasing a Patch (Hotfixes) Version

Let’s say fixes are needed on version X.X.0.RELEASE, and will be released as X.X.X.RELEASE. If it’s the first patch version to be released for this minor version (i.e. version X.X.1.RELEASE), you will need to create the X.X.hotfixes branch. To do so:

git checkout X.X.0.RELEASE (1)
git checkout -b X.X.hotfixes (2)
1 Checkout X.X.0.RELEASE tag
2 Create (and checkout) branch X.X.hotfixes from this commit

If branch X.X.hotfixes already exists, you can just check it out.

git checkout X.X.hotfixes

Then, follow the process described

here to create feature branches, work on fixes and merge them back into X.X.hotfixes.

Once all the big fixes that need to go into the version X.X.X.RELEASE have been merged into branch X.X.hotfix, you can release the patch version. To do so:

  1. Write a release notes detailing the bug fixes in the release_notes.md file found under src/docs/asciidoc/docs in the operatorfabric-core repository.

  2. Use the ./CICD/prepare_release_version.sh script to automatically perform all the necessary changes:

    ./CICD/prepare_release_version.sh -v X.X.X.RELEASE
  3. Commit the changes, tag them and push both to GitHub:

    git add .
    git commit -m "[RELEASE] X.X.X.RELEASE " (1)
    git tag X.X.X.RELEASE (2)
    git push --set-upstream origin X.X.hotfixes (3)
    git push origin X.X.X.RELEASE (4)
    1 Commit the changes
    2 Tag the release
    3 Push the commit
    4 Push the tag

This will trigger the build and tests in GitHub Actions.

If the build and tests are successful, launch manually GitHubActions with jobs : Build , Docker Push and Build and publish documentation

In the case of a patch on the last major/minor version tagged on master, this version will become the latest version. In this case, add the jobs Docker Push - Latest and Build and publish documentation - Latest instead of Build and publish documentation to also update the latest docker images on DockerHub and the current documentation on the website.

You then need to release the client library jars for the hotfix version. To do so, refer to the corresponding section for standard releases.