This document describes procedures for people assisting with FOLIO DevOps to add a new back-end module to the folio-ansible configuration, and make it available in the folio-snapshot and folio-testing reference environments, and make it available in the Vagrant box VMs for snapshot and testing.
The developers responsible for that module would have already ensured that the module is ready, its repository is established, and is operating in FOLIO CI. They would have added a Jira ticket, assigned to our team so that we can schedule and track the work.
Verification and preparation
This is the relevant guide that they would have followed. Still, we need to follow it again. There are tips not repeated here, and it defines verification steps that would be wise to re-follow before committing to infrastructure configuration.
If they have not indicated on their Jira ticket that they have verified that the module will install on a local VM, then remind them to follow the Ensure recent local VM instructions. Otherwise it could be a huge waste of your time.
Investigate the module README and LaunchDescriptor in case there are other special requirements.
Make branch in folio-ansible to add to
Make branch in folio-infrastructure to add to a special file, and also to configure Jenkins.
Configure and test via special Jenkins jobs.
Remove testing configuration.
Merge PR to folio-ansible, then to folio-infrastructure.
Follow the daily backend builds to verify.
Make a branch in folio-ansible,
Add the module to the
These files also assist with building the VMs, and currently provide special configuration for some modules.
Always add new modules to
Only add module entries to
group_vars/snapshot if they need special configuration to over-ride LaunchDescriptor settings, or to provide other additional settings.
If this is also a “core” module, then it will also be added to the corresponding
Normally only added to the main “complete” environments.
Follow the format for the entries of other modules.
Note that the “snapshot” files are alphabetically sorted, as Okapi handles those.
However, the “testing” files have explicit order. The sequence must have the new module declared after the modules that provide its interfaces (see verification step above).
Also note that for the “snapshot” files, the module is declared in the main section.
If the module is not to be pulled in by a front-end UI module via a platform, then it needs to be also declared in the “
Often that is the case because the development of a UI module is not yet ready to require it.
group_vars section could later be tidied, when that is finally happening.)
Make a branch in folio-infrastructure,
The purpose of this file is to declare modules (e.g. mod-ldp) that are not added to a VM (see the previous section) but do need to be added to the folio-testing reference environment.
Add the entry to both sections: Register and Deploy. Follow the format for the entries of other modules. The sequence must have the new module declared after any modules that provide its interfaces (see verification step above).
Add testing configuration for the Jenkins jobs …
Add to the Jenkinsfiles to refer to the folio-infrastructure branch:
BRANCH_TO_BUILD variable to be
Refer the git submodule to the folio-ansible branch. Take care, as this is a tab-delimited file.
[submodule "CI/ansible/folio-ansible"] path = CI/ansible/folio-ansible url = https://github.com/folio-org/folio-ansible branch = folio-2467-refenvs-ncip # <<< add this line <<<
Push the folio-ansible branch and the folio-infrastructure branch.
Jenkins testing configuration
So, select the
Configure link in the top-left panel.
Then in the section “Pipeline : Pipeline script from SCM : Branches to build” replace “Branch specifier” from
refs/heads/folio-2467-refenvs-ncip and then “Save”.
Run folio-testing-test build
If there have been master changes for folio-ansible and folio-infrastructure since branching, then do merge master to branch, and deal with conflicts.
Now run the test build. Select the
Build with Parameters link in the top-left panel.
Follow the results of this build run via its
Console Output and full log.
The AWS instance will be removed and rebuilt. It takes approximately 20-25 minutes.
If not success, then try to interpret the Jenkins output logfile. In some cases, may need to ssh to the instance and inspect the Okapi logfile.
Verify the testing build
After success, then ensure that the new back-end module is in place. (Example ‘curl’ of course needs token and tenant headers.)
curl -s -S \ https://folio-testing-test-okapi.dev.folio.org/_/proxy/tenants/diku/modules \ | jq -r '. | select(.id | match("mod-"))' | sort
Do ssh login to the AWS instance (search the Jenkins build output for “ec2-“ to get the DNS name) and confirm that the module’s docker logs are not spewing errors.
Could also verify via the front-end “Settings : Software versions” page (e.g. folio-testing-test.dev.folio.org). However remember that an automated job might be in-process to rebuild its front-end, as explained at the reference environments.
Add to snapshot platform
This is the separate procedure for adding a backend module to the “snapshot” branch of the Stripes Platform (e.g. to platform-complete).
Before doing this, the successful build of “folio-testing-test” (described in the preceding sections) will provide some comfort, but it is not a thorough test. Of course, it it fails then there is no point doing this section.
The normal process is that a UI module will require interfaces that are provided by back-end modules (declared in the “okapiInterfaces” section of their package.json file). With this, the relevant back-end modules are automatically included. (See separate procedure to install a new front-end module.)
However often the front-end and back-end are not yet ready for each other.
If so, then temporarily add the new back-end module to the
install-extras.json file of the “snapshot” branch of the relevant platform.
(Note that “core” modules are added to both platform-core and platform-complete.)
After the back-end development is eventually ready, then a front-end module will require this module.
When that happens, then the backend module will be automatically included via the normal process, so it can then be removed from this
However some back-end modules are never required by front-end modules. In these cases they are always configured in that file.
If approaching the daily reference environment build times, then the timing of this task can be difficult.
The build-platform-complete-snapshot Jenkins build happens every hour, starting at about 20 minutes past and taking 30 minutes. This will regenerate the yarn.lock and install files of the Platform, and automatically merge them.
So there will be a short window to prepare the branch and pull-request builds, then merge to “snapshot” branch, then its branch build.
After the next “build-platform-complete-snapshot” run, then verify that the new backend module is now in the okapi-install.json file.
Run folio-snapshot-test build
If there is trouble, then revert the addition to the Platform, and investigate and report the issues.
Do pull requests
Now that there is happiness, prepare and merge the pull-requests to folio-ansible and folio-infrastructure.
First be sure to remove all test configuration …
- At the Jenkins interface for “folio-testing-test” and “folio-snapshot-test”, revisit its
Configurepage and revert the “Branch specifier” to be
*/master(which also signals to other DevOps that you have finished with the test builds).
- In the “folio-infrastructure” branch, remove the testing configuration in the Jenkinsfiles and the gitmodules file.
Now make the PRs.
Be aware of the timing of the automated back-end build jobs, as explained at the reference environments.
(If you need to manually run these jobs outside the normal automated schedule, then may need to ask #hosted-reference-envs channel, as people depend on these environments.)
First merge to folio-ansible, then to folio-infrastructure.
Inspect build results
Following the merges, might want to re-run the folio-testing-test build using this now-deployed configuration to ensure no glitches.
Now await the scheduled automated builds.
Verify the results of each backend build, as done above. Remember that folio-snapshot-load will be the same as folio-snapshot.
curl -s -S \ https://folio-testing-okapi.dev.folio.org/_/proxy/tenants/diku/modules \ | jq -r '. | select(.id | match("mod-"))' | sort
To inspect via the front-end, note that the folio-testing-stripes build will automatically follow folio-testing-backend for example.
Delete old branches.
Comment and close the Jira ticket.