I recently joined the CCS (Carbon capture and storage) group at Equinor. Part of our responsibilities is to understand the tools that are being used, and as I am already somewhat familiar with Linux and such I begged my team to take on deployment ownership of OpenGoSim's PFLOTRAN-OGS. At the time it was deployed by the infrastructure team who would rather do other things.

Equinor's Linux cluster conists of multiple virtual machines running desktop RHEL 8 and a few non-virtual machines using IBM LSF. Users order a VM on a webpage and are able to remote them, with all of their private and shared data stored on NFS. Most non-system software is also "distributed" this way. Programs are installed to various locations on a /prog disk that also resides on NFS. This way, users have access to everything they need to do their jobs regardless of where they physically sit or where they are virtually logged in to.

As users don't have ownership of their own machines, nor the expectation of installing their own software, this creates some requirements for us. I call these the Four Commandments:

1. Software is expected to exist everywhere at the same time.
2. Users should not manually install software.
3. It must be possible to update the software without affecting existing users.
4. It must be possible to continue to use an older version of the program.

This, of course, means that the standard way to install software on Linux, using either apt or yum/dnf depending on the specific distribution, is incompatible with all these commandments.

Prior art

A lot of software is deployed in Equinor and we can learn a lot from what others have done before. Vendors will often install their software in their own versioned directory, and use some sort of script to distinguish between them. For example, it is possible to have a "top-level" script that chooses the version (maybe choosing the latest if no version was provided).

Another solution is to keep each version entirely separate with nothing "binding" them together. In this case, upgrading software involves an "email step" where the deployer tells the user to use the newer version.

In either case, there is a separate install directory for each version, and this satisfies commandment three (don't overwrite) and possibly commandment four (allow rollbacks).

My team has previously been closely involved in the Komodo project, used to distribute Python programs and libraries used in underground reservoir modeling and simulation. We took a lot of inspiration from this, and we especially like the symbolic version representations, like stable meaning the newest tested version, and decided to go ahead with a similar system.

Another important aspect of Komodo is that it deliberately has very few manual steps, and the configuration itself is version-controlled. Automation allows for higher trust in the final product, and having version control allows us to recall why decisions were made. The ideal scenario is that a new member of the team can successfully deploy a new version with nothing going wrong.

Building

PFLOTRAN-OGS is made of multiple different components. There is the program itself, but also its dependencies (not developed by OpenGoSim, but by others) PETSC, Hypre, HDF5 and others. Some components are standard enough that we can consider them to exist on any reasonably-configured machine. These include a modern-ish version of Python, C and Fortran compilers, and other such "glue" software. For parallelisation, PFLOTRAN-OGS uses the MPI protocol, which can be provided by either OpenMPI or MPI-CH, both of which are open-source projects. After a tense coin-flip, we arrived at OpenMPI.

PETSC has facilities for automatically downloading and building dependencies for PFLOTRAN-OGS (ie. Hypre and HDF5 that were mentioned earlier). This means that there are a total of two programs that we need to build. Luckily for us, this is the next-simplest possible scenario.

Care is taken to create an install that works also outside of Equinor environments. This has a real-world benefit in that it may work in less complex situations, like my laptop or as part of a testing pipeline using GitHub Actions, without ever exposing Equinor internal network to throwaway simulations. Another benefit is that this makes our builds more resistant to changes in environments. Say, the software has a "hidden" dependency by virtue of being installed by someone in infrastructure some years ago. If we only build it in the Equinor environment, we may not discover this dependency until a major change happens, like upgrading to the next version of Red Hat Linux. This way, we know ahead of time what to expect and won't panic when something breaks. For PFLOTRAN-OGS specifically, the software requirements are easy to fulfil: Standard GNU/Posix environment, reasonably recent C, C++ and Fortran compilers, and a Python interpreter (3.4 and above).

The actual commands are simple. We automate downloading the code for the exact version we're building, extract it, then use the standard old UNIX-y ./configure, make, make install set of commands. Of course, the configuration is a tiny bit more complicated, but the general gist is that we enable some required settings in PETSC and override the default install directory to somewhere in /prog/pflotran/versions/.store/ (the standard location is /usr/local, which is physically on the machine's SSD and not the network file share).

The building process takes time. When recompiling PFLOTRAN-OGS, we don't necessarily need to recompile PETSC, a process that can take around 20 minutes. To save on time and (negligibly) disk space, we have two scripts - build_petsc and build_pflotran_ogs - which we run after each other.

Tying it all together

At this stage, we have successfully built PETSC and PFLOTRAN-OGS. Some features, like the use of MPI for parallellisation, require us to wrap PFLOTRAN-OGS and OpenMPI from PETSC in a manner that is reasonably ergonomic for users. When running PFLOTRAN-OGS directly it only uses a single core, but we wish to maximise the usage of resources. To this end, we have a "runner" script, that handles this.

In addition to the two builder scripts described previously, we have a build_env that links to various parts of the built PETSC and PFLOTRAN-OGS programs, as well as copies the latest version of said "runner" script.

The final install path becomes something similar to /prog/pflotran/versions/1.9.0-1+a1b2c3d4. This name consists of multiple parts:

• 1.9: The major version of PFLOTRAN-OGS
• .0: The "service version" of PFLOTRAN-OGS, currently there is none, which we denote with 0.
• -1: The build number. If we change the build process without changing the PFLOTRAN-OGS source, we increment this number.
• +a1b2c3d4: A shortened "git commit hash". A hexadecimal number that refers to the exact last change in PFLOTRAN-OGS' code repository that we built the project from. It allows us or OpenGoSim determine more exactly the version that is used, in case of bugs.

To make it simpler, we also make "symbolic" links, from latest to 1.9 to 1.9.0 to 1.9.0-1 then finally to 1.9.0-1+a1b2c3d4. This means that /prog/pflotran/versions/latest will point to the same place as /prog/pflotran/versions/1.9.0 (for the time being). Normally, we'll expect users to use latest, but if the specific version of 1.9.0 is warranted, it is also available.

Finally, there is a top-level /prog/pflotran/bin/runpflotran script, which checks the environment variable PFLOTRAN_OGS_VERSION, and forwards whatever arguments the user gave to that "deployment". If PFLOTRAN_OGS_VERSION is not defined, it defaults to latest. This is intended to be a more user friendly way of running PFLOTRAN-OGS without using long paths.

Synchronising

Many offices at Equinor have their own data centres which each their own network file shares. Users' personal files in their home directories, project data, and even programs installed to network file shares all exist independently in each location. This means that we have to manually synchronise the build artifacts.

To propagate changes, we have a deploy script. For each location that we are interested in, we connect to its HPC cluster top node (it is always online and nobody uses it directly to crunch data) and copy the relevant directories using the rsync command. Rsync ensures that files in the source location are identical to the destination location, and skips uploading/downloading if it's not necessary. The downside is that it is so quick we don't have time to get another cup of coffee.

Conclusion

This scheme hopefully serves to deploy PFLOTRAN-OGS in a quick, painless and error-resistant way. It makes it possible to use the program without fear that we may break something along the way, and with the possibility of rolling back to a prior version.

Future work

Uniformity of versions during Ert runs

Ert is a program developed by a team in Equinor to run simulations on probabilistic models. Simulators like PFLOTRAN-OGS depend on exact input numbers, but our models of reality are never so certain. Ert takes models and turns them into, say, 100 possible versions of reality. These "realisations" are exact, and are each analysed with PFLOTRAN-OGS, before Ert combines these overly optimistic results back into numbers with realistic uncertainties. Such runs can take many days to complete.

Now here comes the problem: Suppose that a user starts Ert on Monday and is able to complete 10 "realisations" a day using PFLOTRAN-OGS. On Tuesday we deploy a new version of PFLOTRAN-OGS. We have the situation where the first 10 realisations used one version of PFLOTRAN-OGS, but the next 10 used a different version, potentially completely destroying the results.

To avoid this scenario we can record the exact version of PFLOTRAN-OGS used when the user click "Start" and use that throughout. When we inevitably deploy in the middle of someone's run, the user's will use the older version making the whole run consistent.

Using an off-the-shelf package manager

Komodo is an entirely Equinor creation and is maintained by exceedingly few people. Our scripts are simple, but may pose challenges if scaled up. I want us to investigate whether we can use Nix, Spack or perhaps some other system to manage these environments while satisfying the commandments. Both have users and traction that far outweigh what we in Equinor can accomplish. But, it seems that either choice would need modifications. My hope is that these modifications are simpler than each program deployed in Equinor requiring its own flavour of build and deploy scripts.