Automating Octopus projects with Octopus Project Builder

Since last few months we have started redefining our CI/CD pipelines to use Octopus Deploy for deployment. Octopus is a great tool to define and manage deployment environment and deployments. It allows to nicely separate environment details (like number of boxes, box names), the environment related settings (like URLs, Connection Strings) and the deployment process (steps that have to be performed) from project executables. Moreover, Octopus offers out of the box all the tools needed to propagate packages to all target boxes, install them as windows services or IIS applications – it is a great benefit, because before we had to develop and maintain a quite complicated scripts for doing the same.

Over this few months of work we have found however one deficiency in this tool. All of the project, process, variable or environment configuration has to be done through UI, and as in every UI some of the operations are not easy to be done. Scenarios like moving variables from the project to the variable set, duplicating steps within the process or applying same changes on multiple processes are time consuming and a bit irritating over time.

Jenkins Job Builder

We have had a similar issue with other tool in the past: Jenkins, but we have found a great solution for it: Jenkins Job Builder. The JJB allows to define Jenkins Job in human friendly YAML format, and the beauty of it is that:

  • it is a text, so all the operations like moving variables to a different scope, changing definitions, renaming etc are as simple as text copy-paste/replacement operations,
  • it can be put into a source control system, which allows to see the change history and gives an easy way of restoring previous versions,
  • it can be easily applied to other Jenkins instances (which is very handy in case of migrations and box rebuilding).

Octopus Project Builder

Inspired by Jenkins Job Builder, I decided to spend some time on creating similar a tool for Octopus, the Octopus Project Builder, hosted on GitHub:

It is a very early stage of the project, but I managed to explore the Octopus API a bit with Octopus.Client and Yaml serialization with YamlSerializer.

So, how does it look like?

I have a Project Group with test project:

Project Group

After I run the OPB download command:

OctopusProjectBuilder.exe -a download -d c:\temp\octo -u http://localhost:9020/api -k API-XXXXX

I got the file ProjectGroup_My group.yml with content:

%YAML 1.2
  - Name: My group
    Description: This is the description.

…so the OPB managed to generate YAML for my project group.

Then, I edited the file with this content:

%YAML 1.2
  - Name: My new group
    RenamedFrom: My group
    Description: This is the description. It is generated now.
  - Name: My other group
    Description: This is the description. It is generated now.

and run the upload command:

OctopusProjectBuilder.exe -a upload -d c:\temp\octo -u http://localhost:9020/api -k API-XXXXX

Finally I got my project group renamed and a new group created:

Updated Project Groups

Managing more data

Now there is a time for more complicated stuff, the projects themselves. This is the current work in progress. So far, I noticed that Octopus stores the step action definitions in a bit different key-value format. Here is a sample of how the project may look like:

%YAML 1.2
  - Name: test
    Description: ""
        - RequiresPackagesToBeAcquired: False
            - Value: dev,wip
              Key: Octopus.Action.TargetRoles
          Condition: Success
          Name: Deploy
            - Properties: 
                - Value: Octopus.Features.CustomDirectory,Octopus.Features.ConfigurationVariables,Octopus.Features.ConfigurationTransforms
                  Key: Octopus.Action.EnabledFeatures
                - Value: "True"
                  Key: Octopus.Action.Package.AutomaticallyRunConfigurationTransformationFiles
                - Value: "True"
                  Key: Octopus.Action.Package.AutomaticallyUpdateAppSettingsAndConnectionStrings
                - Value: "False"
                  Key: Octopus.Action.Package.DownloadOnTentacle
                - Value: feeds-nuget
                  Key: Octopus.Action.Package.NuGetFeedId
                - Value: OpenCover
                  Key: Octopus.Action.Package.NuGetPackageId
                - Value: c:\temp\OpenCover
                  Key: Octopus.Action.Package.CustomInstallationDirectory
                - Value: "True"
                  Key: Octopus.Action.Package.CustomInstallationDirectoryShouldBePurgedBeforeDeployment
              ActionType: Octopus.TentaclePackage
              Name: Deploy
          StartTrigger: StartAfterPrevious

Future plans

Playing with Octopus and YAML is an interesting experience and I would like to explore it a bit more.
So far I have a few thoughts what I would like to implement here.

First of all, none of the samples have any IDs put in YAML. I want to build all the correlations basing on the human friendly names. Above I presented a scenario where Project Group has been renamed. Basically it would be possible to specify that the current name is Y, while the previous was X. When OPB will be uploading definitions to Octopus, it will first look for name Y and then X in order to rename it to Y if it is not renamed yet.

The Actions section look a bit complicated here, with a long named keys that are not really user friendly, like Octopus.Action.Package.AutomaticallyUpdateAppSettingsAndConnectionStrings. I worry that they are also not that well documented, so it may be a bit difficult to find them. To overcome this problem I would like to implement some macro/templating mechanism, a bit similar to JJB macros, that would allow to define a template of an action and then easily apply it in various projects.

The next thing, is that OPB will support multiple input files, so it would be possible to split definitions of projects, variable sets etc. On download, it will also put all the definitions separately.

An another thing would be the sensitive data representation. I would like to implement a feature allowing to keep the sensitive data in YAML encrypted where OPB will decrypt it before uploading to Octopus.

Finally, I would plan to support following configuration in OBP:

  • Project Groups,
  • Projects (with process and variables),
  • Variable Set Libraries.

More updates will be posted soon…