Making Travis CI secrets work again

It was a peaceful and quiet late Friday afternoon when we decided to automate some of the manual tasks around releasing one of the Ruby gems we maintain. It will only take an hour, probably even less …

TL;DR

If you are using Travis CI GitHub App (building your project on travis-ci.com, not travis-ci.org) and experiencing strange issues with secrets not working, you probably encrypted those secrets with the wrong repository RSA public key. To create encrypted value for the travis-ci.com, you need to add --pro switch to travis cli command:

$ travis encrypt --pro --add deploy.api_key -r xlab-si/redfish_tools

Hope that this helps, and best of luck with your other bugs!

Now to the main story.

But Friday afternoon is beer time!

We are well aware of that, but it was still a bit early for beer and, you know, it will only take an hour ;) So we started configuring Travis CI to automatically release our gem when we tag a commit.

Things went great. We copied the configuration from one of our other gems that has this already set up, modified it a bit and went to activate the repository on Travis CI. And this is where we hit our first speed bump.

As some of you may already know, Travis CI is merging the open source and commercial platforms and moving to GitHub Apps. As suggested by the Travis CI documentation, we decided to activate our repository on travis-ci.com in order to avoid the migration process in the near future.

This added a few steps to our process, since we needed to grant access to our repositories to the Travis CI app, but nothing out of the ordinary. We were back on track in no time.

Last thing we did is replace the deploy.api_key in our .travis.yml file by running

$ travis encrypt --add deploy.api_key -r xlab-si/redfish_tools

After committing changes to the repo and tagging the release, we sat in our chairs and watched the build log. What a beauty! No errors and a final exit status 0. Beer, here we come!

Just a quick check on RubyGems before we leave the office …

The mysterious case of missing jewellery

As you already guessed, there was no new gem version published. Maybe we are not as awesome as the Travis CI build log suggested? We decided to take a closer look.

And there it was, on the line 554:

Access Denied. Please sign up for an account at https://rubygems.org

Why did Travis CI not fail then? A quick search revealed that this is the intended behavior, since detecting a failure for all supported deploy providers is far from trivial. Fair enough.

But why did the gem push fail? Is there something wrong with our RubyGems API key? In order to get the new gem published and to eliminate the possibility of a bad RubyGems API key being the root of our problem, we decided to manually create the new gem release using the same RubyGems API key. After checking out a fresh copy of repository and running rake release, the new gem appeared on the RubyGems site, so the API key was not the problem.

Did we make a mistake when encrypting the RubyGems API key? To test this out, we run the encrypt command again, carefully copying the API key when requested, committed the changes, tagged the commit and pushed everything to GitHub. And after another minute or so, we were again staring at the same error as before.

Time to switch our brain from beer to debug mode.

How Travis CI secrets work

Before we resume our debugging story, we will squeeze in a quick refresher on how Travis CI secrets work. If you know all about this, feel free to skip to the next section.

Now, each GitHub repository that is registered on Travis CI gets associated with a unique RSA key pair. Private part of that RSA key pair is only accessible to the CI servers while the public part can be downloaded by anyone.

When we want to store sensitive pieces of information in a public place, we can encrypt them using repository’s public key. Encrypted data can be safely stored in the repository, since only the CI server is able to decrypt it using repository’s private key.

Now back to the story.

The devil is in the detail

Since debugging the deployment process on CI server is next to impossible, we put our keyboards aside and started thinking out loud about the steps that involve using an API key. What we came up with is this short list:

1. travis program encrypts our RubyGems API key using repository’s public RSA key.
2. CI process on the server decrypts the data using repository’s private RSA key.
3. Ruby client for RubyGems.org uses a decrypted RubyGems API key to authenticate gem upload.

We all agreed that this is what happens, and then we started adding the details about each step, using official documentation as a reference. We ended up with this slightly extended list:

1. travis program encrypts our RubyGems API key using repository’s public RSA key from travis-ci.org.
2. CI process on the travis-ci.com server decrypts the data using repository’s private RSA key.
3. Ruby client for RubyGems.org uses a decrypted RubyGems API key to authenticate gem upload.

“Gosh darn it!” is not what could be heard at this moment, but we would like to keep this blog family friendly ;)

Now that we knew what is wrong, resolving the issue was almost trivial. Help for encrypt command revealed exactly what we needed:

$ travis encrypt --help
Encrypts values for the .travis.yml.
Usage: travis encrypt [ARGS..] [OPTIONS]
    ...
    -e, --api-endpoint URL
                 Travis API server to talk to
        --pro    short-cut for --api-endpoint 'https://api.travis-ci.com/'
        --org    short-cut for --api-endpoint 'https://api.travis-ci.org/'
    ...

After adding a --pro command line switch to the encrypt command and pushing changes to the GitHub, things started to work as expected.

Bug link or it never happened

To notify the Travis CI team about the outdated documentation, we created a new issue on GitHub where they host the documentation repository. And guess what we found while casually looking over the pull requests … a work-in-progress pull requests that addresses our exact problem! Oh well, just another day in programmers life ;)

Do you have anything to add? Tell us what we could have done better on twitter or Reddit.

Cheers!