Prologue

Since secrets and configuration items are essentially the same from the application standpoint (just some environment variables or files that need to be consumed during startup and, optionally, in the run-time), quite often people use the exact same approach to handle both. In this article I’ll share some thoughts on why I consider this to be a bad practice and suggest an opinionated view on doing it the right way.

Typical cases

The two mistakes in configuration management that I usually see in the wild are:

  • treating secrets as non-secrets;
  • treating non-secrets as secrets.

The typical example of the former would be storing an API token in plain-text form inside a git repository. I won’t be elaborating much on why this is bad because it is obvious: a secret that everyone can see is not really a secret. Although sometimes it really might be the best approach (say, for a junk throwaway secret that you only use in some tests in some development environment), most of the times it is not.

The latter mistake is much more subtle. It usually happens in the companies that have already thought of storing the secrets securely and deployed something like Hashicorp Vault, along with some automation to inject these secrets into the application. People then give in to temptation and decide to store other configs (like endpoints for external services, log verbosity handles, etc.) in the same system. At first this looks somewhat decent, but as the codebase and the infrastructure grow, the problems arise:

  • there is no easy way to reuse the configurations (like building a hierarchical configuration system where each environment shares some common configuration and has a relatively small override) — resulting in either tedious copy-and-paste or in a need to build external automation;
  • there is no easy way to search for values and replace them in bulk — if you need to update an endpoint across twenty microservices, then your options are the same as earlier: doing it manually or writing a lot of code to do it;
  • there is poor access control: a person needs access to secrets in order to check the configurations, which is no-go in production; an alternative to that is splitting the secrets into more paths, making it even more difficult to navigate the secret storage;
  • there are poor audit capabilities: you are really limited in options to determine who, when, and, most importantly, why changed the configuration.

The right way

What is the correct way to handle secrets and configurations then? Below I’ll list some approaches that worked best for me and some of the teams that I worked with:

  • Vault for secrets and secrets only. If you think that a string is safe to be read by anyone, don’t treat it like a secret and put it somewhere else.
  • Single instance of each secret. Never copy and paste secrets across your storage — have multiple apps access the single secret instead.
  • Limited secret access. With configurations separated from secrets, you really don’t have to grant most people permissions on viewing the secrets. They will be much safer this way.
  • Automated secret rotation. Invest heavily in updating the secrets automatically, since it is an activity that must be done regularly.
  • Configuration as text in a git repository. This is the most effective and straightforward way. If you need to update something, just do 
    find . -type f -not -path './.git/*' -exec sed -i "s/oldendpoint/newendpoint/g" {} \;
git add .
git commit -m 'JIRATASK-11: update endpoint for some service'
git push
    Note that apart from doing the job in one simple line of code, it also adds all the context that people will ever need when reviewing the change: your name, date of change, list of affected files and the reason for change (which should be somewhere in the linked Jira ticket).
  • Monorepo for configurations. When it comes to actual code, I really don’t like monorepos, but that’s a topic for another article. For storing configurations though, these work great, as simple commands from above can be used to make updates in multiple microservices at the same time.
  • Layered configurations. Have files like common.env or all.yaml with shared configurations and smaller files like prod.env, dev.yaml, stage.toml, etc., with only environment-specific values to reduce copy-and-paste. Two layers are usually enough, but feel free to add more if you feel like it’s necessary.

Epilogue

Of course there is no single way of managing configurations that magically fits everyone. Countless ways exist to approach this task, and you don’t necessarily use the tools that I listed or the methods that I described. What I encourage you to do instead is to just be aware of the problems described above when designing a system that would meet your own needs. And please, please, store secrets and configs separately.