Passwords, API keys, and secret strings. These secrets all need to be stored, safely shared, rotated and revoked. Although it’s something that should be ubiquitous in this day and age, high-profile cyber security incidents suggest that’s not the case at all.

Poor management of secrets is a real and present threat. The Uber breach in September 2022 started with a simple compromise of a contractor account using a combination of credentials from the dark web and push-bombing to defeat MFA protection. This allowed the hacker to access the corporate VPN. The breach severity suddenly increased when the attacker found admin credentials for a PAM solution hardcoded in a script. The keys to the kingdom were there for the taking.

Psst! I have a secret

Most people working in technology if asked, and prompted a little, would agree that hardcoding passwords is bad. So why does it still happen? Put simply, it works. If you want to get something up and running quickly it can be very tempting to take the easiest path. That is until someone finds it on a filesystem or a colleague commits it to a public repo by mistake. Then the headaches really begin with ensuring there has been no unauthorized access, or compromise. Suddenly any convenience gained at the outset is vastly outweighed. Even in the best-case scenario with well-intentioned users in a private workspace, there is no control or auditing on how the credentials are used once they are in the wild.

What next? Pull those pesky secrets out into environment variables! This avoids the problem of hardcoding, but doesn’t address concerns about exposure in terms of logs, or output. Nor does it help with controlling access or providing auditing and rotation. Similar problems exist when you extract secrets into separate configuration files. The concerns around exposure can certainly be mitigated somewhat by encrypting variables and file values initially and then decrypting at runtime but this is still lacking in terms of best practice.

Keep your secrets, secret

Managing secrets securely is key, pun intended, in maintaining a good security posture. What best practice looks like for you in terms of implementation may vary from someone else. However, good principles remain the same.

• Use an appropriate management system. Ditch the environment variables and embrace services designed for secure management.
• Encrypt, encrypt, encrypt. Your secrets management tooling should encrypt the data it holds, but also think about where that data is being used. Is your application accessing an API KEY securely, but then exposing it in logs? Are you accessing a secret in a Terraform deployment and ending up with it stored unexpectedly in state? Ensure your use of secrets means they remain encrypted in transit and at rest.
• Manage secret lifetimes. Use automatic rotation whenever possible to ensure it happens on a regular basis. Where it isn’t possible, have tightly monitored processes to ensure it happens on time, every time.
• Control access. Always use the rubric of least-privilege. Access only to those users and applications that need it.
• Track secrets access. Monitor the previous control. Is it only the principals that are expected who are reading our secrets?
• Backup secrets. If they’re important enough to secure, they’re important enough to back up. Make sure you have a restoration plan too.

With all that in mind, what tooling does AWS offer to manage your secrets?

AWS KMS

We’ve already noted you should really look to the benefits of a secrets management system of one form or another. But this is the real world and people don’t always do the right thing.

AWS KMS provides the platform to create and manage encryption keys that you can use in your own pipelines, processes and applications to secure your secrets. Of course, that glosses over a great deal of complexity and the overheads of building, maintaining and managing your system to do undifferentiated work.

You may have genuine reasons for choosing to roll-your-own management and leverage KMS directly as part of your secrets management story. But those requirements should be interrogated to ensure you’re not reinventing the wheel and genuinely have a differentiated use case.

AWS Systems Manager Parameter Store

The Systems Manager Parameter Store is a key/value store for text-based configuration items. Passwords, database connection strings, are all listed in the documentation as fair game for storage in the service. What you get is a scalable, hierarchical and versioned storage of your data.

Parameter Store isn’t, primarily, a secrets manager. Nor does it pretend to be. Missing is any kind of automated secret rotation, although you can trigger actions based on events in Parameter Store and assign Parameter Policies to specific items. This allows you to set expiration dates and notifications of lack of rotation. Similarly, KMS encryption is not enforced. When you configure your parameter you define at creation it is either a simple string, or SecureString. Permissions are somewhat granular, assuming you set up your hierarchy in a sensible manner. You can then specify user permissions policies that grant access at different levels of the hierarchy. Using this method you can effectively delegate out access and control of parameters.

Parameters come in two flavours, Standard or Advanced. The difference being the ability with Advanced to store values up to 8k in size (up from 4k Standard), add Parameter Policies, share with other accounts and store more than 10,000 parameters per account, per region. If you stick to the Standard Parameter type and don’t need High Throughput interactions you won’t be charged. Once you move to Advanced there are charges per parameter and API interaction.

In short if your needs are simple, and you can live with the shortcomings, SSM Parameter Store may be an option in your secrets management story.

AWS Secrets Manager

AWS Secrets Manager launched later than the Parameter Store and fills in some of the gaps SSM Parameter Store has as a secrets manager.

First, and most simply, Secrets Manager enforces encryption. Secrets Manager uses AWS KMS to encrypt every version of every secret with a unique data key. These keys never leave the KMS service unencrypted and allow custom settings on the KMS key for audit. Secondly, secret rotation is provided. This is in a managed flavour for limited services but via provided Lambda functions for many more. Secrets can also be individually permissioned using either identity or resource policies.

Cost-wise Secrets Manager is the more expensive option, but we’re still talking about cents. There is a charge for each secret stored and then for API calls to use it. Along with the expected features we have already rehearsed, there are the bonus features of random password generation and full cross-account access. Combine that with the out-of-the box integration with many AWS services and Secrets Manager begins to show its value.

I’ve got the key, I’ve got the secret

AWS provides some useful functionality for secrets management in Parameter Store and Secrets Manager. However both fall short both in terms of specific auditing, though CloudTrail will help you there, and backup. Neither service integrates with AWS Backup and so you are left in a position of needing to implement something on your own.

Other solutions can provide more complete experiences. If the gaps we’ve discussed are deal breakers due to strict compliance requirements or regulations, if you need to store certificates and other material or if you need full end-to-end PAM controls for user, machine and application secrets other solutions on the wider market will be a better fit. Hashicorp Vault, BeyondTrust, and CyberArk are examples that span the range from enterprise password manager through to fully featured access management platforms.

Here to help

Navigating what is a good fit for you and your business can be tricky. What is important is driving secrets management from your holistic requirements, rather than finding yourself in a situation of adopting something that will not meet your needs without extra work and process. This should be in the context of a wider security strategy and good practice. At the Scale Factory we’ve helped many companies with their security story. Whether it’s a specific challenge or a wider review and advice, we can help.

Designing effective systems security for your SaaS business can feel like a distraction from delivering customer value. Book a security review today.

---

This blog is written exclusively by The Scale Factory team. We do not accept external contributions.