Posted by rbanffy 7 hours ago
Did I get that right? Seems a bit anti-pythonic.
I'd be wary of trying to estimate the split of opinions based on who comments.
Here's some extra context [0]: https://github.com/pypi/warehouse/issues/20927. Loooots of links to click in.
the point here being small service providers are _also_ able to get into this system, but I think there's an expectation of success.
The bar for trusted publishers is very high though. I don't think it's high "on purpose" to leave out certain platforms for being too small, but in effect if (for example) your provider doesn't have a security contact point then that's going to rule them out here.
https://openid.net/foundation/sponsoring-members/
We know why PyPI ripped out PGP support and has refused any verification mechanisms that other distributions use for decades.
2. I'm not entirely sure what the link is supposed to prove -- would appreciate a word of clarification if you find the time. It almost feels like a dogwhistle of some kind, but A) I doubt anyone on HN is anywhere near that problematic, and B) there's no clear pattern in those companies other than "interested in ID tech", and none of the usual dogwhistle targets made the list. Also, y'know, OpenAI isn't on it...?
I guess my fundamental confusion isn't about the list and more about what "we know". Are you implying that they are against pure, unmanaged PGP for this system because... I mean, why? Because they're playing the long game to promote a verification/attestation/identification (?) technology that you think offers less stringent privacy guarantees, and thus to sell more ads in 2036?
If so, doesn't that seem like a kinda roundabout way to raise Display Ads revenue, otherwise known as "that one dying industry"?
Anyway, sorry to confuse/frustrate, I was being honest and tried to explicitly not accuse you of dogwhistling. I was making a best-effort attempt to explain my confusion, rather than just replying with "that doesn't make any sense" or something.
I guess I'll briefly elaborate just in case: "dogwhistle" above refers to a specific kind of antisemetic practice where people point at a list of companies or institutions or even just people and say something like "what do you notice?", whith the message ofc being that they're all related-to/run-by Jewish people. Again, not accusing you of that -- those people don't generally come here. But absent any other logic, I am left with confusion.
I will briefly add that this very response also doesn't help, when you very clearly imply that I am "shooting down a comment" in bad faith for some sinister purpose. It, again, comes off as conspiratorial. Why would I --a sad 30 y/o child with my real name and phone number in my bio-- be shooting this down?
I guess my overall response is that this feels like a conspiracy theory without identifying the conspiracy.
As a passionate anarchosyndicalist, I, for one, am thrilled about it. Decentralization is a tool in our toolbox for identifying and dismantling unjust hierarchies, but it is not a panacea, and can be quite damaging when taken to the extreme. See: cryptocurrency speedrunning 200y of bank regulations in 20y.
To put my trust for Python in more concrete terms: they support GitLab. As a passionate GitLab shill, I really couldn't be more impressed! It continues to shock me how infrequently the second most popular forge is acknowledged, much less supported. This shows they're not just, IDK, bowing to Microsoft's demands.
Not only don't we, but I must emphasize that Microsoft did not pay for anything involving the implementation of either Trusted Publishing or attestations on PyPI. I wrote a personal blog post explaining some of my own rationale for why we prioritized GitHub and GitLab for PyPI's initial support targets, and it shouldn't be surprising to anybody who knows me that the reasons are pragmatic, not dogmatic[1].
[1]: https://blog.yossarian.net/2024/11/18/Security-means-securin...
If not--i.e., if only recognized "forges" are ever going to be supported--why not?
Absolutely not claiming its not a pragmatic step, but equal-footing alternatives would be useful.
I'm no tin-foil-hat - but when a website account requires government id (like in the uk), and creating software requires signing by the website, you link all the things - is that a good outcome?
dlt (no provenance): https://pypi.org/project/dlt/#files
datasette (yes provenance): https://pypi.org/project/datasette/#files
[1]: https://github.com/pypi/warehouse/issues/new?template=bug-re...
I'm bought in on keyless signing, and I want there to be many more providers, so I like to see more adoption which builds more momentum.
Codeberg: https://codeberg.org/mfenniak/oidc-jwt Tangled: https://tangled.org/docs/spindles/workflows/#audience
I think OIDC tokens from workflows is basically table stakes at this point, since the alternative is a static secret, which is basically guaranteed to leak eventually.
All that's left is for PyPI to accept these tokens in their set of trusted publishers.
(I haven't looked at Tangled at all. If Tangled has a secret management story for its PKI that's stronger than a private key on disk and can demonstrate that a non-trivial number of PyPI users would benefit from federation, then it's a strong candidate.)
PKI definitely is still hard, but if it can be done well by a smallish number of trusted parties to the benefit of their users, this can have huge ecosystem benefits. Let's hope we live in that timeline.
Like, if I want to host on Codeberg, or Sourcehut, what is the process needed for those platforms to become a trusted publisher?
As it happens, I build my releases on my own hardware using a pypa/manylinux release. Can I provide attestation, or am I required to use a third-party service to get that "x Without provenance" removed from my entry?
For regular developers, provenance makes no difference; whether the package was made in a box controlled by Microsoft with a proprietary stack or on a box controlled by the developer make no difference.
(If you pan through that thread, you'll note that one of open issues with Codeberg is that their underlying server wants to store OIDC private keys on disk at the moment. This is okay for small deployments of personal services, but not for something that PyPI needs to federate with and trust.)
Will PyPI be presenting warnings or statements of caution which might cause users to be concerned about using my distributions? Or imply that my build practices are somehow inferior to depending on a third-party build system?
Do you see any other alternative?
Both of these are latent problems when trying to use PGP in packaging ecosystems; the subset of ecosystems that successfully adopt PGP tend to have very different topologies (namely, small rings of trusted...publishers) rather than an "anyone can publish" architecture like PyPI has.
But to take a step back: if you want attested builds because you think attestation is itself a gold star, then you're working from the wrong priors. Attestations do not imply that a package is secure any more than a hash does; what they are is a cryptographic signature over a set of claims, which allows for authenticated auditing of a project's state over time.
I think PyPI's documentation does a good job of conveying this nuance[2], and that much of the opprobrium of attestations comes from people either not understanding that nuance or imputing conspiracies into it.
[1]: https://github.com/pypi/warehouse/issues/18007
[2]: https://docs.pypi.org/attestations/security-model/#trustwort...
Which is great--but does not in any way require that the attestation can only be produced by working through one of a small set of recognized "forges". You could do this just as easily by, for example, letting me upload to my PyPI account a public key (the same as forges like Github do to allow SSH access), and then letting me sign builds that I upload using the corresponding private key. But that kind of workflow is not being supported. Why not?
That's true, there's some context elsewhere in these threads about how PyPI can enable email attestations. The short version is that PyPI wanted to make this work by default, which was trivial with Trusted Publishing (since it already provides a strong source of verifiable machine identity).
"Why not a public key?" is a good question, and the very short answer is that greenfield signing designs should not assume that all users are able to manage the secrecy of long-lived key materials, much less force them to make parameter decisions when (re)generating materials. This is the same mistake that the Web PKI made for decades, and has spent the last 15 years slowly untangling. The ideal setup is one where users demonstrate control over an identity, and then the signing scheme binds that identity to a short-lived signing key. That's how email attestations work.
From what I can gather from the github issue that was linked to, those would still require the email identity to be vouched for by a provider. (Though if whoever already is hosting my email could be a "provider", that would at least greatly expand the list of providers from the current small number of forges.) There doesn't seem to be any way for me to, for example, link a public key I generate myself to an email address that PyPI already has on file for me.
I believe your prior is that you expect people who read the PyPI page to understand what attestation means. I believe your prior is they will see a notice about "not from a trusted publisher" and not treat it like demerit or warning. I believe your prior is that seeing this message will get people to do more due diligence than they currently do, and that the due diligence will be relevant and appropriate.
Based on my experience with my users, I don't have the same confidence.
I don't understand what "authenticated auditing" means, so why should they?
What project state do people think is important to audit? My code repo is private, hosted on Sourcehut. My build system is private, done locally. The only thing I release is the pre-built binary package. There's very little state to examine, and I can't figure out why anyone cares.
I also have customers who buy a source code license. They have never asked for an audit of the project's state over time.
I read the "Security model and considerations" document. I found it opaque, no doubt because the entire topic is foreign to me, but it doesn't do a good job of explaining. Take the second paragraph:
"The most basic type of attestation (the type enabled by default in pypa/gh-action-pypi-publish) asserts that the project was published by an authorized publisher such as a particular CI provider in a particular code repository.".
I know pypa/gh-action-pypi-publish is some sort of GitHub thing. I don't use GH or third-party CI so that clarification does not help.
It doesn't describe what "authorized publisher" means, other than a generic example of "particular CI provider in a particular code repository". That phrase exists only once in the document. Is it different from a "Trusted Publisher"?
Why are the terms "authorized" and "Trusted" used, instead of something like "affiliated"? If someone is not a trusted publisher or an authorized publisher it's is too easily misinterpreted as "untrusted publisher" or "unauthorized publisher", which seems scary. If someone is not an affiliated publisher, "unaffiliated publisher" sounds less scary. (I'm not saying "affiliated" is the right word, only an example of a roughly appropriate word where the negation of it is not so scary.)
Then later, "It protects against modification of a project after it was built".
I think that's supposed to be "modification of a release after it was built"?
My use of "project" matches the one at https://docs.pypi.org/using-pypi/pypi-basics/ "A project is a collection of releases and files, and information about them. ... A "release" on PyPI is a specific version of a project." I want to modify the project with updated metadata as things change, and to make new releases.
Oh, and I can't find a description of what "build" means in that document. It's used as if everyone expects to know what it means, but a few years ago I learned that modern Python packaging uses "build" different than I do.
When I compile my Python/C extension I "build" it. When I make a wheel I "package" it.
I believe that modern packaging calls the whole thing "build"? Or just the latter step?
Incidentally, "python -m build --sdist" on my project does not work as a replacement for "python setup.py sdist" because my setup.py includes a code generation step to get from a clean repo to the Python/C extension files included in the distribution. I haven't figured out the modern way to handle that. I think it's because of the differences in philosophy of what "build" means, and what actions are allowed during different stages of the distribution process, which is why I think "build" needs a specific definition, just like "project" and "release" have specific definitions.
For context: attestations aren't something you even really enable. They just happen, by default, if you use Trusted Publishing from a forge like GitHub or GitLab (and hopefully, more in the future).
FWIW, your feedback on the docs is very helpful. It's genuinely useful to hear that the phrasing in it is imprecise, and I'll try and improve some of it this week. The main confusion that I think you're having (which is IMO reasonable) is that the doc was written to be theoretically splittable from the "Trusted Publishing" concept, i.e. so that "publishers" (principals who can publish) who aren't tied to a Trusted Publisher can attest to their published files.
Does that mean I, as someone who has projects on PyPI, shouldn't care if I am unable to provide any attestations because I don't want to have to go through one of the recognized forges?
> Authenticity and minimally-scoped credentials should be ambient properties of a system
But if the only way to get them is to go through one of the recognized forges, then they're not "ambient properties" of just PyPI alone, because there's no way for me to use them just with PyPI alone.
Of course, in real life 99.9% of people aren't looking at the page at all, and either following a documentation step which says "install by doing XYZ" or transitively from some other project they install.
I'll compare "attestations aren't something you even really enable" with this from the email attestation proposal: "Another possibility here would be to have it be opt-in: maintainers can mark their already-verified emails as being eligible for attestations on PyPI, e.g. via a checkbox in the user settings." ;)
I also can't help but wonder who reads https://docs.pypi.org/publishing/publishing-to-pypi/ where it says "For instructions on how to package your Python project, please refer to the Python Packaging User Guide.", follows those steps including the one about using twine and TestPyPI, does all that, and then gets to "Publishing package distribution releases using GitHub Actions CI/CD workflows" and gives up, since twine works.
Or goes back to the pypi documentation and sees "To use an API token: Set your username to __token__" when I can't find anything like that at https://packaging.python.org/en/latest/guides/publishing-pac... , Those seem like describing two very different systems, speaking as someone with no experience with them at all, who learned to use twine because of the Python Packaging User Guide.
(I also don't hand out gold stars, and my employer does not direct my work on PyPI.)
[0] https://docs.pypi.org/trusted-publishers/
[1] https://tailscale.com/docs/integrations/identity/custom-oidc
This is the right idea, but it's actually not that simple unfortunately: Trusted Publishing relies on per-IdP claims that are more precise than the `sub` claim (because doing pattern matching on the `sub` claim is brittle and prone to security problems). So we require publisher providers (OIDC IdPs) to provide distinct claims that we can match on. For GitHub and GitLab that's things like the repository name, workflow filename, etc., while for others it might be distinct username and immutable ID claims, etc.
This is all downstream of OIDC-driven federation being a much weaker and more bespoke process than all parties would like it to be. I think the discourse around this could be much improved by people taking the time to understand these complexities and why they pose a challenge to groups with limited engineering resources (like PyPI).
It’s quite sad to see that one of the main (or the biggest?) documentation system for the Python completely alienated its user base and led to this fork.
Edit: I should qualify: Homebrew’s own processes verify its attestations. But that’s not enabled by default on end user machines because it slows installations down and currently requires the `gh` CLI. Neither of those is a tenable long term constraint for a default feature.
if you have any of your files published by a trusted publisher with attestations, then you must have all of your files published that way (in that package).
otherwise pypi determines your project may have been compromised and puts some big red warnings up like [1]. it does not try to look at "this particular source file was never part of trusted publishing" or anything like that. and, CRITICALLY, it does not send you an email or anything like that about the non-acceptable file that you just uploaded, which it REALLY should. I mean the model is, they think these uploads are malicious by an attacker. Emailing the project maintainer(s) would be pretty important! (When pypi wanted to get rid of GPG signatures, they started sending me an email for every single signature that got uploaded. so i know you guys know how to send emails).
it's also AFAICT a permanent switch. once you put a single trusted/attested file up there, your account must now do all files in this way forever (or risk big red warnings. which maybe you're OK with that).
It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this).
It's not[1].
> It would all be easier to take if trusted publishers weren't limited to solely large for-profit corporations (I'm too lazy to research if they're trying to fix this).
As mentioned in several other places, we're always interested in adding more publisher providers, provided they meet our key management and notability requirements.
(This is also the easiest way to reason abductively about PyPI's goals. If PyPI's goal was to support "large for-profit corporations," we'd have dozens of publisher providers of dubious quality and community value, rather than the small subset we currently have.)
[1]: https://discuss.python.org/t/pypis-new-security-indicators-a...
I never claimed pypi is trying to "support large for-profit corporations". I fully understand the "large corporations as trusted publisher" thing is because this particular subset, e.g. github, gitlab, google, are widely known institutions with "good" reputations (in this area). The issue is that pypi is not, as of yet, supporting individual projects that want trusted publishing without relying on a large for-profit corporation. I'm not too put off by it but it is definitely a thing that a private for-profit host overnight becomes horrible (see e.g. bitbucket) and I have to get off that company ASAP. Having some non-profit entities here would be extremely helpful.
I fully agree. I am hopeful that Forgejo or whatever other underlying component of Codeberg can improve its key management primitives here, which would make it very easy to argue for their inclusion as a federated party.