Posted by thatha7777 8 hours ago
> ...
> `Message-ID` is one of the most basic required headers in email.
Section 3.6. of the RFC in question (https://www.rfc-editor.org/rfc/rfc5322.html) says:
+----------------+--------+------------+----------------------------+
| Field | Min | Max number | Notes |
| | number | | |
+----------------+--------+------------+----------------------------+
| | | | |
|/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
... bla bla bla ...
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/|
| message-id | 0* | 1 | SHOULD be present - see |
| | | | 3.6.4 |
|/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
... more bla bla ...
/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/|
| optional-field | 0 | unlimited | |
+----------------+--------+------------+----------------------------+
... every message SHOULD have a "Message-ID:" field.
IIRC this particular rule is a SHOULD because MUAs often send messages without a Message-ID to their submission server, and the submission server adds one if necessary. https://www.rfc-editor.org/rfc/rfc6409.html#section-8.3 The SHOULD lets those messages be valid. Low-entropy devices that can't generate a good random ID are rare these days, but old devices remain in service, so the workaround is IMO justified.
I once had a job where reading standards documents was my bread and butter.
SHOULD is not a requirement. It is a recommendation. For requirements they use SHALL.
My team was writing code that was safety related. Bad bugs could mean lives lost. We happily ignored a lot of SHOULDs and were open about it. We did it not because we had a good reason, but because it was convenient. We never justified it. Before our code could be released, everything was audited by a 3rd party auditor.
It's totally fine to ignore SHOULD.
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
for the client. If you're implementing a server, "the client SHOULD but didn't" isn't a valid excuse to reject a client either.
You can do it anyway, you might even have good reasons for it, but then you sure don't get to point at the RFC and call the client broken.
Yes it absolutely is: https://www.rfc-editor.org/rfc/rfc2119 is quite clear.
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
If the server has considered fully the implications of not having a Message-ID header, then it MAY continue processing.
In general, you will find most of the Internet specifications are labelled MUST if they are required for the protocol's own state-processing (i.e. as documented), while specifications are labelled SHOULD if they are required for application state-processing in some circumstances (i.e. other users of the protocol).
That is not a rule.
In this situation the server can reject any message if it wants to, and not doing a SHOULD tests the server's patience, but it's still ultimately in the "server wanted to" category, not the "RFC was violated" category.
The RFC is a request for comments. The specific one in question is about Internet Mail.
If server implementers want their mail to be delivered these are things they SHOULD do.
That's it.
It isn't something you can give to your lawyer, and nobody cares about your opinion about what you think "should" means you can make someone else do. This is how it is.
How does Google know whether or not the sender has a valid reason? They cannot know that so for them to reject an email for it means they would reject emails that have valid reasons as well.
You and I have different definitions of "clearly"
It is not required for the protocol of one SMTP client sending one message to one SMTP server, but it is required for many Internet Mail applications to function properly.
This one for example, is where if you want to send an email to some sites, you are going to need a Message-ID, so you SHOULD add one if you're the originating mail site.
> How does Google know whether or not the sender has a valid reason?
If the Sender has a valid reason, they would have responded to the RFC (Request For Comments) telling implementers what they SHOULD do, rather than do their own thing and hope for the best!
Google knows the meaning of the word SHOULD.
> it means they would reject emails that have valid reasons as well.
No shit! They reject spam for example. And there's more than a few RFC's about that. Here's one about spam that specifically talks about using Message-ID:
The server "considers" nothing. The considerations are for the human implementers to make when building their software. And they can never presume to know why the software on the other side is working a certain way. Only that the RFC didn't make something mandatory.
The rejection isn't to be compliant with the RFC, it's a choice made by the server implementers.
I don’t care what the protocol rfc says, the client arbitrarily rejecting an email from the server for some missing unimportant header (for deduction detection?) is silly.
Yes. https://www.rfc-editor.org/rfc/rfc2821#section-6.3 refers to servers that do this and says very clearly:
These changes MUST NOT be applied by an SMTP server that
provides an intermediate relay function.
> Stop hiding behind policy and think for yourself.
Sometimes you should think for yourself, but sometimes, and friend let me tell you this is one of those times, you should take some time to read all of the things that other people have thought about a subject, especially when that subject is as big and old as email.
There is no good reason viva couldn't make a Message-ID, but there's a good reason to believe they can't handle delivery status notifications, and if they can't do that, they are causing bigger problems than just this.
“MAY This word, or the adjective "OPTIONAL", mean that an item is truly optional… An implementation which does not include a particular option MUST be prepared to interoperate with another implementation which does include the option, though perhaps with reduced functionality. In the same vein an implementation which does include a particular option MUST be prepared to interoperate with another implementation which does not include the option (except, of course, for the feature the option provides.)”
Note how it explicitly calls out interoperation with implementations that do or do not implement MAY. As a exception that proves the rule, we can reasonably assume that not interoperating with a system ignoring a SHOULD rule is a correct implementation and it is the fault of whoever is not implementing SHOULD.
The standards, to my observation, tend to lag the CVEs.
Side-note: If someone has built a reverse-database that annotates RFCs with overriding CVEs that have invalidated or rendered harmful part of the spec, I'd love to put that in my toolbox. It'd be nice-to-have in the extreme if it hasn't been created yet.
CVE classify a lot of things that have nothing to do with security.
Not having a Message-ID can cause problems for loop-detection (especially on busy netnews and mailing lists), and with reliable delivery status notification.
Dealing with these things for clients who can't read the RFC wastes memory and time which can potentially deny legitimate users access to services
> It seems that Gmail is being pedantic for no reason
Now you know that feeling is just ignorance.
That should have already happened. Google is not the "first stop".
> hard ban the sender server version until they confirm
SMTP clients do not announce their version.
Also I don't work for you, stop telling me what to do.
> A midway point that involves a doom switch is not a good option.
No shit. That's almost certainly a big part of why Google blocks messages from being transited without a Message-ID.
Is it still a strong spam signal? Hard to say. Sources disagree. But as with laws, heuristics, once added, are often sticky.
"SHOULD" is basically, if you control both sides of conversation, you can decide if it's required looking at your requirements. If you are talking between systems where you don't control both sides of conversation, you should do all "SHOULD" requirements with fail back in cases where other side won't understand you. If for reason you don't do "SHOULD" requirement, reason should be a blog article that people understand.
For example, "SHOULD" requirement would be "all deployable artifacts SHOULD be packaged in OCI container". There are cases where "SHOULD" doesn't work but those are well documented.
I’m doing some work with an email company at the moment. The company has been in the email space for decades. Holy moly email is just full of stuff like this. There is an insane amount of institutional knowledge about how email actually works - not what the specs say but what email servers need to actually do to process real emails and deal with real email clients and servers.
The rfcs try to keep up, but they’re missing a lot of details and all important context for why recommendations are as they are, and what you actually need to do to actually process real email you see in the wild. (And talk to the long tail of email software).
This conversation makes me think about cornering some of the engineers with a microphone. It’d be great to talk through the specs with them, to capture some of that missing commentary.
Note "the full implications must be understood and carefully weighed before choosing a different course". Gmail and the other big hosters have full-time spam teams who spend a lot of time weighing implications, so I assume the implications of this was weighed.
Must = external requirement
I cannot fathom how you think should* would act as a requirement in any sense of the world.
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
I think it's a gray area
- If the receiver declines your message because "Message-id" is required - then I blame the receiver; because that's not true
- If the receiver declines your message because "most systems do include it, and it's lack of presence is highly correlated with spam email", then it's on the sender
Admittedly, the end result is the same.
Now, let's assume that if it is the latter (it's spam related), and Google were to accept the message, but then internally bin the message, it would be worse. At least in this case, they are bouncing the message. Because of this, the sender is at least aware that the message wasn't delivered.
Also, the author was able to get their mail delivered to a personal gmail.com address. The issue was with a Google Workspace custom email domain. This further makes me think of this as a security/spam related issue. Google is clearly capable of processing the message without a Message-id, they are just refusing for business customers.
My takeaway is that I think that Google is doing the least-wrong thing. And by being explicit in how they are handling it, it at least made the debugging for the author possible.
Also note: in a quick reading of RFC5321 (SMTP), rejecting messages for "policy reasons" is an acceptable outcome. I'm not sure if it applies completely here. The author should probably also be taking into account RFC5321 (SMTP) instead of just 5322 (message format).
That's the annoying part to me.
An email is an email. By applying different rules for rejection on different mailboxes, gmail creates a system where it's harder for would-be implementers to test compliance.
If tomorrow gmail creates a new type of mailbox, will there be a third set of rules to have your message delivered?
This here is a trivial case of simply not testing deliverability at all.
That is, when some hotline tell me that they just sent and email with the information, I ensure they hold the line until I got the actual email and checked it delivers the relevant information to fulfill the intended process. And when I want to make sure an email was received, I call the person and ask to check, waiting until confirmation.
It’s not that much SMTP/IMAP per se as the whole ecosystem. People can legitimately get fatigue of "is it in my junk directory", "it might be relayed but only after the overloaded spam/junk analyzer accept it", or whatever can go wrong in the MUA, MSA, MTA, MX, MDA chain. And of course people can simply lie, and pretend the email was sent/received when they couldn’t bother less with the actual delivery status.
There are of course many cases where emails is fantastic.
They once made all emails from my very reputable small German email provider (a company that has existed and provided email services long before Google existed) go into a black whole - not bounce them back or anything like that, mind you, their servers accepted them and made them disappear forever. I was in contact with the technicians then to get the problem fixed and they told me it's very difficult for them to even reach anyone at Google. It took them several days to get the problem fixed.
Of course, no one will ever be able to prove an intention behind these kind of "technical glitches." Nothing of significance ever happened when Google had large optics fiber connections with NSA installed illegally and claimed to have no knowledge of it, so certainly nothing will happen when small issues with interoperability occur and drive more people to Gmail.
For what it's worth: having seen some of how the sausage is made, Google isn't particularly interested in screwing over a small reputable German provider. But they also aren't particularly interested in supporting such a provider's desire to route messages to their users because the provider is small. At their scale, "I've forgotten how to count that low" is a real effect. And email, as a protocol, has been pretty busted for decades; it's not Google that made the protocol so open-ended and messy to implement in a way that providers will agree is correct.
> Nothing of significance ever happened when Google had large optics fiber connections with NSA installed illegally and claimed to have no knowledge of it
Nothing of significance outside Google. Inside, Google initiated a technical lift that turned their intranet into an untrusted-by-default ecosystem so that data was encrypted on the fiber (as well as between machines within a datacenter, to head off future compromised-employee attacks). That process took at least five years; I suppose there's a scenario where it was all smoke and mirrors, but being on the inside in the middle of the process, I watched several C-suite who are not particularly good actors be bloody pissed at the US government for putting itself into Google's "threat actor" box and making that much work for the system engineering teams.
Also, an engineer at Google then made an end-to-end email crypto plugin for Chrome, including a flag that was a nod-and-middle-finger to the information revealed in the Snowden documents. https://techcrunch.com/2014/06/04/nsa-mocking-easter-egg-fou...
After all, linguistics is full with examples of words that are spelled the same, but have different meaning in different cultures. I'm glad the RFC spelled it out it for everyone.
On the other end, we may receive messages with or without. Both are valid. We MUST therefore accept both variations.
The second one is a consequence of the former. So yes Google is the violating party.
When the docs disagree with the reality of threat-actor behavior, reality has to win because reality can't be fooled.
So, it's fairly explicit that the sender should use message-id unless there's a good reason to not do so. The spec is quiet about the recipients behavior (unless there's another spec that calls it out).
(No seriously, I’m asking; are there examples of where it’s actually different from a MUST)?
Also this reminds me of something I read somewhere a long time ago: when specifying requirements don’t bother with SHOULD. Either you want it or you don’t. Because if it’s not a requirement, some people won’t implement it.
I guess the one time it’s good is if you want an optional feature or are moving towards requiring it. In this case Google has decided it’s not an optional feature.
backward compatibility makes it hard to add MUST. using SHOULD is a good alternative
No it absolutely does not mean that. It means, by explicit definition which is right here, that text is exactly that definition, that no one requires it. They can't require it, and still be conforming to the spec or rfc. That's the entire point of that text is to define that and remove all ambiguity about it.
It's not required by anyone.
The reason it's there at all, and has "should" is that it's useful and helpful and good to include, all else being equal.
But by the very definition itself, no people require it. No people are allowed to require it.
Any that do, are simply violating the spec.
SHOULD means that if you don't that, bad things are likely to happen, but it will not immediately break at the protocol level and during discussion in the IETF some people thought there could be valid reasons to violate the SHOULD.
Typically, IETF standards track RFCs consider the immediate effects of the protocol but often do not consider operational reality very well.
Sometimes operational reality is that a MUST gets violated because the standard is just wrong. Sometimes a SHOULD becomes required, etc.
Certainly for email, there is a lot you have to do to get your email accepted that is not spelled out in the RFCs.
Should just means the thing is preferred. It's something that is good and useful and helpful to do.
That is not "must unless you can convince me that you should be excused".
For most cases you should use three points of contact. However, there may be other situations for example if someone is giving you a leg up, or you can pole vault, where another solution is preferred.
Anyway, in general you can expect that doing unusual but technically valid things with email headers will very often get your messages rejected or filtered as spam.
For consumers, ignoring a SHOULD mostly affects their own robustness.
But here Google seems to understand it as a MUST... maybe the scale of spam is enough to justify it. Users are stuck between two parties that expect the other to behave.
This is 100 percent the case, and why these things are this way.
If you wanted to make email two point oh, I dont think it would look a lot like what we have today.
But gmail accepts emails without message-id on personal mailboxes apparently.
Would this make mass emails and spam harder, absolutely. Would it be a huge burden for actual communications with people, not so much. From there actual white/black listing processes would work all that much better.
It would simply close the loop and push the burden of the messages onto the sender's system mostly.
And yes, you can decide from the envelope, and a higher chance of envelope validity.
> If you wanted to make email two point oh, I dont think it would look a lot like what we have today.
Rspamd and spamassassin have missing MID check in their default rules, I am sure that most antispam software is same.
So at that point the ID has no value to me except being obliged to carry it around with the message, so maybe the originating system can at some point make sense of it. But then there is obviously no reason to ever reject mail without it, it's an ID valid for the sender and the sender didn't care to include one, great, we save on storage.
Your framework of analysis is based on someone else's database key ids being irrelevant to you. That's true.
But another framework of analysis is tracking statistical correlations of what spam looks like. Lots of spam often don't have message ids. Therefore it's used as a heuristic in scoring it as potential spam. That's why other postmasters even without SpamAssassin independently arrive at the same answer of trying to block messages without a message id. Example: https://serverfault.com/questions/629923/blocking-messages-w...
An unrelated frustration of mine is that Message-ID really should not be overridden but SES for instance throws away your Message-ID and replaces it with another one :(
Sure, you can send email with whatever headers you want, use weird combos, IP addresses, reply-to, and it might be still a technically valid email, but not something that should land in people's inboxes.
Also, a payment processor not testing their email on the most popular email provider in the world is quite ridiculous.
- In most cases, you are expected to follow it.
- You can choose not to follow it, but you must have a very good reason.
For example, RFC 7231 say that there should be DATE header but some embedded devices have no real-time clock so it ok not to implement.
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
Having said that, I regret my original characterization of the Message-ID header as a "requirement" and have updated the blogpost to be fair to all sides.
Thank you for bringing this up.
Battle with spam has been for long part just trying to algorithmically fingerprint the scam bots and reject the message if it looks like it wasn't sent by "real" mail server/client.
So a lot of things that are optional like SPF/DKIM are basically "implement this else your mail have good chance of being put into spam automatically".
For example, why does Google handle this differently for consumer and enterprise accounts? Well it's Google so the answer could always just be "they are disorganized" but there's a good chance that in both cases, it was the pragmatic choice given the slightly different priorities of these types of customers.
Once you deviate a bit from the standard, you're down a slippery slope. Its not that difficult to use pragmatism to justify wrongdoing.
So these are mostly quality-of-life reasons, it’s not a reason to reject an email.
That is the difference.
Another example may be a lightweight implementation of a spec in a limited and/or narrow environment, which remains technically compliant with full implementations of a spec but interaction with such a limited/narrow environment comes with awareness about such limitations.
I've dealt with many worse cases than this, where the systems I was integrating with were doing things that weren't even close to reasonable, but they had the market power so I sucked it up and dealt with it for the sake of my users. Maybe Google's wrong here, but how do you not just implement the solution anyway?
But they just did (make it work). The logical assumption is that most ppl did the same, just used another email provider. Why would viva care? (same as google, why would google care?).
the email situation is the same pattern at scale. google workspace has the market power to enforce whatever interpretation they want, and the RFC debate is basically irrelevant from a business perspective. your users don't care that your reading of the spec is correct, they care that they didn't get the receipt.
the part about a payment processor not testing deliverability is wild though. that should be in the first week of any transactional email setup: send test emails to gmail, outlook, yahoo, protonmail, check headers, verify SPF/DKIM/DMARC, and actually monitor bounce rates. the fact that a major processor missed something this basic suggests the email infra was probably a "set it and forget it" setup from years ago that nobody ever revisited.
To unblock myself, I switched to a personal @gmail.com address for the account. Gmail's own receiving infrastructure is apparently more lenient with messages, or perhaps routes them differently. The verification email came through.
Chances are that the decision-makers in most companies don't care about the technicalities (i.e. which email you used for registration) - they want to get up and running.
The reason that Viva doesn't care, I assume, is the reason Google workspace doesn't care: they're both too big to care for 5% of their clients won't do the extra work. They know that their, usually much smaller clients, will "figure it out" by i.e. using another setup that works™. So why bother?
This is the major issue that most of the discussion is missing. It doesn't matter how you want to interpret the word SHOULD, if you want to send to google workspace, you MUST include a message-id. It's not like this is some fly-by-night server with 12 clients.
If you absolutely and completely don't want to include the message-id, then you need to have a warning that your service can't be used by Google Workspace customers. This used to be common practice, blocking communication to servers that behaved badly, and I sort of wish we'd bring it back.
We'd love to share this exciting announcement but you'll a different email app.
Although I guess the argument will be that email clients should use AI to summarise the HTML into a plain text summary.
Is there actual value to this? e.g. Is the output of Lynx's text dump better for plain-text email clients than whatever they'd display for html emails?
"Major European Payment Processor" really just translates to "Major European Incompetence Center".
However, I've also worked at a financial institution which used core systems by Harland Financial Systems. Their "encryption" for data in transit from teller workstations to the core system was just a two byte XOR, and they sent the key at the beginning of the connection!
Was so unbelievable to be able to crack this in under a half-hour after noticing patterns in a PCAP. Wouldn't have believed it if I hadn't seen it with my own eyes.
That fraud was good enough for our regulators and theirs, so I have no doubt the industry is filled with rotten incompetence through and through.
It's amazing that society even functions at all.
Do Europe financial institutions have the same level of corruption as the USA? Such as a credit card company authorizing credit card transactions with incorrect expiration date to maximum profit, Bank of America? Or opening new accounts without consumer consent, Wells Fargo?
Incompetence and corruption only slightly overlap in most cases, i.e., being competent at corruption is a very real thing. The incompetently corrupt, usually end up punished... and there are few and far between...as we all very well know.
The kind of schemes you mentioned are generally not going to be how "corruption" will manifest itself in European financial institutions, because although it is also difficult to speak in general across Europe since the EU has not yet subsumed democratic self-determination all across the continent yet, so there is wide variation; the competent corruption is largely in the form of money laundering and tax evasion, not lower level quantitative schemes that would quickly come to light because Europeans are also a lot more cognizant of money and value than Americans, so people are paying attention a lot more closely and will raise hell over a single cent, where Americans are known to have hundreds of dollars draining out of their pockets every month just alone on recurring payments for things they don't even use anymore and don't bother dealing with it.
What we all don't really seem to internalize as a human species, is the absolutely demonic type of pernicious nature of "banking", i.e., a kind of LotR, ring, that consumes you especially if you are weak... and human, or at least European civilization seems to frequently go through periods of immense weakness where things are going springily and everyone is dancing to the music the "bankers" are playing as they are pandering our pockets, and when they realize they could get away with that and all the pockets are plundered, they move on to plundering our homes, then our accounts, then they want to take our first born... "Banking" is like humanity's cocaine, the seemingly innocuous, feel good drug that will consume your soul if you do not rage and fight against that demon taking over aggressively. It's no coincidence that cocaine is so widely used by the most parasitic elements of European societies, especially in "banking"/finance in general.
I recently switched from Gmail to Fastmail and by and large I’m happy with it. But I’ve been surprised by the amount of spam and (particularly) phishing emails I get in a regular basis. Google might be too strict in its filtering but it does serve a legitimate purpose.
Fingers crossed that the experience will be the same for you.
I've been a happy customer otherwise for years, for what its worth.
I don't think either are. The payment processor should be sending it, but, at least according to the RFC, it is incorrect to reject an email that doesn't have it. I suspect the reason it is SHOULD, and not MUST is for backwards compatibility with software that predates the RFC that adds the message-id header.
Maybe there is a correlation between missing that header and being spam, but then it should go to the spam folder, not be outright rejected.
----------------------------
The experience with support is also similar to experiences I've had with support at many companies. I provide enough details that an engineer could probably easily fix the problem, but the support representative just dismisses it, and it is doubtful an engineer even hears about it.
To wit: basically everything in this world is a "SHOULD", at best. The rules are a conversation.
GMail, on the other hand, is damn close to 100%. And it does it through excruciating application of heuristics like "don't trust agents that don't set SHOULD headers".
Google is a big enough target to justify spending the resources on dedicated attacks against their infra. Other providers may simply get less spam because their email domain shows up less often in the sources attackers use to pick targets.
But hey, if you're in a business domain where categorically leaving 6 million potential clients-who-are-demonstrated-to-spend-on-things isn't an issue? One fewer thing to worry about, right? ;)
The second thing is, what email software are they using ? If it was any relatively used software I would not expect this problem to arise (maybe it is some commond software but misconfigured).
Third, while the header is not mandatory, I usually read SHOULD as a "if you don't implement it prepare for possible problems". SHOULD is not MAY.
Fourth, they should be thankful that Google bounced the messages with some appropriate error explaining how to solve it. I have plenty of issues in the past with both Google and Microsoft where they accept the message for then sending to /dev/null
Comments on issues use the format <[OrgName]/[RepoName]/issues/[IssueNumber]/[CommentID]@github.com>
A mitigation to this would be to take the combination of message ID and the sending domain and use that as the unique value, because message ID is not guaranteed to actually contain a domain label that's owned by the sender.
For example SendGrid's message IDs are <[RandomValue]@geopod-ismtpd-[Integer]>.
When orgs frame problems like this, it erodes trust in the message they try to convey. Email isn't a tough problem, but its a problem nobody wants to really deal with. Email is simple - its a text based protocol, that started out open, but now you need to add security to ensure your email is delivered.