Posted by RGBCube 9 hours ago
Some aspects of the described behavior are as we intended and some are not. The cause is not exactly as described in the blog post. As for mitigation, we are already testing a patch of the unintended behavior on a subset of our infrastructure. If any of you try to reproduce the blog post's findings you may get confusing results throughout the day.
We will also re-evaluate whether the intended behaviors are acceptable or not. Some of this is a trade-off between multiple aspects of privacy, and multiple aspects of user experience.
Please note that this is my current understanding, which may change. I was only made aware of this an hour ago, and most of that time was spent talking with Ops, considering what to do immediately, and writing this post.
Finally, for those of you who do security research: when you find a security or privacy issue, please consider notifying the maintainer/vendor before publishing your findings, even if you intend to publish right away.
It's also worth stating that the client (including the cli client -- which, with a bit of work, you can get running in most situations where you'd use native wireguard) by default has a key rotation interval of I think 72 hours.
`mullvad tunnel get` will show it and `mullvad tunnel set rotation-interval <hours>` will change it. This is the preferred mitigation method of the post.
I personally don't mind having a pseudo-static IP (some other suppliers offer a static IPv4 as a feature!) as I wish to prevent network-level snooping from my ISP and governments. It's also worth stating that I think having a smaller IP space is an advantage for a privacy VPN: there are more potential users acting behind any given externally visible IP. Combined with technologies like DAITA (which effectively adds chaff to the tunnel) and multi-hop entrances and I personally think that this service really does plausibly make harder the life of those who snoop netflows all day.
Sorta odd you don't support one of Europe's most popular distros.
>The Mullvad VPN app is available in our repository for the following supported Linux distributions:
Ubuntu (24.04+) Debian (12+) Fedora (42+)
The only thing I see on the issue you linked is a way to jerry-rig the fedora package. When I tried that I kept getting untrusted key warnings. You can skip them of course, but it kind of undermines any type of trust here
How to report a bug or vulnerability
... we (currently) have no bug bounty program ... send an email to support@mullvadvpn.net
It should always be assumed that someone else (if not several someone elses) have already discovered the same flaw and are currently taking advantage of it while users remain totally unaware of their actual risk. By going public immediately, you give as many of those users as possible a chance to protect themselves.
Waiting to disclose something harmful when the users in danger could otherwise take steps to make themselves safe would be like not warning people entering a building not to go in because of a gas leak until after you've contacted the building owner and the fire department has shown up.
That's not what they said though. They said "please consider notifying the maintainer/vendor before publishing your findings, even if you intend to publish right away" (emphasis mine)
The flipside of course is ... does your disclosure increase the risk?
> aiting to disclose something harmful when the users in danger could otherwise take steps to make themselves safe would be like not warning people entering a building not to go in because of a gas leak until after you've contacted the building owner and the fire department has shown up
I don't think it's like this at all. The risk of a gas leak is not increased by telling people about it and can't be prevented after its occurred. To stretch your analogy, I'd say its more like you've found the gas leak and instead of turning off the gas supply are instead running around outside the building shouting about how there's a gas leak.
When you've got that much on the line you have to assume that the risk is already present for all users. It's true that there's always a chance that some users won't find your disclosure in time and additional would-be attackers who weren't aware of it already will start taking advantage of the flaw, but the alternative is that no users are safe.
> The risk of a gas leak is not increased by telling people about it and can't be prevented after its occurred.
It's true that warning people not to enter wouldn't make the gas more dangerous, but it limits the death count of the impending explosion. It keeps at least some people from entering the building and walking into a death trap.
There's no way to shut off the gas supply when you can't control what's already running on user's devices and more users are downloading and installing the buggy code all the time. It's really not a perfect analogy. The point is that immediate action will save some people, while waiting around means that nobody has a chance of being saved.
why are companies so entitled to get free security research/audits?
If so, I guess we just have different opinions on the ethics involved here.
As for our support team they are responsive and experienced. Several of them have worked with us for many years and do offensive security research in their free time.
Unlike many organisations we don't see customer support as a cost center, just like we don't see security as a cost center. Our support team represent our customers, and as a consequence contribute a lot to how we prioritise our roadmap.
I second this.
Clearly the person who wrote "Oof" has never emailed Mullvad support.
Whenever I have emailed Mullvad support I have received a prompt reply from a human being who clearly actually cares about taking ownership of the question and seeing it through to resolution.
I have also witnessed first-hand the support person taking the question to an internal team member where it requires additional input. So there are clear paths for escalation if circumstances require it.
Finally the support mail allows for PGP encryption of communications too.
(I am not a Mullvad shill. Not a Mullvad employee. Just a satisfied customer)
I'm not familiar with how you run your company -- without the context you gave most people would hesitate emailing support@ for security issues.
"Just email support@" feels like you don't care. That you do, and that your support team is awesome, doesn't change the fact that there are other companies out there who's aren't. Security people are human with human egos, and they want to feel special, so giving them a special way to reach you, even if it's the same thing behind the scene, makes a world of difference.
This sounds like how I'd design a VPN if I were an intelligence agency.
Make it look like an accidental misconfiguration and if an insider who isn't an NSA mole does somehow discover the logging, there's a fair chance they'll turn a blind eye anyway. After all, if you work at a VPN, publicly outing your employer for logging will tank the business, then you and your colleagues will all be out of a job.
I guess we’ll see how they respond.
Mullvad have been taken to court over this in relation to a copyright infringement case.
TL;DR The judge permitted people to take a fine-tooth comb to Mullvad's infrastructure and no logging was found[1].
[1] https://mullvad.net/en/blog/mullvad-vpn-was-subject-to-a-sea...
I recall a PRISM slide showing the diagram of Google and the public internet, with a big arrow on GFE saying, quote, “SSL added and removed here! :-)”
If NSA aren’t installed at Cloudflare, I wonder what they are even doing.
Hmm do we want them to decide what stuff is shady and what isn't?
We're already allowing payment processors to do that and it's not good.
That nonetheless doesn't help them unless they are doing active MITM. In order to do that they'd have to have at least some physical presence at Cloudflare or on the path to Cloudflare.
People didn’t care when they learned about PRISM, why would they care now when it’s a known fact? The sane stance would be to assume Cloudflare is in cahoots with NSA.
The NSA leaks dominated news cycles for the entirety of 2013.
This is as helpful as Whatsapp's so called E2E encryption comms (that just happens to not be applicable by default in certain situations).
it does give better peering. reduces latency a bit for me.
but you can also see from curl or traceroute, that the endpoint you talked to was a cloudflare ip and your ssl ended there. after that you can't see inside cloudflare.
I think more people than you would expect would be happy to accept that as the price for protection against malicious actors
That doesn't mean collusion
Either way, if they were directly colluding with Google, they would have had a much simpler time siphoning off that data.
The funny thing about that era is you knew they started using Cloudflare because they went from stable with constant uptime to going down and showing a Cloudflare banner randomly all the time for a good year or so. They ran worse with Cloudflare than they did while they were allegedly getting DDoSed. The whole company glows, as the late great HN commenter Terry Davis would've said.
Oh my god, this is how & when I realize that Terry Davis (Rest in peace) used to use Hackernews too: https://news.ycombinator.com/threads?id=TerryADavis
https://news.ycombinator.com/item?id=10061171 (From this comment written by terry):
"I wrote all the code from scratch, including a 20,000 line of code compiler that makes x86_64 machine code from HolyC or Asm and operates AOT and JIT.
My JIT mode is not interpreted. It optimizes and compiles to x86_64 machine code.
I was chosen by God because I am the best programmer on the planet and God boosted my IQ with divine intellect." -Terry A Davis.
Anyone with a few crypto currencies in their wallet that can click a button on any of the booter services with botnets for hire.
This is a massive issue in my view, it allows correlation across multiple VPNs exit nodes, but that’s it. It doesn’t allow to identify you automatically. It does significantly lower the bars for identifying you though, but the requirements are still high.
Hopefully they fix this soon.
I can’t believe this type of “let’s make it a hash or something sensitive” still happen, and at mullvad, of all places. Why not randomise it simply?
If you squint a bit, it looks a lot like a "Nobody But US" (NOBUS[1]) scheme. A few more identifying bits could tip the scale for party that has a whole host of other bits on a list of suspects, without being useful to most other people.
Their ads on San Francisco's public transit are good.
Security is always a balance. Always
AI is showing that everything has a weak spot (wondering where are the "I don't make mistakes with C" now people are - but that's for another discussion)
There's another commenter mentioning this makes sense because exactly it avoids them keeping information on which customer is matched to which server. You know, one of the things you don't want to log
Could it be done better? Probably.
Here's a better idea, logging off is 100% safe
Meanwhile 99% of the normies will go for NordVPN
Let me specify: The user must have entered his data on one site which the attacker has control of. That is a high bar still.
Sure, there are other intelligence agencies, but that's the one I'd be the most worried about. Since either they run it, or they would know of it and want to emulate the idea, or know of it and have access to it from the partner agency running it. Or they are not a threat to me.
There's also the issue of no publicly known cases where someone that used Mullvad being deanonymized through the VPN but instead being discovered through some other opsec failure. If an intelligence agency has this capability they have been sitting on it for almost 2 decades without making use of the data. Hard to believe.
Wow, I didn't realize Mullvad was this old! Then again, maybe they weren't popular enough back then for intelligence agencies to target them? For instance, Mullvad kinda rode WireGuard's popularity wave by being the first(?) VPN provider to implement the protocol. Big ads on billboards came even later. So maybe they only became a target in recent years?
So does your comment...
I think its safe to assume that intelligence agencies have other options available to them, such as country-wide timing attacks.
I don't know the answer, but there are two ways to take it:
1. Submarining to destroy confidence in an actually trustworthy, decent VPN company
2. They're an intelligence front.
For me, Mullvad have the appearance of the greatest likelihood of being legit since they're not aggressively pushing their product with lies and fear mongering. That gels with my vibe. If they're an intelligence front, well, most VPNs probably are as well, so I'm no worse off.
Luckily I'm not doing anything that would get me in the kind of trouble for which multi-jurisdictional cooperation is worthwhile.
I don't see how the author is arriving at this ">99% chance" purely from the numbers provided in the article. Assuming the first (banned IP) seed and the second seed are both in the range 0.4423 - 0.4358 (a stronger assumption than is justified by the example), all this tells us is that the first and second IP addresses both have seeds in a range that would contain 0.4423 - 0.4358 = 0.65% of all Mullvad users, which 0.0065 * 100,000 = 650 users. We've eliminated >99% of users as "suspects", but we haven't actually gotten >99% accuracy in identifying an individual across multiple exit IPs.
In more Bayesian thinking, the overlap in potential seeds is great evidence to think these IP addresses represent one and the same person (or Mullvad VPN account at least), but as far as I can tell, that's not what the author is saying.
What are the chances that someone uses this vpn, joins your forum the day after someone was banned, and has an ip in a similar range?
For most small websites this would be strong evidence.
Why can't it aim to solve what it can do? TOR is a great example: the TOR network itself can't perfectly anonymize you due to browser fingerprinting, but users of the TOR Browser get both the TOR network resisting deanonymization on a network level and a browser with plenty of anti-fingerprinting measures built in. A VPN could aim to prevent deanonymization on a network level so that users who want to stay anonymous can use the VPN in combination with fingerprinting-resistant software.
If I'm on a public VPN, I don't want anyone to know who is making the request, including the terminating IP.
Think about it. By your logic, VPNs shouldn't be used for torrents because VPNs shouldn't anonymize you to the terminating IP. Whereas they work gangbusters for that.
If you are talking about private VPNs.. Mullvad isn't one.
But today’s internet is essentially a giant ad network.
The fact that Tor does not intend to tackle the timing problem is plainly stated on the Tor website.
Edit: In hindsight I regret making this comment. It was unnecessary, but removing it now would look weird.
Putting aside the IP correlation across multiple servers, at first I wondered why even keep the user IP stable on one server. But I think it makes sense because as the author states other VPNs usually have only one IP per server so they are essentially simulating that. The advantages for the user are, if they find a server that works for accessing some service they can connect to that server again and it will work again because they get the same IP.
The IP correlation across multiple servers they should fix though with something like rand.seed(user_pub_key + server_id)
On the flip side, if they’re getting banned by a service because of a noisy neighbor on the same IP, they’d have no way to work around that, no?
All things considered, there are just an incredibly small number of IPs shared among all users, no matter the allocation strategy.
I'm a little confused on this... what is stopping third parties from doing key rotations like the main app clients if it is detailed in the repo how to do it?
Knowing to do so, primarily.
It does seem ridiculous once you spell it out like that, and then you have to realize that it’s plausible to de-anonymize even Tor users by controlling exit nodes.
Things you connect to or log in to are clearly going to be able to ID you at least with in the context of the login that you use regardless of what the VPN does.
I'm logged into HN through Mullvad as it happens. I usually leave it on regardless of what I'm doing because what I'm doing isn't my ISP's business even though I'm pretty happy with them.
Most likely these people just look to hide their torrenting, saying political shit on Twitter from employer and not share their choice of porn with local ISP. Also just adding one more layer between them and occasional scammer who can sometimes infer more broad geodata from their IP leaked from yet another database. Oh and now to avoid "Show your ID" page on the same porn sites.
It works well enough for this goal. Not everyone needs NSA-proof solution.
PS: Obviously more tech savvy people understand importance of hiding traffic on public WiFi, but I doubt average Joe the VPN user will buy VPN for this.
What percent on people on Hacker News who say they care about privacy live without Google, Apple, Microsoft and Facebook accounts?
How many people outside of HN do you think care about privacy for real? Like about adtech surveillance and not about their naked photos leaking?
I doubt either % is very high sadly. We tend to say we care, but very few people actually do anything or use self hosted solutions or not tied to Apple or Google ecosystems.
You really cant blame VPN providers for selling on "privacy" hype and not delivering because most people dont care either way.
Might be I wrong, but I feel in west for most normal people use VPNs for torrents, watching porn and hidding activity from school or employeer. Small subsets are also sport fans who bypass geo blocking and people scheming for cheaper regional prices on netflix / steam / consoles.
I blame mullvad for messing up, but I do not suspect them of working with some state sponsored surveillance programme at the moment.
Do you have any facts? I know they really on _additional_ stuff, but do you have sources showing that they never use cookies or source IPs?
On that topic, though, is the Mullvad Browser, who's entire intention is to defeat browser fingerprinting.
Also if the threat model you're addressing w/ VPN usage is anything other than "I don't want my ISP to know what I'm doing" you need to use/do something else.