Posted by pjmlp 2 days ago
True. but you don't solely rely on the declaration, do you? lots of power comes from static analysis.
From P3465: "why this is a scalable compile-time solution, because it requires only function-local analysis"
Profiles uses local analysis, as does borrow checking. Whole program analysis is something you don't want to mess with.
* template specialisations
* function overloading
* I believe const generics is still not there in Rust, or its necessarily more restricted.
In general metaprogramming facilities are more expressive in C++, with different other tradeoffs to Rust. But the tradeoffs don't include memory safety.
Template specializations are not in Rust because they have some surprisingly tricky interactions with lifetimes. It's not clear lifetimes can be added to C++ without having the same issue causing safety holes with templates. At least I think this might be an issue if you want to compile a function instance like `void foo<std::string_view>()` only once, instead of once for each different string data lifetime.
For function overloading this serves two purposes in C++ and I think Rust chooses a better option for both:
First, when there are similar features with different parameters, overloading lets you pretend the feature set was smaller but making them a single function. So e.g. C++ offers a single sort function but Rust distinguishes sort, sort_by and sort_by_key
Obviously all three have the same underlying implementation, but I feel the distinct names helps us understand when reading code what's important. If they're all named sort you may not notice that one of these calls is actually quite different.
Secondly, this provides a type of polymorphism, "Ad hoc polymorphism". For example if we ask whether name.contains('A') in C++ the contains function is overloaded to accept both char (a single byte 'A' the number 65) and several ways to represent strings in C++
In Rust name.contains('A') still works, but for a different reason 'A' is still a char, this time that's a Unicode Scalar Value, but the reason it works here is that char implements the Pattern trait, which is a trait for things which can be matched against part of a string. So name.contains(char::is_uppercase) works, name.contains(|ch| { /* arbitrary predicate for the character */ }) works, name.contains("Alex") works, and a third party crate could have it work for regular expressions or anything else.
I believe this more extensible alternative is strictly superior while also granting improved semantic value.
Just to clarify, does this mean NTTP in C++ is unsound as-is, or that trying to port C++ NTTP as-is to Rust would result in something unsound?
The Rust equivalent would need to be checked by the compiler and I think this only really delivers value if it's a feature in the safe Rust subset. So, the compiler must check what you wrote is sound, if it can't tell it must reject what you wrote. And that's why they decided to do the integer types first, that's definitely sound and it's a lot of value delivered.
As a whole concept you could probably say that the C++ NTTP is "unsound as-is" but that's so all-encompassing as to not be very useful, like saying C++ integer arithmetic is unsound. It's such a big thing that even though the problem is also big, it sort of drowns out the problem.
Noticing that std::abs is unsound has more impact because hey, that's a tiny function, why isn't it just properly defined for all inputs? But for the entire NTTP feature or arithmetic or ranges or something it's not a useful way to think about it IMO.
It explains its own motivation.
They wouldn't. The point is, if you were serious about making a memory-safe C++, this is what you'd need to do.
Simply making C++ compilers compatible with one another is a constant struggle. Making Rust work well with existing C++ code is even more difficult. Thus, it is far easier to make something like Clang understand and compile C++-specific annotations alongside legacy C++ code than making rustc understand C++ types. Moreover, teams of C++ programmers will have an easier time writing annotated C++ than they would learning an entirely new language. And it's important to recognize how deeply entrenched C++ is in many areas, especially when you consider things like OpenMP, OpenACC, CUDA, HIP/ROCm, Kokkos, etc etc etc.
On paper it does sound quite promising, but looking at the Github repo I can't help but shake the feeling that it is more talk than action. It seems to have basically been abandoned before it ever saw any adoption?
Most of the profiles being argued for C++ only exist in PDF form, there isn't a single C++ compiler where you can validate those ideas, but trust the authors, the ideas will certainly work once implemented.
I don’t know about real world adoption, nor do I think it’s really about what I’m saying. The proposal was made public a few weeks ago. There hasn’t been much time for real world projects to adopt it, if that were even the goal. The point is that it does exist today, and you can try it out. It’s more than just a design on paper.
He single handedly wrote his own C++ front-end and then proceeded to implement a butt load of extensions which other members of the committees poo-pood as being to hard to implement in the compiler.
Every couple of weeks he implements something new. He's a real joy to follow.
Now that virtual address space is cheap, it's possible to recompile C (or presumably C++) with a fully-safe runtime (requiring annotation only around nasty things like `union sigval`), but this is an ABI break and has nontrivial overhead (note that AddressSanitizers has ~2x overhead and only catches some optimistic cases) unless you mandate additional annotation.
I’ve got some programs where the ASan overhead is 10× or more. Admittedly, they are somewhat peculiar—one’s an interpreter for a low-level bytecode, the other’s largely a do-nothing benchmark for measuring the overhead of a heartbeat scheduler. The point is, the overhead can vary a lot depending on e.g. how many mallocs your code does.
This does not contradict your point in any way, to be clear. I was just very surprised when I first hit that behaviour expecting ASan’s usual overhead of “not bad, definitely not Valgrind”, so I wanted to share it.
Every set of constraints prevents many useful programs. If those useful programs can still be specified in slightly different ways but it prevents many more broken programs, those constraints may be a net improvement on the status quo.
Does it? You didn't list any. It certainly prevents writing a tremendous number of programs which are nonsense.
Controlled mutable aliasing is fine. Uncontrolled is dangerous.
Except that's a really harsh and unproductive way to phrase it. The existence of "unsafe" is not a "confession" that safe code is impossible so why should even try.
Progress in safety is still made when an unsafe capability is removed from the normal, common, suggested way to do things and it is moved to something you need to ask for and take responsibility for, and hemmed in by other safe constructs around it.
There is movement towards adopting the same approach for C++ by some vendors. The biggest problem is that there's a lot of crap C++ out there buried in erroneous custom comparators and so if you change how sorting works even very slightly you blow up lots of fragile nonsense written in C++ and from their point of view you broke their software even though what happened is they wrote nonsense. Next biggest problem is that C++ named their unstable sort "sort" so sometimes people needed a stable sort and they got an unstable sort but in their test suites everything checked out by chance, and with a new algorithm now the test blows up because it's an unstable sort...
Yes, it prevents many useful programs.
I think it also prevents many many many more useless broken incorrect programs from wreaking havoc or being used as exploit delivery vehicles.
An implementation of `sort` would start with: `assert (begin.allocation_base == end.allocation_base)`. Most likely, this would be implicit when `end - begin` or `begin < end` is called (but not `!=`, which is well-defined between unrelated pointers).
If we ignore the uninitialized data (which is not the same kind of UB, and usually not interesting), the `strlen` loop would assert when, after not encountering a NUL, `s.allocation_offset` exceeds `100` (which is known in the allocation metadata).
struct S {
int arr1[100];
std::string s;
int arr2[100];
};
void foo(S& s) {
//arr1 and arr2 are in the same complete object
//so they are in the same allocation
std::sort(std::begin(arr1), std::end(arr2));
}
To make it sound you really need to track arrays specifically (including implicit single-element ones), not just allocations.It's surely somewhat feasible, as the constexpr interpreters in compilers do track it, but something like that would probably be way inefficient at runtime.
That's really an aliasing problem, not specific to arrays. The tricky part about tagging memory with a full type is that a lot of code, even in standards, relies on aliasing that's supposed to be forbidden.
Still, even if the `sort` isn't detected (due to the user insisting on permissiveness rather than adding annotations), it would still detect any attempt to use `s` afterward.
As for a limited array-specific solution ... I can't think of one that would handle all variants of `sort((int *)&s[0], (int *)&s[1])`. And do we want to forbid `sort(&s.arr[0], (int *)&s.s)`?
I'd bet that there are corners of non-UB vlaid programs that would be sensitive to the fact that pointers are no longer simple numbers, but maybe that's wrong or at least could be worked around.
I would add that to make this memory safe for multi-threaded programs, you also need to implcitly synchronize on fat pointer accesses, to prevent corrupted pointers from forming.
The strlen could be made safe by having the malloc return the address 100 bytes before the end of the page. If it did make it to the end of the 100 bytes it would fall over safely. Result would be nonsense of course. Of course if you read bytes < than the returned ptr you would have 4k-100 bytes before it fails on bad page.
This assumption doesn't hold for real world systems -- most allocations are small, so heap allocators typically support a minimum allocation size of 16 bytes. Requiring each allocation to have its own page would be a 256x memory overhead for these small allocations (assuming 4k pages, and even worse as systems are moving to 16k or 64k page size). Not to mention destroying your TLB.
Also, guard pages only solve the problem if the program tries to access the entire array (like when sorting or reading a string). For other types of out-of-bounds array access where an attacker can control the index accessed, they can just pass in an index high enough to jump over the guard page. You can "fix" this with probing, but at that point just bounds-checking is much simpler and more performant.
It certainly helps, but is not a full solution.
The hard part is figuring out which words of memory should be treated as pointers, so that you know when to alter the reference counts.
Most C programs don't rely on all the weird guarantees that C mandates (relying on asm, which is also problematic, is probably more common), but for the ones that do it is quite problematic.
Static allocations only need full heap compatibility if `dlclose` isn't a nop.
And TLS is the forgotten step-child, but at the lowest level it's ultimately just implemented on normal allocations.
I sure hope you don't use any stack frames while writing the stack frame allocator.
I wrote a compiler extension just for this issue since there wasn't any.
Works great with a single thread.
There's overhead (depending on how much you're willing to annotate it and how much you can infer), but the only "hard" thing is the race between accessing a field and and changing the refcount of the object it points to, and [even ignoring alternative CAS approaches] that's easy enough if you control the allocator (do not return memory to the OS until all running threads have checked in).
Note that, in contrast the the common refcount approach, it's probably better to introduce a "this is in use; crash on free" flag to significantly reduce the overhead.
If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.
It comes down to the blast radius for a mistake. A mistake involving UB can potentially result in completely arbitrary behavior. A mistake in the safe subset of a language is still a mistake, but the universe of possible consequences is smaller. How much smaller depends on the language in question.
> I'm also concerned with the differing functional behavior between debug and release
IIRC this was a compromise. In an ideal world Rust would always panic on overflow, but the performance consequences were considered to be severe enough to potentially hinder adoption. In addition, overflow checking was not considered memory safety-critical as mandatory bounds checks in safe code would prevent overflow errors from causing memory safety issues in safe Rust.
I believe at the time it was stated that if the cost of overflow checking ever got low enough checking may be (re)enabled on release builds. I'm not sure whether that's still in the cards.
It's not ideal and can lead to problems as you point out when unsafe code is involved (also e.g., CVE-2018-1000810 [0]), but that's the nature of compromises, for better or worse.
[0]: https://groups.google.com/g/rustlang-security-announcements/...
BTW, I found one of the rust rfc documents helpful for understanding the borrow checker. Do you know if there is a similar rust RFC document for the upcoming polonius borrowchecker, even if it's just a working copy? I'm having trouble finding anything beyond some blog posts.
body {
background-color: #1f1f1f;
color: #efefef;
}.sourceCode {
background-color: #3f3f3f;
}
If anything as simple as the following needs lifetime annotations then your proposed solution will not be used by anyone:
const int& f4(std::map<int, int>& map, const int& key) { return map[key]; }
Your function would look like this in Rust:
fn f4<'a>(map: &'a Map<i32, i32>, key: &i32) -> &'a i32 { ... }
You don't need much more than a superficial understanding of Rust's lifetime syntax to understand what's going on here, and you have much more information about the function.The search term for those who'd like to follow up is "Superoptimization", which is one of the perennial ideas that programmers get that will Change the World if it is "just" implemented and "why hasn't anyone else done it I guess maybe they're just stupid", except it turns out to not work in practice. In a nutshell, the complexity classes involved just get too high.
(An interesting question I have is whether a language could be designed from the get-go to work with some useful subset of superoptimizations, but unfortunately, it's really hard to answer such questions when the bare minimum to have a good chance of success is 30 years of fairly specific experience before one even really stands a chance, and by then that's very unlikely to be what that person wants to work on.)
Some amount of non-local inference might also be possible for templated C++ code that already lack a proper separate compilation story.
The reason to not do this isn’t due to impossibility, but for other factors: it’s computationally expensive, I’d you think compile times are already bad, get ready for them to get way worse. Also, changing the body can break code in competently different parts of your program, as changing the body changes the signature and can now invalidate callers.