Much Ado About None: Sentinel Values in Python

Posted by sebg 2 days ago

Much Ado About None: Sentinel Values in Python(florianwilhelm.info)

38 points | 26 comments

pwdisswordfishq 2 hours ago|

So much trouble to avoid introducing the optional monad.

woodruffw 2 hours ago||

Python’s type system has union types (including Option), and using them is pretty ergonomic. As TFA points out the problem is that `None` gets contextually reused in all kinds of places where a proper sentinel type is more appropriate.

aw1621107 2 hours ago||

Pretty sure Python's union types don't nest, though. If you could nest optionals then I feel that reduces the need for separate sentinel types since e.g., you can distinguish between "value not present" (None) and "null value present" (Some(None))

amluto 1 hour ago|||

Indeed. Option[Foo] really just means Foo | None - you can’t have an Option[Foo] at runtime. Fundamentally, Python’s static typing system describes a language that is a sort of mediocre match for Python.

Another fun one is List[Foo]. Runtime Python has lists but you can’t ask a list what type of list it is, and you can’t write isinstance(a, List[Foo]).

ozyschmozy 2 hours ago|||

Can you explain or give an example of nested optional types? I don't quite see how that makes sense

aw1621107 2 hours ago|||

The idea is that optional values are either some value or no value, but sometimes the inner "some value" bit can itself be an optional value (in Rust this would be Option<Option<T>>, in C++ it'd be std::optional<std::optional<T>>, in Python it'd hypothetically be Option[Option[T]], etc.). Such a situation gives you three possible states:

- You have an inner value and the inner value is something non-null (an integer, a string, another optional, etc.)

- You have an inner value and the inner value is null

- You do not have an inner value

As a more concrete (albeit somewhat contrived) example, say there's some API you can query for the owner of a particular document ID where the document ID may or may not refer to an actual document (e.g., maybe you got it from a cache and the document has since been deleted). One way to express this would be (using Rust syntax)

    fn get_owner(id: DocumentId) -> Option<Option<UserId>> { ... }

So if you get None, the document does not exist; if you get Some(None), the document exists but it has no owner; and if you get Some(<id>) the document exists and it has an owner.

(This can probably be done better by creating new enums, but it's an example so give me a bit of slack)

A naive port of this signature to Python would be

    def get_owner(id: DocumentId) -> Option[Option[UserId]]: pass

But to type checkers this is equivalent to

    def get_owner(id: DocumentId) -> UserId | None: pass

So now you're in the pickle the OP describes: you can end up with a None, but you no longer knonw whether it means "no document" or "no owner". The Python solution is to basically create a different name for either the outer None (e.g., MISSING as in the OP) or the inner one (e.g., NO_OWNER or the like).

what 1 hour ago||

This seems really silly to me. Why does get_owner need to differentiate between the document exists and maybe has an owner or the document doesn’t exist and thus has no owner. In both cases getting no owner should suffice? Why are you using get_owner to try and figure out if the document exists? Can you give an example where this distinction actually matters?

aw1621107 1 hour ago||

> In both cases getting no owner should suffice?

Depends on what exactly you plan on doing with the information, I'd imagine. I don't think it's necessarily a given that "no document" and "no owner" should be handled identically (e.g., searching for the updated document to try again vs. returning an operation-not-allowed error).

> Why are you using get_owner to try and figure out if the document exists?

Potential TOCTOU issues, perhaps? As I mentioned the document could have vanished out from under you so detecting this situation could let handle things appropriately (though to be fair that argument could probably extend to future operations as well).

Maybe a versioned filesystem would have made a bit more sense, where None would indicate a new version is available? Another example I've seen mentioned is trying to get a key-value pair from a JSON where the key may be missing.

But in any case, I did say it was a bit of a contrived example, so cut me a bit of slack :P It was just the first thing that came to mind.

The examples sibling comments give are probably more practical.

what 29 minutes ago||

> I don't think it's necessarily a given that "no document" and "no owner" should be handled identically (e.g., searching for the updated document to try again vs. returning an operation-not-allowed error).

If you care whether a document exists or not, you should use an API that exclusively tells you that instead of expecting everything related to documents telling you if the document exists or not. Otherwise you’re overloading the meaning. A non existent document has no owner. An API to fetch the owner of the document should not care if it exists or not, it does not matter.

aw1621107 19 minutes ago||

As I said, that sounds like a recipe for potential TOCTOU errors.

But again, this was an off-the-cuff example that I explicitly described as contrived from the start. I could probably flesh it out to make more sense, but I think it'd be easier just to refer to sibling comments rather than bikeshed this to death.

__s 1 hour ago||||

Say you select nullable column into vec, Vec<Option<T>>. vec.get returns Option<Option<T>>, which distinguishes null from out of bounds

I'd like nested Nullable in ClickHouse to use CoalescingMergeTree to support postgres unchanged TOAST values easily without getting mixed up on nulls

yccs27 1 hour ago|||

If you have a map, get(key) returns Option[ValueT]: None if the key is missing, Some(value) otherwise. If the value type itself is, say, Optional[String], your return type is Optional[Optional[String]]. Here None still always means the key is missing, while Some(None) means the value is just None.

Waterluvian 2 hours ago||

I’m curious what that would practically look like in Python.

I bet this would be a good use of vibe coding. Experiment with what-ifs. I can imagine being able to “show not tell” and reduce the signal-to-noise of comment opinions.

amluto 1 hour ago||

It would look like a very different language that would be thoroughly incompatible with the existing ecosystem.

throwaway2037 38 minutes ago||

Here is the sample code from the article:

    > MISSING = sentinel("MISSING")

    > def update_email(user_id: int, email: str | None | MISSING = MISSING) -> None:
    >     if email is MISSING:
    >         return  # nothing to update
    >     elif email is None:
    >         db.delete_email(user_id)
    >     else:
    >         db.set_email(user_id, email.lower())  # narrowed to str

I see this style frequently from long time Python programmers. The language is so dynamic that people begin to think this is a good way to program. Hot take: It is not. Why not two different functions called delete_email() and set_email()? That would be more clear.

tlonny 21 minutes ago|

I think this sort of pattern makes more sense when the fn is update_user and allows the caller to mutate multiple fields at once.

lpghatguy 3 hours ago||

When I was writing a bunch of Lua, I was a big fan of having a None sentinel value (Lua uses 'nil' for representing absence) and it was useful for a lot of things similar to Pydantic's MISSING.

This distinction is something that I think is actually handy about JavaScript, and distinguishing between 'null' and 'undefined' is sometimes useful for similar reasons.

nomel 45 minutes ago|

I feel like faster exceptions would be a better solution for this sort of thing. Leave functions returning the happy path with excepted values. Exceptional things, like missing values being handled at the level where of abstraction that actually cares to catch it:

chowells 3 hours ago||

I like it. I can't see how to do better while remaining compatible with Python idioms.

michalc 1 hour ago|

You could even say you never knew what you were… MISSING ;-)

RantyDave 3 hours ago|

"A simple cache returns None both for a missing key and for a key that maps to None."

No, it returns KeyError. Or at least it should.

0cf8612b2e1e 2 hours ago||

The author’s example chose to use the syntax that returns None on missing. If they wanted clarity the standard API does raise.

  cache = {“key”: “value”, “some input”: None}
  cache.get(“missing_key”)

  cache[“missing_key”]

There are better examples that could have been highlighted.

lyorig 3 hours ago|||

Perhaps the author has had their fair share of run-ins with libraries that didn't think their API through like that. But yes, they should.

nomel 41 minutes ago||

Very many people have been taught that exceptions are bad and should be avoided, because google avoids them.

travisd 3 hours ago||

it’s maybe less “pythonic” but at $dayjob we’ve very actively moved away from exceptions for things in favor of things that are statically type checked. too many things that become runtime bugs instead of type checking bugs.

with match now being available, we even return explicit error objects and the ergonomics are pretty good (reminds me of rust).

nomel 39 minutes ago||

How are these two things related? If you don't use exceptions, you just have to implement them yourself by manually bubbling up the error, all the way up the stack, rather than using the dedicated side channel to do so.

The problem I see is there's no way to enforce the check that you actually handled/bubbled up the error. With exceptions, you just crash if you don't handle it, which is much preferred if determinism is the goal. Is there some way to use the type system to do this? Maybe incoming are UnhandledResult outgoing are some HandledResult or something like that?