Go, PET, Let Hen - Curious adventures in (Commodore) BASIC tokenizing

Posted by masswerk 7/5/2025

Go, PET, Let Hen - Curious adventures in (Commodore) BASIC tokenizing(www.masswerk.at)

23 points | 10 comments

OhMeadhbh 7/5/2025|

This takes me back a few years. I spent HOURS writing BASIC programs to analyze other BASIC programs as a kid. My favourite PET trick was to hide the basic source by putting a comment (REM statement) at the beginning and end of the program. Then POKEing the address of the ending comment in the "next line" link in the first line. It turns out that when the interpreter was running the program, it didn't use the "next line" link, it just assumed the bytes following the current line were the beginning of the next line. But the LIST command //did// use the link. So you could get a program to run perfectly fine, but when someone did a LIST, the only thing they saw were the two comments.

I can't remember if this worked on the C64, but it worked on the 4016 and 4032's in our high school's computer lab.

jim_lawless 7/5/2025|

You could do similar things on a C64 and other computers. You might try this out on a C64 emulator such as VICE.

10 REM NOTHING TO SEE HERE

20 PRINT "HELLO!"

POKE 2049,1

Run it. You'll see HELLO! LIST it and you'll continuously see line 10. If you try to LIST 20 the machine pretty much locks up.

Screen image is here:

https://jimlawless.net/images/remtrick.gif

(note that in the above image, you'll see two RUN lines ... it appears that I captured the screen as it was in mid-scroll... )

LocalH 7/5/2025||

At least on the C64, you could also put a line containing REM shift-L in the program, and the LIST command would crash out when encountering it.

masswerk 7/5/2025||

The problem being that the LIST routine should handle a comment like a string (which is how it is parsed and stored: a comment is essentially an unquoted string extending to the end of the line), but doesn't bail out of keyword expansion, whenever it encounters a REM token.

[Edit]

Coincidentally, a shifted "L" is PETSCII code 0xCC. Which is just one after the highest available token in Commodore BASIC 2.0 / V.2. (The last one being 0xCB, `GO`.) Therefor, a lookup into the keyword list will yield the terminating zero-byte, which probably causes the problem. (E.g., by defeating what was intended to be an unconditional branch instruction.)

(In BASIC 4.0 for the 40xx/80xx PETs, this is actually a valid token, namely `CONCAT`, which is expanded by LIST without further issues. Meaning, this kind of LIST protection can be broken by simply loading the program on one of the later PETs.)

LocalH 7/6/2025||

That's why I said "at least on the C64". This particular bug probably also happens on the VIC-20, but I don't know if PET BASIC 2.0 is the same or not. I'd also imagine the "protection" would break if loaded on BASIC 7 or 10 (although 10 was never finished, but it was based on BASIC 7 AFAIK, so there's that).

masswerk 7/6/2025|||

It also happens on the PET, before BASIC 4.0.

I had a cursory look at it: there's no check for REM, of any kind, but there's, of course, a check for quoted strings. For any tokens, an offset count is calculated by subtracting 0x7F from the token and then a loop starts searching for set sign-bits in the keyword list, decrementing the counter each time, it finds one. If the counter is zero, we must be right at the keyword, we're looking for. – But in those cases, where we are not…

For values larger than 0xCC (shift-L), the routine actually wraps around. E.g, shift-M is listed as FOR (0x81), shift-N as NEXT (0x82), and so on. This is, because the keyword list is exactly 256 bytes long (including GO and the terminating zero-byte, like on the C64) and is inspected by an indexed load instruction. So the index register just wraps around.

(BASIC 4.0, on the other hand, has to deal with a longer keyword list anyway, so it uses a more complex method to read the list, involving a pointer. Thus it happily spills over into the list of error messages, which follows immediately after this and happens to be encoded in the same way. Therefor, it will expand any excess-tokens into error messages. I guess, this will be the same with BASIC 7, and so on, if they are anything like this.)

masswerk 7/8/2025|||

Small update: I did a write-up on LIST and its mishaps (covering the PET to the C64 – so, no BASIC 7, etc.)

https://www.masswerk.at/nowgobang/2025/the-remarkable-misadv...

jklowden 7/5/2025|

Why do I remember that every C64 BASIC keyword was a 2-byte integer? A typing shortcut was to enter the first letter, followed by a "shifted" high-bit character. Every keyword was represented that way.

Variables were also 2-bytes, but ASCII. The user could enter a longer name, but only the first two characters were significant.

masswerk 7/5/2025||

Yes, variable names are 2 bytes, in their stored memory location in RAM. As these must be 7-bit ASCII bytes, the sign-bits and their distribution over these two bytes is used to encode the type. And all simple variables take 7 bytes of memory in total, regardless, whether the remaining 5 bytes are actually needed to store the data or not.

  sign-bits   type (payload)

  0   0   ... floating point number (1 byte exponent, 4 bytes mantissa)
  1   1   ... integer (2 bytes)
  0   1   ... string (1 byte length, 2-bytes pointer to location)
  1   0   ... FN function (2 bytes pointer to BASIC, 2 bytes pointer to parameter variable)

In a program (the BASIC text), though, variables names are stored in full and in plain ASCII, at whatever length of characters.

LocalH 7/5/2025||

Not every keyword could be abbreviated with only two characters. The linked article actually discusses this mechanism. Once tokenized, the keywords only took up a single byte.