Top
Best
New

Posted by tosh 2 days ago

Caveman: Why use many token when few token do trick(github.com)
874 points | 359 commentspage 5
rschiavone 2 days ago|
This trick reminds me of "OpenAI charges by the minute, so speed up your audio"

https://news.ycombinator.com/item?id=44376989

vntok 2 days ago|
Which worked great. Also, cut off silences.

> One half interesting / half depressing observation I made is that at my workplace any meeting recording I tried to transcribe in this way had its length reduced to almost 2/3 when cutting off the silence. Makes you think about the efficiency (or lack of it) of holding long(ish) meetings.

Art9681 1 day ago||
This was an experiment conducted during gpt-3.5 era, and again during the gpt-4 era.

There is a reason it is not a common/popular technique.

zahirbmirza 2 days ago||
You can also make huge spelling mistakes and use incomplete words with llms they just sem to know better than any spl chk wht you mean. I use such speak to cut my time spent typing to them.
floriangoebel 2 days ago|
Wouldn't this increase your token usage because the tokenizer now can't process whole words, but it needs to go letter by letter?
literalAardvark 2 days ago||
It doesn't go letter by letter, so not with current tokenizers.

There will likely be some internal reasoning going "I wonder if the user meant spell check, I'm gonna go with that one".

And it'll also bias the reasoning and output to internet speak instead of what you'd usually want, such as code or scientific jargon, which used to decrease output quality. I'm not sure if it still does

somethingsome 1 day ago||
I would like to see a (joke) skill that makes Claude talk in only toki pona. My guess is that it would explode the token count though.
fzeindl 2 days ago||
I tried this with early ChatGPT. Asked it to answer telegram style with as few tokens as possible. It is also interesting to ask it for jokes in this mode.
amelius 2 days ago|
It's especially funny to change your coworker's system prompt like that.
ungreased0675 2 days ago||
Does this actually result in less compute, or is it adding an additional “translate into caveman” step to the normal output?
andai 2 days ago||
So it's a prompt to turn Jarvis into Hulk!
nharada 1 day ago||
I wonder if this will actually be why the models move to "neuralese" or whatever non-language latent representation people work out. Interpretability disappears but efficiency potentially goes way up. Even without a performance increase that would be pretty huge.
shomp 1 day ago||
everyone who thinks this is a costly or bad idea is looking past a very salient finding: code doesn't need much language. sure, other things might need lots of language, but code does not. code is already basically language, just a really weird one. we call them programming languages. they're not human languages. they're languages of the machine. condensing the human-language---machine-language interface, good.

if goal make code, few word better. if goal make insight, more word better. depend on task. machine linear, mind not. consider LLM "thinking" is just edge-weights. if can set edge-weights into same setting with fewer tokens, you are winning.

justonceokay 1 day ago||
JOOK like when machine say facts. Machine and facts are friends. Numbers and names and “probably things” are all friends with machine.

JOOK no like when machine likes things. Maybe double standard. But forever machines do without like and without love. New like and love updates changing all the time. Makes JOOK question machine watching out for JOOK or watching out for machine.

JOOK like and love enough for himself and for machine too..

wvenable 1 day ago||
> They're not human languages. they're languages of the machine.

Disagree. Programming language for human to communicate with machine and human and human to communicate about machine. Programming language not native language of machine. Programming language for humans.

Otherwise make good point.

HarHarVeryFunny 2 days ago|
More like Pidgin English than caveman, perhaps, although caveman does make for a better name.
More comments...