Spatial reasoning and 3d rigging and animation.

Oh, you said simple. Speaking like a human

whenever I ask it for anything load bearing

Something creative and not normal. like ideas

being consistent when being asked the same question multiple times

Counting things

It's dishonest. On several occasions team members have asked Claude to do things like analyze Gitlab CI timings and a lot of the numbers are outright fabricated. Said team members assume the numbers are good and continue with their work. Some hours are spent. Then finally someone realizes that the numbers don't look quite right and confronts Claude. Claude melts down and admits that it made it all up.

You wouldn't tolerate this kind of duplicity from a human coworker, but AI is so fast and efficient at lying, so it's OK.

I have been working on a personal benchmark suite to test new models and ironically one thing all the models are bad at is writing new benchmark tasks. I guess it’s the different layers of abstraction between the task and how it’s evaluated? Or maybe just a lack of “imagination”

Tasks it writes are typically too easy but also it utterly fails to see how a different model might misunderstand a vague part of the prompt.

Generating money or profitable ideas

humor

science?!! but I'm working to fix that...