I've tried something like this and ran into a problem which I'm not seeing a fix for. Generally, to really apply this method, you need to understand the person asking the question.

>Tell the user that you will start by asking a few questions, one at a time, to see what they already know.

This puts it into the session each time. Do this once and it seems to work. By the 5th time, you are getting tired of reconfirming your mastery. This ends up depending upon the user to have a pretty solid foundation of what they know and then use this skill to expand the boundary a bit, and even then the AI can end up making so many assumptions that a person has trouble following it along.

I was thinking some sort of concept mapping kept long term might help, but such a mapping can become really difficult to build. Which should be expected when we consider that teaching isn't just its own discipline, but has many sub-disciplines targeted to specific areas (even if some of it, like classroom management, doesn't apply to this scenario).

I made a few attempts at a knowledge tracking tool for personal use to track how much I knew Japanese beyond a simple JLPT level or similar with the plan to tie it into something like this, but it wasn't great. I could 'cheat' and just tap into my Wanikani API to see my progress, but that still skips so many other areas of Japanese, and even when WaniKani says something is burned, I really want AI to still use it until I truly internalize it like I do words in English. I suspect other areas of knowledge are similar.

This works really well.

A positive take: this works much better than the many projects on gh that did this before Jane Street. This would be evidence that creativity and execution still matter, even when the product is just an md file

A cynical take: network effects are everything. tens of projects doing the same thing and this only gains traction when an already popular org does it

someone at jane street likes https://www.mathacademy.com/how-it-works

I see a lot of this at my company too - everyone re-inventing the same ideas over and over. I wonder how much people actually adopt other teams' implementation of this. In my experience I'm bombarded with a thousand different iterations of these things that everyone wants to advertise as a sort of "Look at me! I'm so AI!" but I simply don't have time to evaluate every different persons variation on these things, I certainly don't have time to really thoroughly benchmark them, and from experience a lot what I needed a carefully tuned skill to achieve 6 months ago is trivial for a better tuned agent today. I'm literally getting to the point where even my own skills are beginning to bit-rot.

If it was this easy, educators would have written up sets of standard Socratic teaching prompts long ago. Like perhaps in the time of Socrates himself. Or the British empire or John Dewey etc. There is probably some great research in the field of Education on this approach.

LLMs should first learn how to write English for humans.

Right now what they produce reads like legalese, or patents.

I suspect this is because the AI labs are optimizing for agents talking to agents, rather than AI talking to humans (which is harder to optimize, and does not improve intelligence as fast).

Sensei, please teach me the ways to reach alpha just like you.

Jokes aside, pretty cool. This is just a skill, and it'd be easy to dismiss it. But I do want to become a better horse whisperer. Prompt engineering used to be mocked a lot, especially on HN, and skills really is just that but with a new branding.

I think Duolingo's days and the likes are counted, AI is just a great way to quickly get up to speed on a subject or a language.

I assume this isn't going to cause the LLM to start jumping me with questions about the nature of piety or justice

I WANT TO SEE THIS COMMENT ON COMMENTS

This comes off as AI consultant slop from a year ago. I wonder who is in charge of AI at Jane Street.

My client just got done firing someone who thought filling the codebase up with files like this would be a good idea.

It would seem the AI revolution is happening at wildly different rates depending on where you are standing.