Posted by dash2 2 days ago
I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the best people in every field will be using AI to amplify their work in good ways.
"At the other end of the distribution, AI students who spend more than 65 minutes on their homework receive homework and exam scores similar to those of non-AI students, suggesting that these students do not use generative AI for homework assignments. However, this group consists entirely of students who adopted generative AI no more than Öve months. Six months after adoption, no AI student spends more than 65 minutes completing their homework (see Figure A5). This is consistent with the gradual process of learning how to use AI tools. It also suggests that AI crowds out the highest level of e§ort."
"Interestingly, in the range of 50-65 minutes, the median and the interquartile range of exam scores of AI and non-AI students are similar. This implies that, in the range where AI students and non-AI students have overlapping homework times, students who spend the same amount of time completing homework on average receive similar exam scores."
"This pattern shows that students who spend the same amount of time on homework learn similarly, with or without generative AI. In other words, generative AI reduces time spent learning for the majority of AI students but not learning efficiency for those who spend the same time studying as the non-AI students."
I'm confident it's an amplifier for people who know how learning works and already do a lot of it, successfully. However the level of "learning fluency" I'm talking about isn't reached for many until late college or grad school, and sometimes not at all. So I'm not surprised by the quoted results for 12-18 year olds.
"The negative learning effects are larger for students with higher initial achievement. The differences in the estimated full (6-10 month average) effects are substantial, with a 50% gap between the most negative effect (-24 percent) for the highest tercile and the least negative (-16 percent) for the lowest tercile. "
Not top 10% as you asked, but the closest to what you asked. My working hypothesis is that top performance is highly correlated with willingness to work hard, and AI decreases the motivation to work hard.
I think if you take a physics class where the student is intelligent and intrinsically motivated through their own interest (I admit this is rare) then AI probably helps.
they're not designed to measure general aptitude, or function as admissions criteria, or screen for job applications, or any other numerous things they are used for.
there can be many questions of pedagogy. one of them is, what do our exams measure and how do we use them? professors who say, "My exam is designed to measure who studies, not be used for all these other purposes that they are actually used for" - I don't buy it. It's the same as late night comedians saying they are not responsible for solutions, even when spending 90% of their air time making political jokes.
THIS is the pedagogical issue, that pedagogy has NEVER caught up with the scope of responsibilities. This is acute in STEM - I mean, the humanities departments are generally pretty well run, all things considered, in this regard. Generative AI is accelerating that pre-existing crisis.
Huh? They're designed to measure how much you know. They can't see how much you study, nor would they have reason to be interested.
At the end of the day though what matters is what you know. Furthermore, if it's a serious subject, it shouldn't matter whether you learned it from this teacher or from another school and teacher, as long as your knowledge is correct. Knowing the idiosyncracies of this particular teacher should not factor into the grade. A serious subject can be learned on one continent and examined on another. Bullshit courses are all about learning pet peeves and hobby horses of a particular teacher.
The "slightly higher" performance is based on statistically insignificant samples (between 4 and 20 students, depending on the context, out of the total population of 26,000): https://bsky.app/profile/benjaminjriley.bsky.social/post/3mt...
Citation needed? I have no clue where you got this from. I hadn't even heard of it as a conjecture, let alone as something anyone accepted, let alone as gene rally accepted...
1. They don't do any homework.
2. All the in-class time is split between the teacher babysitting and playing social worker to problem students, and lecturing, with little to no opportunity to actually practice what they've learned?
I understand that some students don't have home environments that are conductive to doing homework well. I understand that some students are enrolled in five hours a day of extracurricular university-application-padding activities. I understand that some students have incredibly poor screen discipline and impulse control.
But I don't understand that anyone has magically figured out how to teach complicated things to students, and have it stick without them spending a lot of time practicing what they are learning.
As anyone who has tried to do something hard knows, the first step to being good at something is to spend a lot of time being pretty shit at it.
A student who has written and received feedback on 500,000 written words is going to be way better at writing than that same student who wrote 50,000, just like someone who has put 5,000 hours of focused practice into playing the piano is going to be better than my dumb ass, who has only put 100 hours in.
(If you found the solution to get good at stuff without practicing it, I'd love to get good at piano without putting any homework in on it.)
We could probably cook up thousands of examples of the same problem (YouTube DIY tutorials, GPS navigation, etc.)
Same can be said of technology in general tbh.
Suppose it's good to learn how elastic the brain is, in both directions, at a young age where it doesn't matter.
We need to shift the incentives by adding ruinous penalties for things that are currently quite commonplace if they are done by large players. Some dude training his own AI on his own computer can scrape and train. The fine for OpenAI or Meta using a single copyrighted book without permission should be in the tens or hundreds of millions.
What we're seeing currently in our society is a "loophole inversion" where the rules have an effect mainly via their loopholes. The most profitable activity is to find loopholes and exploit them as frenetically as possible to gain as much advantage as you can before the loophole is closed, or get people hooked on the loophole so it's retroactively legalized. Entities that are big enough to do this are big enough because they have lots of money behind them. Entities doing the same kinds of things without lots of money are not really doing much harm. So the best approach is to adopt a "sliding scale" in which even tiny violations by wealthy actors result in penalties enormously greater than fairly large violations by small players.
LLMs make significant mistakes frequently and smart people have no way of judging those mistakes outside their domain expertise. They are also sycophantic and great at being an echo chamber which makes people feel smart even if they are not.
So I think the burden of proof is on you to prove that they somehow amplify intelligence, it seems highly unlikely.
Smart people know LLMs confabulate and tell them they’re Absolutely Right! Smart people don’t want to be embarrassed by trusting the hallucination machine and revealing their gullibility to others.
All of those sound like flaws and defects of dumb people?
Raising the noise floor like this only makes it that much harder to find "Smart" people, which we were already doing terrible at.
I use Claude every single day, but this is such a bad tradeoff. Maybe it will help me standup a quick fix when that is needed. Maybe it can help me dig through documentation to find relevant bits and figure out the unstated assumptions underlying it. Maybe it helps me generate test cases.
Meanwhile, my day to day life is now noise. All social media is noise. All content is noise. Slop pours onto me from all directions. Writing more test cases isn't helping me.
Am I smart? Am I dumb? I don't care, right now I'm deafened
I'm using Claude at work myself and am impressed with the product, but notice that this is the only reason I need to use it at all. Our product pages were shit to begin with, now they're AI-generated and somehow even worse. Our procedures are incomprehensible spaghetti with enough arbitrary context switching to give a sadistic Soviet municipal administrator an erection at the thought of watching anyone try to actually follow them.
Use AI to create inefficiencies, then use AI to bypass them. Those who can't do the latter will struggle to survive.
Clarification: to value “smart” people, which we were already doing terrible at.
It does give us a new heuristic, though: people who are willing to completely cut generative AI out of their lives (cold-turkey, if you ever started using it) are a much smaller group of, predominantly thoughtful, people. You do have to give up Claude to be part of this group, but from what you say, that's no great loss, and no longer being deafened is worth it.
This has considerable advantages over conventional elitism, because the barrier-to-entry is negative in almost all cases.
The one exception I've found is assistive tech, where the state-of-the-art is so poor that vibecoded slop is genuinely an improvement over the state-of-the-art, and in many cases the tooling simply isn't available to make your own assistive tech (unless you want to bootstrap an entire networked computing environment, which isn't very helpful when you want to do your online banking and do not, in fact, work at your bank).
But there are not many principled exceptions where you could seriously argue that the trade-off is worth it. Take mathematics, for example, which we often see touted as a "good use-case" of generative AI. The primary advantage of generative AI in mathematics is being able to search though a vast corpus of ivory towers and inconsistent terminology (without proper attribution) to locate and connect ideas that can help solve problems. The deficiency this is addressing is elitism, inadequate communication, and inadequate indexing within academic mathematics. This problem is entirely created by the academic mathematicians, and has been known for nearly a century (per https://en.wikipedia.org/w/index.php?title=Nicolas_Bourbaki&...):
> Bourbaki was founded in response to the effects of the First World War which caused the death of a generation of French mathematicians; as a result, young university instructors were forced to use dated texts. While teaching at the University of Strasbourg, Henri Cartan complained to his colleague André Weil of the inadequacy of available course material, which prompted Weil to propose a meeting with others in Paris to collectively write a modern analysis textbook.
To my knowledge, this is the only organised project to clean up and improve mathematical communication. Everything else (Metamath, Mizar, AFP, Lean) is yet another ivory tower. The Wikipedia article on this topic (https://en.wikipedia.org/wiki/Mathematical_knowledge_managem...) risks deletion as non-notable, that's how little anyone's actually trying. They made their own bed, and generative AI will only provide a brief respite from having to lie in it. (I was surprised how many other "compelling" use-cases evaporated when I applied this razor to them: the sibling comment https://news.ycombinator.com/item?id=49392265 points out one such.)
Vibe-coding assistive tech which doesn't yet exist, as a temporary scaffold to improve the quality-of-life of yourself and others in a social world dominated by non-essential access barriers is, to my knowledge, the only exception to this principle that can be justified. If you treat people who make other excuses, or who don't even bother with excuses, as not worth listening to, you lose little – and doubly-so, if you make your stance clear, so that others know the "cost" of gaining your attention.
We need to restructure the system to treat failure as a signal instead of a disaster. Grades should come from hard randomized exams with unlimited retakes so one bad day won't hurt you. Homework should be optional material for self study, evaluated by teachers if you choose to do it but never forced.
Hold back students for individual classes instead of a whole grade so failing one can't ruin your social life and teachers are more willing to do it. F students will realize they have to study, start actually learning and then pass on the second time. No big deal. It happened to my friends in college, no reason they can't do it in high schools.
Discipline is a skill and it's one you have to get from experience. If you try and force kids to study when they don't want to "for their own good" you're not actually helping them. Everyone needs to find their own path. Let people fail.
At Caltech, homework was assigned but had no bearing on your grade. The grades were based on the midterm and final exams.
But not mastering the homework usually resulted in flunking the exams. There were "retch" sessions after each homework assignment that was staffed by a grad student, and the purpose was to help the students understand the homework problems. I knew only one person (Hal Finney) who was so smart he didn't need to do the homework.
I learned the hard way that the path to success was:
1. never miss a lecture, no matter what
2. take notes by hand during lecture
3. do the homework on time, and make sure you understand every problem. Take advantage of the retch sessions.
And that worked for me.
We didn't call them retch sessions, and they were taught by the instructors though. We also were encouraged to peer tutor and since we were all restricted to one building the homework was always group work allowed.
Also had badges to track time spent in the building for required study hours, though some people gave up and just slept at their desks when they started sliding down the grade scale and the hours racked up.
Generally I think I did 30 hours of studying/homework (went up and down depending on what was being taught, but was around that) a week (for 12-15 hours of actual lecturing), with some of my friends putting in 50% more. Generally the only day we weren't there was Saturdays. Most of the day Sunday was usually spent in class preparing for the next week.
The general pattern at Caltech was 2 hours of study for every hour of lecture. Which was quite a shock to me.
I found that (1) I didn't need to take anywhere near as many notes during lecture and (2) I could ask way, way, way more relevant questions.
This also helped tremendously when it came to studying for the actuarial exams.
So the endgame was figuring out what the tests in previous years looked like (cause it was likely gonna be a copy paste affair), do a targeted study run for those exercises and 9/10 you would pass.
As a student of a top-tier French Master's degree, I consulted with a teacher to deal with exhaustion and to ask for a class rescheduling for my case. Explaining my situation, the teacher looks at me and interjected:
—Waitwaitwait. You... you went to all lectures!?
And he was right. Rather than following the curriculum, I should have developed my taste for various engineering topics and only used the classes as entertainment.
When I was there, a long time ago, exams were timed and were usually open book open note. Blue books were filled in. You were trusted to adhere by those rules, and most students did their exams in their dorm rooms.
The evidence that the students honored the rules was some exams resulted in a 50% failure rate.
As for me, I went there because I wanted to learn the material. I did not care about getting a diploma. (Mine is in the basement somewhere.) I did not take any "easy A" classes, because I wanted a return on my time and tuition investment. (Though, easy A classes were hard to find at Caltech.) I wasn't even going to attend graduation, but my parents showed up and I attended to please them.
The classes, year by year, were dependent on mastering the previous year's classes. So if you cheat with AI, you're digging yourself into a bigger and bigger hole. Caltech rewires your brain. If you don't learn the stuff, you're going to be one of those EEs who carries around a card with V=A*R, V/A=R, V/R=A printed on it.
Optional homework is often a disaster. At best, students would do it right before an exam and the goal of education is not to just pass exams. They’d probably still get a lower score than if they did the homework when they were supposed to.
What I think is better is to have a due date, but just make the maximum 10% each day it is late. So after 2 days, the highest score you could receive would be 80%.
I liked that system because it gave some flexibility with deadlines while still encouraging you to turn things in on time.
Do you have evidence for this beyond your friends (who were accepted into college)?
Education system, contrary to popular belief/name, isn't tailored to educate but to select winners and losers which then will be picked on the job market.
That's why it seems absurd when you think about it as an institution that aims to educate. That's because that isn't the real purpose of it. The purpose is to stratify and classify early.
Schools do not operate in a vacuum; they serve as credentialing gatekeepers for a hyper-competitive capitalist job market. If everyone could easily retake exams until they got an A, grades would lose their primary utility for employers and universities: differentiation. Society relies on schools to provide a neat hierarchy of candidates that for one reason or another thrived in difficult environment of adolescent schooling.
The system often prioritizes compliance, endurance of boredom, and social maneuvering over actual critical thinking precisely because those traits align with corporate hierarchies.
this rhetoric is pretending to be an alternative to coercion. IMO the ideas you are talking about are well trodden and are still coercion nonetheless.
Study design: "David Stromberg of Stockholm University and Victor Lei and Wu Yanhui of the University of Hong Kong set out to fill the gap. They tracked 27,000 pupils aged 12-18 in China, where ai adoption has been fast. Around 80% reported using models such as Doubao and DeepSeek; the other 20% formed the control group."
As with most training the journey is the point, not the destination.
That said, I think smart use of AI could help. It could explain concepts in a way that might help you understand better, it could probe your knowledge in a more dynamic way by tailoring questions, and so on. This requires the AI be restrained by some harness, not free to write down the answers for you.
People say that all the time, but does anyone really think a lack of good explanations for things is a limiting factor in 2026? Or even 2010?
AI gives you a way to get the same help without asking another human. Say of that what you will, but not everyone was comfortable asking other humans for help even back then.
Kids won't be using LLM tooling as a personal tutor following some sort of Socratic method, they'll ask it to solve their home/coursework for them and blindly copy/paste the answer. Hell, they'll just manually copy down what's on their screen if copy/pasting isn't possible for whatever reason.
Obviously exceptions exist, but I'd wager from being an ex-kid myself the type of kid who would genuinely use these tools for actual proper self-tutoring would be an extreme rarity.
How would this magic harness look like and why would anyone use it?
Put that in your agents.md.
Learning the material (using the text book and videos) was what they had to do at night and class time was spent working through problems or applying the material in some way.
I do like it though, and I like that college generally leaned that way more (and was better balanced, as it had substantially less time in classrooms, so you could study during the day).
IMHO there's a lot to recommend this style. A lot of students learn better when they can wrestle with the material at their own pace, on their own time, in their own setting. It quells a lot of anxiety about "I'm not following what the teacher is saying, am I stuped, will I look bad in front of all my peers?" And then class time can be spent identifying holes in your knowledge and getting instant feedback from an expert, which is where they are most useful.
Your answer - middle school - I find it extremely young for this, which is extra interesting.
That year I sent an email to her teacher to let them know that I'm the one responsible for her not doing all the work and if that's a problem, we need to talk. It wasn't a problem.
The one class that was flipped was a relief because she could breeze through the lesson much faster than would normally be spent on it in class and the amount of time on exercises was limited to the class time slot.
This is a pedagogical problem that AI merely exposed. Educators need to figure out How to make students choose the scenic route instead of having them optimize for the most efficient completion of a task.
All of this was at university of which 3 of them were at the same university.
Especially the artsy game design program definitely did not feel like it was preparing me to be a cog in some giant corporate wheel.
If you skip this, well your brain won't develop as much at period of life it's able to do so.
It is really a combination: you need to memorize a large corpus of information to operationalize knowledge — understanding a bunch of theorems in mathematics does not help much (even if you are able to prove them when you see them) if you can't remember the boundary conditions they hold under.
Or having good understanding of foreign language grammar won't help you if you do not memorize words that you need to express your thoughts.
Good literature will show you some of life's challenges and potentially let you think through them in a non-stressful situation (other than "I've got to finish the last 200 pages by Monday" ;)).
At the outskirts of one's knowledge this is inevitably the case. My impression it is still useful to know that a certain implication is possible - and one can look up the exact conditions.
AI is a big change; pedagogy is going to change too.
I mean, does it? The way I see it, the best LLMs have to offer is infinite patience (until they inevitably and unpredictably start to confabulate, which should be a gigantic red flag, anyhow), I don't see how LLMs can be used to explore teaching paradigms that haven't been explored before. We know pretty well from centuries of empirical experimentation how children's brains develop under different stimuli. It's not exactly something the software industry needed to "hack".
Using the forklift as a spotter and to assist in loading weights increased gains. Then again, a human can do all those things, and provide real human connection.
You don't need a final answer but you can't just answer "I don't know" otherwise you risk being slimed (shout out to anyone who still remembers "You Can't Do That on Television")
Is an acceptable answer.
Career exams tend to be a mix but a lot less ideology.
Your career is more akin to the totality of school than it is to any specific facet of it imo.
The social aspects are more important than the exam sitting most of the time.
> The social aspects are more important than the exam sitting most of the time.
To spell it out more clearly: every single aspect of your career is an exam. The interview is an exam, quite literally. The day-to-day responsibilities (meetings, planning, problem-solving, collaborating) are parts of the exam. If you're failing at those or automating them away, then what is your role? The assignments (solve X bug or add Y feature) are parts of the exam too. Failing to do these things will mean that, yes, you are failing the exam.
In the case of some management, is causing them to unlearn, forgetting about proper review and maintenance practices.