Schools teaching “how to use AI” are missing the point: these things aren’t magic machines, they’re guessing games.
LLMs have no ability to reason or understand anything. They just regurgitate words using a probabilistic model. They are little more than a fancy autocomplete, and autocomplete isn’t always right (see also DYAC).
https://garymarcus.substack.com/p/llms-dont-do-formal-reasoning-and
They are not useful for learning if the information they give is wrong.
Neural networks are a tool and they have their uses, but something that generates garbage that is factually wrong a significant percentage of the time while burning our world to do it is not the tool our students need.