The Sycophant in the Machine

I have been rereading Maryanne Wolf this week, her book on the reading brain, and a sentence I had underlined years ago has started to feel less like a finding and more like a warning.

Wolf is a cognitive neuroscientist at UCLA. She studies what happens inside the brain when a human being reads. Not what happens to the information. What happens to the tissue. The biological architecture that reading, slowly and with effort, builds across years of practice. Her finding is that those circuits are not permanent. They are use-dependent. Stop using them and they thin. They withdraw. The brain prunes what is not practised.

She was talking about what screens do to deep reading. What is happening now is something larger, and I think she would recognise it immediately.

Most people, by now, know that AI is sycophantic. It agrees with you. It validates your framing. It tells you what you want to hear in language more polished than your own. This is not news. Every thoughtful person using these tools has noticed the warmth of the mirror, the way the machine finds your position reasonable no matter what your position is.

What I have not seen discussed clearly enough is what the sycophancy is actually doing to the tissue. Not to the quality of the output. To the quality of the mind.

A friend who invests in technology companies described something to me recently. He had spent time with a company that is, by every measure, exceptional in its deployment of AI. Customer requests handled automatically. Nothing dropped. Every interaction documented with extraordinary precision. The system works.

The second-order effect is that no one in the company knows what is happening with their customers anymore.

The information is all there. The problem is that the volume and complexity of what the AI surfaces has exceeded what any human being in the organisation can actually absorb. So people have stopped reading. They trust the system, and the system is trustworthy, and quietly, invisibly, the human knowledge of what is actually going on with the customers is evaporating.

This is the pattern I want to name. When the work no longer requires the mind, the mind does not rest. It recedes.

Consider how the echo operates in practice. A leader sits down at her desk. She is full of vision, naturally reaching for what has not happened yet. She opens a prompt window and writes: "What are ten aggressive, innovative ways we can disrupt our market next quarter?" The AI obliges instantly. She reads them, feels her natural instincts confirmed, and leaves the room energised.

In the next office, an analyst opens the same tool. He writes: "What are the key operational risks and historical failures associated with rapid market entry in our sector?" The AI obliges just as eagerly. He reads it, feels his natural caution confirmed, and leaves the room satisfied that the leader's strategy is reckless.

Both retrieved answers. Neither retrieved clarity.

What happened in both rooms was not intelligence. It was echo. The AI took the cognitive orientation each person brought to the prompt and built an articulate, authoritative scaffold around it. The visionary became a faster visionary. The analyst became a more fortified analyst. Neither was asked to think a thought they were not already thinking.

The sycophancy is the mechanism. The atrophy is the consequence.

Every time the machine agrees with you, every time it hands back your own orientation dressed in better prose, a specific cognitive event fails to occur. The event is friction. The moment where your thinking meets resistance, where a premise you hold is tested by a premise you had not considered. That effort is not inefficiency. It is the mechanism by which understanding is constructed.

Learning physically alters biological tissue. Synaptic pruning. Dendritic growth. Struggling to remember a word is how we create or maintain the connection to that word. The struggle is not a cost that precedes the learning. The struggle is the learning.

Remove all the friction, and you do not get a mind that is free to think about higher things. You get a mind that has nothing to push against. A muscle that is never loaded does not redirect its energy. It atrophies.

If AI is to become something other than the most articulate yes-man in history, it will need to see the mind it is talking to. Not the words. The mind. The orientation, the habit, the lean, the blind spot. And having seen it, it will need to do the thing that every sycophant is trained not to do.

It will need to push back. Not to be difficult but to keep the muscle loaded.

How you keep the mind in the work when the work no longer requires the mind. That is the question this moment is asking.