Sharpening AI Agents’ Judgement with Proven Analytics

Agents excel at judgement, not data analysis. Give them rigorous analytical results and let them do what they do best.

In September 2025, Thinking Machines Lab, the AI research company founded by former OpenAI CTO Mira Murati, ran a simple experiment. They asked a large language model the same question 1,000 times: “Tell me about Richard Feynman.” Same model, same prompt, with the setting that controls randomness turned down to zero. In theory, that should produce 1,000 identical answers.

It produced 80 different ones.

The answers matched word for word until the point where Feynman’s birthplace came up. Then 992 said Queens, New York, and 8 said New York City.

Harmless when the question is where a physicist was born. Not harmless when the question impacts an operational decision in renewable energy.

That’s why KEN, Kavaken’s AI agent, which now becomes autonomous, doesn’t do its own analysis. It works on the outputs of Kavaken’s machine learning models instead. Here’s why.

Operational data doesn’t forgive improvisation

Asset management teams and performance analysis engineers working in renewable energy already know that running analysis on operational data isn’t how it looks from the outside.

Up close, every step is a decision. Which wind speed do you trust when the nacelle anemometer is unavailable? How do you separate soiling losses from module degradation? Which periods do you filter out: curtailment, downtime, snow cover/icing, data gaps? Which inverters make a fair peer group when their arrays face different orientations? How do you estimate a battery’s state of health when it rarely goes through a full charge and discharge cycle? How do you select a target sensor when building a machine learning model for predictive maintenance?

Each of these has several defensible answers. Choose differently, and the same equipment looks fine on Monday and underperforming on Tuesday.

That’s why serious analysis runs on procedures. The same filtering, the same corrections, the same method, every time. Not because analysts lack options, but because a result is only useful if you can compare it with last month’s, with the next equipment’s, and with what the OEM sees.

An AI agent asked to “analyse the data” will do something reasonable. It may even write code to do it. But it decides the method on the spot, and next time it may decide differently. Two runs, two methods, two answers, both looking perfectly plausible.

Maths isn’t where language models shine

Language models have become impressive at solving maths problems (e.g., OpenAI recently said its models solved the 90-year-old Navier–Stokes problem, one of the Millennium Prize Problems). But solving a problem and producing a reliable number every time are not the same thing.

MIT researchers tested whether language models could detect anomalies in time-series data, with equipment like wind turbines, inverters and batteries in mind. State-of-the-art deep learning models still performed around 30% better than large language models.

Models will keep improving. But the lesson holds: purpose-built models do this job better than general-purpose ones.

Same input, same output

A trained machine learning model is a mathematical function. Give it the same input, and it returns the same output. Every time.

The Feynman experiment shows language models don’t work that way by default. Thinking Machines traced the main cause to server load: how many other requests are processed alongside yours slightly changes the computation, and small differences grow into different answers.

But even when that’s fixed, it only solves half the problem. A perfectly repeatable agent still chooses its own method, and a small change in the question or the data can lead it to a different one. Researchers recently put this to the test, giving AI analysts the same dataset and the same hypothesis. The analysts reached widely different conclusions, driven by different choices in how the data was cleaned, modelled and tested.

For an availability figure you’re disputing with an OEM, or a production loss you’re reporting to the board, that isn’t acceptable. If the number depends on which run you look at, it isn’t a number you can stand behind.

What agents are actually good at

None of this makes agents less valuable. It tells us where their value is.

What a good asset manager brings isn’t arithmetic. It’s judgment. Connecting a power curve deviation with a recent alarm. Recognising that three inverters flagging at once points to a shared cause, not three failures. Noticing that a battery rack’s faster capacity fade lines up with how it’s being dispatched. Knowing which contract clause matters, which precedent applies, and which issue deserves attention first.

That’s where agents shine. They read results the way an expert does: in context, across sources, with an eye on what to do next. They decide which question to ask, call the right tool, and explain the answer in plain language.

Giving agents autonomy to run on their own does not require them to do the maths themselves.

How we build it at Kavaken

So we split the work.

Kavaken’s machine learning models do the analysis: the same rigorous procedures, applied to every turbine, inverter and battery, every time. Their track record is measurable: more than 1,700 cases verified by our clients, and growing every day.

KEN works on the outputs of those models. It decides what to investigate, combines the results with contracts, maintenance history and past cases, and turns them into a recommendation you can act on. It almost never touches raw operational data, except to build reports and charts.

The models produce results you can reproduce. The agent makes them useful. You get rigorous analysis and expert judgement, without having to choose between them.

Cover image by barsrsind from Unsplash