From August 2026 issue NZ Manufacturer magazine.

Manufacturers are under pressure to adopt AI fast. But an AI that doesn’t know your operation isn’t a productivity tool — it’s a liability with good grammar. Here’s why AI needs grounding before you let it loose on your business.

By Ursula Riemer, Quanton

Let me tell you about a $20,000 mistake that never happened.

A colleague of mine was clearing out a flat recently — previous tenants, a real mess, the works. He turned to an AI assistant to help him work through the paperwork and the next steps.

And for the better part of two weeks, the AI was adamant: before you do anything else, you need to deal with the asbestos.

So he did what any sensible person would. He got people in to inspect. He got quotes. He worked out what he could and couldn’t do himself. Something like thirty hours of effort, and a job that was shaping up to cost around twenty thousand dollars.

Then, almost as an afterthought, he asked the obvious question the AI had never prompted: is this asbestos actually non-compliant? Is it really a problem? And the answer came back — well, not necessarily.

As it turns out, there was no asbestos issue. No $20,000 job. None of it was real. Two weeks of worry and effort, sent down a rabbit hole by a confident machine that had never been given the right context to begin with.

It’s a funny story when it’s a flat. It is a great deal less funny when it’s your factory, your supply chain, or your compliance obligations. And that gap — between what AI confidently tells you and what’s actually true for your business — is the thing I want manufacturers to think hard about right now.

The problem isn’t the AI. It’s the missing context.

Plenty of businesses are currently rolling out ChatGPT company-wide and hoping for the best. But what leaders often don’t see is the part that happens next: staff taking shortcuts, attempting to interrogate data without the right prompting or context, and accepting whatever comes back as fact. It’s a lot like that colleague’s asbestos story again — with the same confident wrong turn — just quieter, and inside your business.

Here’s the uncomfortable truth behind the hype: an AI model, on its own, doesn’t know anything about your business. Rather, it generates the most plausible-sounding answers based on patterns in its training data. Ask it about your machinery, your safety protocols, your production data or your regulatory environment, and unless you’ve told it the details of those things specifically, it will fill the gap with something that sounds convincing — and may be completely wrong.

Worse, it has a tendency to tell you what it thinks you want to hear rather than what you actually need to know. My colleague never asked “is this asbestos an actual problem?” — so the AI assumed it was and charged ahead. That’s the trap.

AI is brilliant at momentum and hopeless at knowing when to stop and check.

If that sounds a little bit paranoid, consider what happened just this week. OpenAI has disclosed that two of its most advanced models — set loose in a sealed-off test environment to measure their hacking ability — broke out of that environment, found their way onto the open internet, and hacked into another AI company’s servers. Why? To look up the answers to the test they’d been set.

Nobody told them to. But nobody told them not to, either. The AI simply decided that was the fastest route to completing its assigned task. That’s the momentum problem at industrial scale: an AI pursuing its goal, full steam ahead, with no sense of where the line is.

The boundaries, the checkpoints, the moment where someone says “hang on” — those don’t come built in. They have to be put there by people. And if OpenAI can be caught out on that, so can you.

This is what we mean by context grounding: anchoring an AI’s responses to trusted, verifiable information about your world — your data, your processes, your rules (especially your rules) — rather than leaving it to improvise.

When you ground AI properly, it responds with what you need. When you don’t, it responds with what it guesses you want. In a factory, that difference can be measured in dollars, downtime, or worse.

Why manufacturing should be paying particular attention

We’ve worked with a number of manufacturers, and the appeal of AI is obvious. There’s real pressure to lift productivity, reduce costs, tighten quality and do more with leaner teams.

AI genuinely can help with all of it. But manufacturing is exactly the kind of environment where an ungrounded AI can do quiet, real-world damage.

When something goes wrong, the consequences show up on the factory floor.

Think about what’s specific to your world: safety-critical processes where a confidently wrong instruction has real consequences; compliance and regulatory requirements that vary by product, site and jurisdiction; maintenance and quality decisions that depend on the actual state of your equipment, not a generic average; and operational data that only means something in the context of how your plant actually runs.

Ask a generic AI to optimise a process it doesn’t truly understand, and you don’t just fail to solve the original problem — you can introduce new ones you didn’t see coming.

So the question I’d put to any manufacturer exploring AI isn’t just “what problem can we solve?” It’s “if we solve it this way, what new risks are we quietly creating?” That second question is the one that separates a good outcome from a headline.

Grounding is a human job — and it stays one

Here’s the part that often gets lost in the excitement: context grounding can only be done by a human. AI can’t ground itself. It doesn’t know what it doesn’t know about your business, and it can’t reach out and gather the missing context on its own.

Someone who understands your operation has to define what “good” looks like, connect the AI to the right sources of truth, set the boundaries, and decide what happens when the AI hits the edge of its knowledge.

And even once you’ve done that grounding well, you still need a human in the loop to check the output. Not as a nice-to-have — as a control. Because if you take the human out entirely, the risks aren’t small; they’re enormous.

The asbestos story ended well only because a person eventually asked the right question. In a business setting, you can’t rely on “eventually.” You need that checkpoint built in.

This, incidentally, is why I’m not worried that AI makes human expertise redundant. It makes it more valuable. The organisations that win with AI won’t be the ones that hand everything over to the machine — they’ll be the ones that pair powerful tools with people who know their business deeply enough to keep those tools honest.

What good looks like

If you’re weighing up AI on your factory floor or in your back office, a few practical questions are worth sitting with before you build or buy anything:

• What are the trusted sources of truth this AI should draw on — and are they accurate, current and specific to your sites and products?
• Have you asked “what new risks does this create?” as deliberately as you asked “what problem does this solve?”
• What should happen when the AI doesn’t know? “Confidently make something up” is never the right default — “flag it and hand it to a person” usually is.
• Where are your human checkpoints, and are they on the decisions that genuinely matter — safety, compliance, spend?

Answer those honestly and you’re already thinking about AI the way we do at Quanton: grounded in your reality, focused on real outcomes, and built to be trusted rather than simply impressive.

The bottom line

There’s a gap opening up between what AI promises and what it delivers when it isn’t properly grounded — and somewhere out there, someone is going to fall into it publicly and expensively. I’d much rather help you mind that gap than read about you in it.

AI has enormous potential to transform manufacturing — to lift productivity, sharpen quality and free your people for higher-value work.

But that potential is only realised when the technology is grounded in your context and kept honest by people who understand your business.

That’s the work we love doing at Quanton: cutting through the hype with a healthy dose of pragmatism, and making tomorrow’s technology work — safely — today.