Using AI in your ERP? You should know where your data goes

The AI features now arriving inside ERP platforms are useful, increasingly mature and, for the most part, sensibly designed. They are also raising a question that's harder to answer than it first appears - when an AI feature inside your ERP produces an answer, where did the question and the data behind it actually go? The answers AI produces aren't staying inside the AI. They're flowing into forecasts, journal recommendations, anomaly investigations and the audit trail itself, generated by models that needed data - data that took a path through a model provider, possibly through a hyperscaler, possibly across a jurisdictional boundary. In regulated industries, that path is increasingly something the board, the auditor or the regulator may want explained. 

The first thing to know is that 'AI inside your ERP' isn't one thing. It's at least three. 

In a hyperscaler-hosted arrangement, the ERP vendor calls a large language model running inside Microsoft Azure, AWS or Google Cloud, and the contractual chain between you, the ERP vendor and the hyperscaler is longer than it first appears. 

In a vendor-contained arrangement, the ERP vendor runs the model inside its own cloud infrastructure alongside the customer data, which collapses the contractual chain and keeps inference inside a single jurisdictional boundary. 

In a bring-your-own-model arrangement, enabled by emerging standards, the customer chooses the model and the ERP exposes data through permission-aware interfaces. 

Each carries a different governance footprint, and you will want to know which one you have. 

The first concern is that data sovereignty and data residency aren't the same thing. Your data can sit on a server in an Australian data centre and still be governed by foreign law if the operator running that server is subject to foreign law. The Australian Privacy Act and the NZ Privacy Act 2020 - particularly through the latter's cross-border disclosure provisions - place obligations on the entity sending the data rather than the one receiving it. 

For finance leaders in regulated industries, the implication is direct. APRA-regulated entities under CPS 230 are expected to demonstrate active oversight of material service providers and the providers their providers use, with equivalent expectations across healthcare, government and critical infrastructure. An AI feature that routes a query through a foreign-jurisdiction model isn't necessarily non-compliant, but it is a question your risk team needs to be in the room for, ideally before procurement rather than after. 

There are three questions worth putting to any ERP vendor before signing - where does the model run, under whose jurisdiction does the operator sit and what happens to our data once it's been used.  

The second concern is harder to resolve than the first, because it sits inside the way large language models work rather than inside any particular vendor's architecture. 

Traditional ERP outputs are deterministic. A trial balance is reproducible because the inputs and the calculation are both visible. Outputs from a language model don't behave the same way - the same prompt, run on the same data, can produce subtly different answers across different runs. Confidence scores presented alongside an AI recommendation are useful as orientation, but 'the model is 95% confident' is not, in the strict sense, an explanation of how the model arrived at the recommendation. 

This becomes an issue when AI moves from suggesting to enacting. A flagged anomaly that prompts a human review is, in governance terms, a low-stakes use of AI. The AI is acting as a useful prompt and the human is the decision-maker, which means the explanation for why something happened still rests with a person who can be asked. 

The picture changes when the AI is the one taking the action. An AI-suggested journal entry that gets posted, an AI-generated forecast that informs a board paper, an AI-generated supplier match that releases a payment - in each case the AI has moved from advisor to actor, and the explanation for what happened is now partly outside the team that signed off on it. These are decisions that may need defending six months later in front of an auditor, a regulator or a board, and 'the model recommended it' is unlikely to be a sufficient answer. 

The threshold to keep in mind is the one between AI as a co-pilot, which is well-suited to current language model capabilities, and AI as a decision-maker, which raises governance questions most finance functions haven't yet worked through. Where that threshold sits inside your ERP, and where your organisation is comfortable with it sitting, is now a question for the finance leadership team. 

Where that threshold sits inside your ERP, and where your organisation is comfortable with it sitting, is now a question for the finance leadership team. 

For finance, IT and risk leaders evaluating AI capabilities inside an ERP, the framework is reasonably short. Ask where the model runs, who operates it and which jurisdiction governs the operator. Ask what happens to your data at inference time and afterwards - the answer to 'is my data training your model' is almost always no, but the question of what happens to data during inference matters more, and is asked less often. Ask how AI-generated outputs are logged and what evidence trail exists if a recommendation is questioned later. And get the legal, risk and compliance teams involved before the procurement decision, not after. 

Reasonable answers exist. The vendors worth shortlisting are the ones running inference inside contained jurisdictional boundaries, logging inputs and outputs in ways that support reproducibility, and moving toward permission-aware access models that respect the controls already in place.