Safe AI in monitoring should not determine the condition of a structure from free text. First, a deterministic module calculates statistics and rules, then an evidence pack is created, a language model can describe it, and an authorised person makes the decision. Require the vendor to prove that every stage works, to show the data policy, and to provide a mode that works without the model.
In brief
- Separate the calculator from the narrator. The same value and state should be reproducible without the generative service.
- The model should receive the smallest necessary package, not all raw series and project documents by default.
- Human oversight needs time, competence, the ability to reject the result, and access to evidence. A simple "approve" button is not enough.
- Model failure must not disable thresholds, alarms, or the basic numerical report.
- Retention, location, and use of data for model improvement depend on the specific vendor, configuration, and contract, not on the feature label alone.
In a sales presentation, every platform can call itself "AI". For a CTO and an engineer, the more important questions are less flashy: which module calculated the z-score and how it differs from an alarm threshold, whether the model saw raw data, what was written in the log, who changes the threshold, what appears after API key loss, and whether the recommendation can be rejected without stopping monitoring.
If the vendor cannot draw the flow and show sample input and output, the buyer takes on invisible risk. The best protection is not the declaration "a human remains in the loop" but the architecture, acceptance tests, and assigned responsibility.
An overview of what AI actually does in structural monitoring, and what remains a plain rule or calculation, also helps as a starting point for evaluating offers.
Four layers: calculator, evidence, narrator, decision
Human oversight over AI is the designed ability of a competent person to understand the system's role and limitations, assess evidence, reject the result, and make a responsible decision. The NIST AI RMF requires defining, assessing, and documenting oversight processes in line with organisational policy. It also stresses that human and AI roles can range from full autonomy to a purely supporting opinion.
In structural monitoring, a safe flow can be written as follows:
MEASUREMENT DATA + CONFIGURATION + QUALITY
|
v
DETERMINISTIC CALCULATOR
statistics, thresholds, anomalies, statuses
|
v
EVIDENCE PACK
scope, counts, results, sources, limitations
|
+------+------+
| |
v v
AI NARRATOR NO-AI REPORT
text from facts numbers and rules
| |
+------+------+
v
HUMAN DECISION
verification, action, responsibility record
The calculator must be repeatable: the same data and configuration produce the same result. The evidence pack limits the model context to the information needed for commentary and preserves the basis for the conclusion. The narrator organises the text, points to findings, and proposes next steps within the data boundaries. The human distinguishes suggestion from decision and takes into account the asset, procedure, inspection, and the consequences of error.
| Layer | Responsibility | Minimum acceptance evidence |
|---|---|---|
| calculator | numbers, thresholds, statuses, z-score | reference test and algorithm version |
| evidence pack | scope, sources, quality, limitations | sample payload and schema |
| narrator | description of supplied facts only | number-consistency and error-handling test |
| human | interpretation and operational decision | role, competence, time and decision record |
Why "human in the loop" alone does not protect
A human can approve a result without real control. If they see only a smooth paragraph without samples, scope, thresholds, and a chart, they act like a stamp. If they receive dozens of recommendations per day, fatigue appears, along with a tendency to accept automatically. Oversight requires work design, not just a role label.
The AI Act, in Article 14, describes for high-risk systems, among other things, the person's ability to understand capabilities and limits, monitor performance, recognise the tendency to rely too much on the output, interpret it correctly, and reject, override, or interrupt operation. This does not mean that every analytical module in structural monitoring automatically falls into that category. Classification is assessed for the specific use case. Still, the principles are a good test of oversight quality even outside legal obligations.
NIST points to automation bias, that is, overreliance on a seemingly precise recommendation. In monitoring, the risk increases when the text sounds certain while the evidence is hidden. That is why the narrator should cite calculated values, scope, and limitations, and the interface must allow fast access to the data.
Illustrative example. The model writes: "immediate inspection recommended." The pack, however, shows that the analysis covers 18 samples, the baseline is empty, and the last reading is three days old. The correct reaction is neither blind execution of the recommendation nor ignoring it. The human rejects the model's conclusion, starts the NO_DATA procedure, and orders a route inspection. The evidence pack made the correct action possible.
13 vendor questions with the required proof
A "yes" answer is not enough. For each question, ask for an artefact that can be reviewed during acceptance.
| No. | Question | Required proof |
|---|---|---|
| 1 | Which results are calculated deterministically, and which are generated by the model? | component diagram and responsibility table |
| 2 | What exactly is sent to the language model? | anonymised sample input payload |
| 3 | Does the model see raw series, asset names, or personal data? | field map and minimisation rules |
| 4 | Who sets the OK / WARNING / ALARM states? | rule code, configuration and reference test |
| 5 | How is the narrator output validated? | response schema, bad-format test and number test |
| 6 | What does the user get after a failure or missing key? | demo of fallback mode without the model service |
| 7 | How does the human reject the result and document the decision? | procedure, screen and audit record |
| 8 | How do you limit overreliance on the recommendation? | interface design with evidence and limitations |
| 9 | Which provider and model are used, and how is a change approved? | configuration, change log and regression test |
| 10 | Where, how long, and for what purpose is the payload processed? | data processing agreement, region, retention and subprocessors |
| 11 | Can the data be used to train or improve services? | current provider policy and contractual clause |
| 12 | How do you test hallucinations, omissions, and changes in model behaviour? | evaluation set, criteria and regression results |
| 13 | What is the plan for incidents, AI shutdown, and provider change? | procedure, responsibility, recovery time and exit test |
Questions 10 and 11 must be repeated after a provider or model change. A platform vendor may configure the model layer, but the processing policy belongs to the specific service and contract in force for that deployment. Do not accept the statement "data do not train models" without a document that covers the chosen variant and region.
Question 12 concerns numbers, not just style. The test should intentionally include missing data, conflicting fields, a value outside the range, an empty anomaly list, and a unit change. The narrator should preserve the caveat, not invent a cause, and not change the status calculated by the calculator.
Failure modes that must be exercised before purchase
First, availability failure. The model service does not respond, hits a limit, returns a timeout, or the key expires. Monitoring and alarms should still work, and the user should receive a deterministic report and a clear message that the narration is missing. The queue must not block current notifications.
The second mode is a formally incorrect response: a missing field, invalid JSON, a different unit, or text outside the schema. The system should reject it or replace it safely, not show half a report as if it were a full assessment.
The third is a response that sounds convincing but does not match the pack. The model may flip signs, rewrite 0.8 as 8, or add a cause not present in the data. The validator may catch part of the mismatch, but human review and a link to the evidence are also needed.
| Failure mode | Safe behaviour | Unacceptable behaviour |
|---|---|---|
| no model service | numerical report and explicit fallback | no alarms or a blank screen |
| bad format | rejection and error record | partial text treated as a full report |
| invented number | validation, warning, no publication | overwriting the calculator value |
| stale payload | block and show freshness | recommendation in the present tense |
| model change | regression test and approval | silent production behaviour change |
| policy change | re-assessment of contract and minimisation | assumptions based on previous terms |
| automatic reliance | evidence, training, right to reject | forced approval without context |
The NIST AI RMF organises risk management under Govern, Map, Measure and Manage. It requires, among other things, documenting roles, third-party risks, generalisation limits, performance monitoring, and response and withdrawal plans. ISO/IEC 42001 treats a similar problem as an AI management system in the organisation. Neither document replaces testing the specific flow on your data.
Acceptance procedure for one report
Choose a frozen data set for which the engineer knows the expected statistics. Run the calculator twice and compare the result. Then save the evidence pack and generate the narration several times if the model is nondeterministic. Every text must preserve values, scope, units, and limitations.
Then remove the model key or simulate a response timeout. Check whether the numerical report, alarms, and dashboard still work. Provide a wrong format and a contradictory recommendation. The system should reject it. Finally, the user rejects a formally correct narration and records the reason. This tests real oversight, not just the error-free path.
If the result goes to portfolio managers, also check the rules described in the guide on how to prepare a monitoring report for management: the summary must not remove evidence or limitations.
Checklist before signing the contract
- [ ] The diagram separates calculator, pack, narrator and decision.
- [ ] Each status has a deterministic source and a reference test.
- [ ] The payload to the model is minimised and available for audit.
- [ ] Raw series are not sent without justified need.
- [ ] The output has a schema, validation and numeric consistency control.
- [ ] The fallback works without a key and without access to the provider.
- [ ] Alarms remain independent of the narrator.
- [ ] The user can reject the result and record the reason.
- [ ] Roles, competence and review time are defined.
- [ ] The contract describes region, retention, subprocessors and security.
- [ ] Rules for training or improvement are confirmed in writing.
- [ ] A model change triggers regression testing and approval.
- [ ] The incident and AI-layer shutdown are exercised.
- [ ] The exit plan covers export of evidence and configuration.
What this looks like in Inclify
In Inclify, per-channel statistics, z-score against the baseline window, jumps, sample count, and statuses are created deterministically in the database. The WARNING / ALARM thresholds and the NO_DATA alarm work independently of the language model. The model has no tools to change the configuration and does not set the safety state.
The optional narrator receives a package of calculated results: scope, status, channel numbers, list of anomalies, and diagnostics. Raw time series are not passed through. The response should describe findings and recommendations in a structured format. When the key is missing or the language layer is turned off at deployment level, the platform returns a deterministic fallback text; numerical analysis still works.
The model and provider are configurable. Therefore, the processing location, retention, and any use of data for service improvement depend on the selected provider and the contract. Inclify does not make a general statement that data never serve for training. Such a guarantee must be confirmed for the specific deployment.
The narrator does not receive tools to edit the project. Its response is validated as a structured output, and an error leads to a safe fallback text.
Limitations: architecture does not replace responsibility
A deterministic calculator can also have a wrong formula, a bad baseline, or incorrect data. Repeatability makes testing easier, but it does not prove engineering correctness. That is why a data quality pre-flight is performed first, and the results are compared with the reference and the asset context.
A human is not automatically a good safeguard. They need competence, time, access to data, and a real right to object. If the procedure rewards fast approval, "human in the loop" becomes a formality. Oversight effectiveness must be measured by tests, rejected results, response time, and incident analysis.
The AI Act has complex scope and classification criteria, and obligations depend on role and use case. A building monitoring system is not automatically a high-risk system just because it uses AI. This article is not legal advice. Classification, contracts, and obligations should be assessed for the specific deployment with the right specialists.
FAQ
Can the language model determine the ALARM state?
It should not in the architecture described here. The state should come from an explicit, testable rule and approved configuration. The model may describe the exceedance and point to the procedure, but it does not change the value or the threshold. This keeps the alarm active even during provider failure and makes its basis reproducible without generative text.
What does it mean that the report works without AI?
It means that statistics, anomalies, thresholds, statuses, data quality, and the basic summary are created without calling the language model. A missing key or a timeout does not remove the information needed for response. The user loses the convenient narration, not the deterministic evidence or the alarm functions. That is the required safe degradation mode.
Does sending statistics instead of the full series solve privacy?
It reduces the data scope, but it does not close the issue. The project name, device, time, anomaly, and description may still be confidential information, and sometimes personal data when combined with other sources. Field minimisation, processing basis, contract, retention, security, and vendor assessment are still needed.
How often should the narrator be tested after deployment?
After every change of model, provider, prompt, payload schema, or validator, and periodically on a fixed regression set. Also monitor production errors and user rejections. A change in an external service can affect style or compliance even if the platform code does not change, so the version must be logged.
Does a human have to read every AI report?
The scope of review depends on risk and purpose. An auxiliary report may have a different workflow from a recommendation that affects robots or access to the asset. Still, you must define who is responsible, when review is required, and which results may remain informational only. Oversight must be proportional and operationally feasible.
How can you confirm that the data are not used for training?
Not by a general platform statement. Ask for the terms of the specific provider, service variant, region, retention settings, list of subprocessors, and a contractual clause covering input and output data. Verify again after a model or provider change. If the guarantee is required, include it in acceptance and audit criteria.
Sources and further reading
- NIST AI RMF Core - Govern, Map, Measure and Manage functions, human oversight, validation, third parties, and incident response.
- NIST AI RMF - Human-AI Interaction - human roles, interaction limits, and the risk of overreliance.
- NIST AI RMF Playbook - Measure - documenting oversight, go/no-go decisions, and independent assessment.
- ISO/IEC 42001:2023 - AI management systems - the official standard for an organisational AI management system.
- Regulation (EU) 2024/1689 - AI Act - official text, including Article 14 on human oversight for high-risk systems.
- NIST AI Risk Management Framework 1.0 - the full document on AI trustworthiness and risk management.
What next
Send the vendor the table of 13 questions and ask for artefacts before the demo. Then perform acceptance on one frozen report and in model-key-free mode. Talk to the Inclify team if you want to see this flow in a pilot project and compare a deterministic report with optional narration.