Prerequisites

Before configuring monitoring, ensure the following:

At least one AI system has been classified in your Aikraft workspace. Monitoring is always scoped to a specific classified system; you cannot enable monitoring for a system that has not completed the classification questionnaire.
For MLOps integrations (MLflow, Weights & Biases), you will need an API key or service account credentials from those platforms. See Integrations for setup instructions.
Post-market surveillance monitoring (as required under EU AI Act Article 72) is only available on the Pro plan and above. Starter plan users can access drift alerts and incident logging.

What Aikraft Monitors

Aikraft’s monitoring module tracks four categories of compliance-relevant signals:

1. Model Drift

Statistical drift in model inputs and outputs over time. Aikraft compares the distribution of live production data against the baseline established at classification time. Supported drift metrics include:

Data drift: Population Stability Index (PSI) on input features
Concept drift: change in the relationship between inputs and predicted outputs
Prediction drift: shift in the distribution of model output scores or classes

2. Data Changes

Changes to the upstream data pipelines feeding the system, including schema changes, new data sources, and significant changes in null rates or cardinality for key features. These are relevant because training/validation data characteristics must be documented under Annex IV Section 4.

3. Incident Reports

A structured log for recording incidents involving the AI system — unexpected outputs, user complaints, near-misses, and actual harms. Under EU AI Act Article 73, providers of high-risk AI systems must report serious incidents to the relevant national market surveillance authority. Aikraft’s incident log is designed to support that reporting requirement.

4. Regulatory Updates

Aikraft’s legal team tracks updates to the EU AI Act, its delegated and implementing acts, published harmonised standards, and guidance from the European AI Office. When a change is relevant to one of your classified systems, it is surfaced as a monitoring alert with a plain-language summary and a link to the source document.

Connecting MLOps Tools

MLflow

In your Aikraft workspace, go to Settings > Integrations > MLflow.
Enter your MLflow Tracking Server URI (e.g., https://mlflow.yourcompany.com).
Enter a service account token with read access to the relevant experiment and model registry.
Select the registered model name to link to your Aikraft system.
Click Test Connection, then Save.

Once connected, Aikraft polls the MLflow model registry every hour. When a new model version is registered, Aikraft records the version, run ID, and key metrics in the monitoring timeline and may trigger re-documentation if the change is significant.

Weights & Biases

Go to Settings > Integrations > Weights & Biases.
Enter your W&B API key and the entity/project path (e.g., myorg/candidate-scoring).
Select which runs to track: All runs or Production-tagged runs only (recommended).
Click Save.

W&B experiment metrics are pulled into the monitoring timeline and surfaced alongside drift alerts, giving you a unified view of model performance and compliance health.

Both MLflow and Weights & Biases integrations are available on Pro plans and above. See Integrations.

Setting Alert Thresholds

Navigate to Monitor > [System Name] > Alert Settings to configure thresholds for each signal type.

Signal	Default threshold	Configurable range
Data drift (PSI)	> 0.2 (warning), > 0.25 (critical)	0.05 – 0.5
Prediction drift	> 10% shift in output distribution	5% – 30%
Input schema change	Any change triggers alert	On/Off
Days since last human review	90 days	30 – 365 days
Regulatory update relevance	Medium and above	Low / Medium / High

For each alert, you can configure:

Notification channel: email, Slack, or both (see Integrations for Slack setup)
Severity level: Info, Warning, or Critical
Recipients: any team members in your workspace

Click Save Thresholds when done. Changes take effect on the next monitoring cycle (hourly).

The Incident Log

The incident log is accessible under Monitor > [System Name] > Incidents. To record a new incident:

Click Log Incident.
Fill in the incident form:
- Date and time of occurrence
- Description: what happened, what the system output was, and what the expected output should have been
- Severity: Minor, Serious, or Critical (aligned with EU AI Act Article 73 definitions)
- Affected users or groups: approximate number and nature
- Immediate action taken: steps already taken to mitigate
Click Save. The incident is added to the log with your username and a timestamp.

Serious and Critical incidents generate a draft regulatory notification report, pre-filled with information from the incident form and your system’s technical documentation. This report must be reviewed by a qualified person before submission to the national authority.

Post-Market Surveillance Under Article 72

EU AI Act Article 72 requires providers of high-risk AI systems to operate a post-market surveillance system that is proportionate to the risk and the scale of deployment. In practice, this means:

Actively collecting and reviewing data about system performance in production
Reviewing serious incidents and near-misses
Updating technical documentation and risk assessments when necessary
Reporting to relevant authorities within defined timeframes (15 days for serious incidents, 2 days for incidents causing death or serious injury)

Aikraft’s monitoring module is designed to satisfy Article 72 requirements on Pro plans and above. The Surveillance Report (available under Monitor > Surveillance Report) generates a quarterly summary of monitoring activity, incident log entries, drift events, and any corrective actions taken. This report can be exported as PDF for inclusion in your conformity assessment file.

Notification Settings

To manage how you receive monitoring alerts:

Go to Settings > Notifications.
Toggle each alert type on or off for email and Slack.
Set a digest frequency if you prefer batched alerts (Daily Digest or Weekly Digest) rather than real-time notifications.
Configure quiet hours to suppress non-critical alerts outside business hours.

Individual team members can customise their own notification preferences under Profile > Notification Preferences without affecting the workspace-level settings.