Requirements Engineering for AI: Beyond User Stories

Traditional software requirements engineering centers on user stories with deterministic acceptance criteria: "Given X, when Y, then Z." This works well for rule-based systems where inputs map predictably to outputs.

AI systems break these assumptions. Machine learning models produce probabilistic outputs. Performance depends on data quality and distribution. Edge cases are infinite. Stakeholders often struggle to articulate what "good enough" means for inherently imperfect systems.

Requirements practices designed for deterministic software fail when applied to AI. Here's why—and what to do instead.

1. Probabilistic Outputs Defy Binary Acceptance Criteria

Traditional user story for deterministic system:

This works because database lookups are deterministic: email either matches or doesn't.

Attempting same pattern for AI system creates problems:

These criteria are impossible to satisfy. No fraud detection model achieves 100% precision and recall. Reality requires accepting trade-offs between false positives and false negatives.

2. Data Quality Is a First-Class Requirement

Traditional software treats data as input to validate, not as a constraint on system feasibility. AI systems are fundamentally limited by training data quality and availability.

Common data-related AI project failures:

• Insufficient volume: Need 10,000 labeled examples, have 500
• Distribution mismatch: Training data doesn't represent production scenarios
• Label quality: Inconsistent annotations make learning impossible
• Protected characteristics: Data contains bias that will perpetuate in model outputs

Data requirements must be surfaced and validated early, not discovered months into development when retraining is costly.

3. Performance Is Multidimensional

Traditional software performance typically focuses on speed, throughput, and resource utilization. AI systems add accuracy, fairness, robustness, and explainability—often with inherent trade-offs.

Example trade-offs in fraud detection:

• Higher sensitivity (catch more fraud) → More false positives (annoy legitimate customers)
• Higher specificity (fewer false positives) → More false negatives (miss fraud)
• Higher accuracy overall → May perform poorly on minority groups (fairness issues)
• Better explainability (simpler model) → Lower overall accuracy

Requirements must specify how to navigate these trade-offs, not simply demand maximum performance across all dimensions.

4. "Done" Is a Moving Target

Traditional software reaches "done" when functional requirements are satisfied and bugs are fixed. AI systems require continuous monitoring and retraining as data distributions shift.

Model performance degrades over time due to:

• Concept drift: Underlying patterns change (customer behavior evolves, fraud tactics adapt)
• Data drift: Input distribution shifts (new product lines, different customer segments)
• Label drift: Definition of correct outputs changes (regulatory updates, policy changes)

Requirements must address ongoing model maintenance, not just initial deployment.

Effective AI requirements engineering requires four components beyond traditional functional specs:

1. Performance Baselines & Targets

Replace binary acceptance criteria with measurable performance baselines and targets across multiple dimensions:

Baseline: Current state performance without AI

• Fraud detection: Manual review catches 78% of fraud
• False positive rate: 5% of legitimate transactions flagged
• Review time: 45 minutes per flagged transaction

Minimum Viable Performance: Threshold below which AI isn't useful

• Must catch at least 85% of fraud (beat baseline)
• False positive rate under 8% (acceptable user friction increase)
• Prediction latency under 100ms (real-time requirements)

Target Performance: Ideal but not mandatory

• Catch 95% of fraud
• False positive rate under 3%
• Explainable predictions for analyst review

This framework acknowledges reality: AI systems improve incrementally. Ship when minimum viable performance is reached, continue improving toward targets.

2. Data Requirements Specification

Treat data as a first-class requirement with explicit quality and availability criteria:

Volume Requirements:

• Minimum: 5,000 labeled fraud examples + 50,000 legitimate transactions
• Target: 20,000 fraud examples for better recall on rare fraud patterns

Quality Requirements:

• Label consistency: Inter-annotator agreement ≥ 90%
• Completeness: Missing values in critical fields < 2%
• Representativeness: Dataset covers all transaction types in production

Bias & Fairness Requirements:

• Performance parity: False positive rate within 5% across customer demographic groups
• Representativeness: Dataset includes sufficient examples from all customer segments

Validate data requirements before committing to AI approach. If data doesn't exist, requirements shift to data collection or alternative solutions.

3. Operational Monitoring Requirements

Define monitoring and alerting criteria for production AI systems:

Performance Monitoring:

• Track precision, recall, F1 score daily
• Alert if performance drops >5% week-over-week
• Alert if prediction latency exceeds 200ms (2x target)

Data Monitoring:

• Track input feature distributions
• Alert if feature distributions shift >2 standard deviations from training data
• Alert if missing value rates increase >50% relative to baseline

Fairness Monitoring:

• Track false positive/negative rates by customer demographic groups
• Alert if disparate impact exceeds 10% between groups

Monitoring requirements ensure teams detect model degradation before it causes business harm.

4. Human-AI Interaction Requirements

AI rarely operates in isolation. Specify how humans will interact with AI outputs:

Override Capability:

• Analysts can override AI fraud predictions with documented rationale
• Override data feeds back into model retraining pipeline

Explainability:

• For flagged transactions, system shows top 3 contributing factors
• Explanations understandable to non-technical fraud analysts

Escalation Paths:

• High-risk predictions (>$10K transactions) auto-escalate to senior analysts
• System tracks prediction confidence; low-confidence cases get human review

Human-AI interaction requirements prevent AI from becoming a black box that users can't trust or effectively use.

1. Start with Problem, Not Solution

Teams often jump to "We need a machine learning model to..." without validating whether ML is the right approach.

Bad Requirements Prompt:

"Build a deep learning model to predict customer churn."

Good Requirements Prompt:

"We're losing 15% of customers annually. We want to identify at-risk customers early enough to intervene. What signals indicate a customer is likely to churn, and how confident do we need to be before taking action?"

Starting with the problem allows exploration of multiple solutions: rule-based systems, simpler statistical models, or non-technical interventions might be more appropriate.

2. Co-Create Performance Trade-Off Boundaries

Don't ask stakeholders to specify exact precision/recall targets upfront—they can't. Instead, explore trade-offs through scenarios:

Scenario A:

"The model catches 95% of fraud but flags 10% of legitimate transactions for review. Customer service receives 500 complaints daily from users whose transactions were incorrectly blocked."

Scenario B:

"The model catches 85% of fraud and flags 3% of legitimate transactions. Customer complaints drop to 150 daily, but the company loses an additional $50K monthly to undetected fraud."

Stakeholders can evaluate concrete scenarios more effectively than abstract metrics. Document which trade-offs are acceptable, which are not, and where flexibility exists.

3. Validate Data Availability Early

Before committing to AI approach, verify data requirements can be satisfied:

Data Discovery Questions:

• What data exists today that could be used for training?
• How is that data currently labeled or structured?
• What data quality issues exist (missing values, inconsistencies)?
• What data *doesn't* exist that would be valuable?
• How difficult/costly would it be to collect missing data?

If required data doesn't exist and can't be obtained, requirements must shift to data collection or alternative approaches.

4. Define Continuous Improvement Process

AI systems require ongoing refinement. Requirements should specify how improvement happens:

• Retraining cadence: Model retrained monthly with new data
• Performance review: Quarterly review of production metrics and fairness indicators
• Feedback loops: Analyst corrections and overrides incorporated into next training cycle
• Sunset criteria: If performance drops below minimum viable threshold for two consecutive months, escalate to project sponsor

1. Accepting Vague Success Criteria

Anti-Pattern:

"The AI should be highly accurate."

Better:

"Minimum 85% precision and 80% recall on held-out test set representative of production distribution. Performance parity (within 5%) across customer demographic groups."

2. Ignoring Negative Cases

Anti-Pattern:

"Model should identify all high-risk transactions."

Better:

"Model should identify 90% of high-risk transactions while maintaining false positive rate under 5%. Low-risk transactions should not be unnecessarily delayed."

3. Deferring Data Requirements

Anti-Pattern:

"We'll figure out the data situation once we start building."

Better:

"Before committing to ML approach, validate that we have 10,000 labeled examples with inter-annotator agreement ≥ 85%. If not, first milestone is data collection."

4. Treating AI as Set-and-Forget

Anti-Pattern:

"Once deployed, the model is done and we move to the next project."

Better:

"Model requires monthly retraining and quarterly performance reviews. Budget includes ongoing maintenance: 0.25 FTE data engineer, 0.25 FTE ML engineer, 0.1 FTE product owner."

AI requirements engineering differs fundamentally from traditional software requirements because AI systems are:

• Probabilistic, not deterministic
• Data-dependent, not just logic-dependent
• Multidimensional in performance, not binary pass/fail
• Continuously evolving, not "done" after deployment

Successful AI requirements practices require:

• Performance baselines and targets across multiple dimensions
• Explicit data quality and availability requirements
• Operational monitoring and alerting specifications
• Human-AI interaction and explainability requirements

Organizations that adapt requirements engineering practices to AI characteristics avoid common pitfalls: vague success criteria, data surprises, unacceptable trade-offs, and operational failures.

Requirements engineering for AI is harder than traditional software—but getting it right is the difference between AI systems that deliver value and expensive failures.