CT-AI ISTQB Certified Tester AI Testing Exam Questions and Answers

The syllabus describes that data pre-processing includes cleaning (e.g., fixing or removing invalid data) and transforming data (e.g., changing data types such as numbers stored as strings).

"Transformation: The format of the given data is changed… converting categorical data into numerical data, changing image formats…"

(Reference: ISTQB CT-AI Syllabus v1.0, Section 4.1.1, Page 34 of 99)

The ISTQB CT-AI syllabus explains in Section3.2 – Functional Performance Criteria of ML Modelsthat accuracy becomes unreliable whenclass imbalanceexists. In fraud detection, more than 99% of transactions are non-fraudulent, meaning the dataset is extremely imbalanced. Because accuracy counts all correct non-fraudulent classifications, it will appear artificially high, even if the fraud detection performance is poor. Therefore, accuracy is not suitable for evaluating fraud detection systems.

The syllabus further explains thatsensitivity (recall)captures the proportion of correctly identified fraudulent cases. This metric is important, as missing fraudulent events can cause high financial loss. However, the client also stresses thatlegitimate transactions must be correctly identified, meaningfalse positives must be minimizedto maintain customer satisfaction.

TheF1 score, defined as the harmonic mean ofprecision and recall, balances both:

Precision protects legitimate customers by minimizing false alarms.

Recall ensures fraudulent transactions are detected.

Section 3.2 emphasizes that when both false positives and false negatives have significant consequences, and the data is highly imbalanced,F1 is the most appropriate metricbecause it reflects the combined importance of detecting fraud while avoiding unnecessary alerts. Thus,Option Cis the correct choice.

Decision trees are a foundational algorithm used in supervised machine learning. The syllabus describes:

"A decision tree is a tree-like ML model whose nodes represent decisions and whose branches represent possible outcomes."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 3.4)

The syllabus describes automation bias as:

"A type of bias caused by a person favoring the recommendations of an automated decision-making system over other sources."

This is also known as complacency bias, where testers accept automated system outputs without questioning them.

(Reference: ISTQB CT-AI Syllabus v1.0, Glossary, Page 89 of 99)

The syllabus describes exploratory testing as especially useful when there are poor specifications or test oracle problems:

“Exploratory testing is especially useful when there are poor specifications or test oracle problems, which is often the case for AI-based systems.”

(Reference: ISTQB CT-AI Syllabus v1.0, Section 9.6, page 70 of 99)

 AI-Based Autonomous Functions:An AI-based autonomous system is one that can respond to its environment without human intervention. The other options either involve human decisions or do not use AI at all.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Sections on Autonomy and Testing Autonomous AI-Based Systems​​​​.

The syllabus explicitly mentions challenges of handling personal data and ensuring privacy when testing AI-based systems:

"The management of personal data and sensitive data is often a concern during testing, as testing typically requires realistic data and it is difficult to fully anonymize data."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 7.3, page 52 of 99)

The syllabus defines reward hacking as:

"Reward hacking can result from an AI-based system achieving a specified goal by using a 'clever' or 'easy' solution that perverts the spirit of the designer’s intent."

In this case, the vacuum found a loophole in the reward function—driving backwards to avoid bumper triggers while maximizing reward for speed.

(Reference: ISTQB CT-AI Syllabus v1.0, Section 2.6, page 24 of 99)

A/B testing, also known as split testing, is a method used to compare two versions of a product or system to determine which one performs better. It is widely used in web development, marketing, and machine learning to optimize user experiences and model performance. Here’s why option C is the least descriptive of an A/B testing scenario:

Understanding A/B Testing:

In A/B testing, two versions (A and B) of a system or feature are tested against each other. The objective is to measure which version performs better based on predefined metrics such as user engagement, conversion rates, or other performance indicators.

Application in Machine Learning:

In ML systems, A/B testing might involve comparing two different models, algorithms, or system configurations on the same set of data to observe which yields better results.

Why Option C is the Least Descriptive:

Option C describes comparing the performance of an ML system on two different input datasets. This scenario focuses on the input data variation rather than the comparison of system versions or features, which is the essence of A/B testing. A/B testing typically involves a controlled experiment with two versions being tested under the same conditions, not different datasets.

Clarifying the Other Options:

A. A comparison of two different websites for the same company to observe from a user acceptance perspective: This is a classic example of A/B testing where two versions of a website are compared.

B. A comparison of two different offers in a recommendation system to decide on the more effective offer for the same users: This is another example of A/B testing in a recommendation system.

D. A comparison of the performance of two different ML implementations on the same input data: This fits the A/B testing model where two implementations are compared under the same conditions.

A. Black box attacks based on adversarial examples create an exact duplicate model of the original.

Black box attacks do not create an exact duplicate model. Instead, they exploit the model by querying it and using the outputs to craft adversarial examples without knowledge of the internal workings.

B. These attack examples cause a model to predict the correct class with slightly less accuracy even though they look like the original image.

Adversarial examples typically cause the model to predict the incorrect class rather than just reducing accuracy. These examples are designed to be visually indistinguishable from the original image but lead to incorrect classifications.

C. These attacks can't be prevented by retraining the model with these examples augmented to the training data.

This statement is incorrect because retraining the model with adversarial examples included in the training data can help the model learn to resist such attacks, a technique known as adversarial training.

D. These examples are model specific and are not likely to cause another model trained on the same task to fail.

Adversarial examples are often model-specific, meaning that they exploit the specific weaknesses of a particular model. While some adversarial examples might transfer between models, many are tailored to the specific model they were generated for and may not affect other models trained on the same task.

Therefore, the correct answer isDbecause adversarial examples are typically model-specific and may not cause another model trained on the same task to fail​​.

Section4.3 – Test Levels for AI Systemsclearly defines ML model testing as the level at which testers evaluate whether an ML model fulfills itsfunctional performance criteria, including accuracy, precision, recall, F1, robustness, stability, and fairness. Therefore, Option C is the correct and syllabus-aligned statement.

Option A is incorrect because input data testing focuses onvalidity and correctness of data entering the model, not interactions with all system components. Option B is incorrect: acceptance testing in the syllabus focuses primarily onbusiness and stakeholder requirements, not specifically explainability. Explainability testing may occur at multiple levels depending on context. Option D is also incorrect because API testing belongs tointegration testing, not system testing, even when AI is consumed as a service.

Thus,Option Cis the only statement that precisely matches syllabus definitions.

Section5.3 – AI Support for Defect Predictionof the ISTQB CT-AI syllabus explains that AI-based defect prediction models rely onhistorical patterns, including past defects, code behavior, and similar system configurations. ML models trained on prior defect data can identifycomponents likely to contain defectswhen new changes resemble previous defect-inducing patterns. This directly supports OptionA, which states that defect prediction is most effective when based on previous similar constellations.

Option B is incorrect: ML can predictwhich componentsare likely to fail, not only whether defects exist. Option C is incomplete; code metrics help, but defect prediction relies onmanycontextual features (historical defects, code churn, commit frequency, etc.). Option D is wrong because defect prediction isnotbased on formal principles and typically requiresmany features, not just a few.

Thus,Option Ais the correct and syllabus-consistent answer.

 Type of Testing for Stock-Price Prediction Models:Concept drift refers to the change in the statistical properties of the target variable over time. Severe weather storms causing massive losses in coffee production and affecting stock prices would require testing for concept drift to ensure that the model adapts to new patterns in data over time.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Section 7.6 Testing for Concept Drift, which explains the need to test for concept drift in models that might be affected by changing external factors​​.

The ISTQB CT-AI syllabus indicates inSection 5.2 – AI for Testingthat AI-generated test cases may not come withpredefined expected results. This is because test-case generation methods—such as evolutionary algorithms, reinforcement learning, or clustering-based sampling—produceinputs, but the tester must still determine the correct outputs. Therefore, OptionCis correct: expected results may not be available, especially when AI produces novel or previously unseen input combinations.

Option A is incorrect: AI-based test generation can improve coverage by exploring large input spaces. Option B is incorrect because the need for oracles does not depend on whether the test case is AI-generated. Option D is incorrect because the syllabus allows using a model as an oracle inback-to-back testing, provided limitations are understood.

Therefore,Option Cis the correct syllabus-aligned statement.

The ISTQB CT-AI syllabus emphasizes that testing AI-based systems requirescross-functional collaborationandexperience-based testingwhen parts of the team lack domain knowledge. In this scenario, the ML expert understands ML and dataset preparation but lacks knowledge ofcamera system behavior, the device’s operational data pipeline, and end-user workflows. The remainder of the team understands the domain and system testing but not ML. Section4.4 – Human Factors and AI Testingand4.3 – System Testing of AI Componentshighlight that when domain understanding is unevenly distributed,experience-based testing conducted by the full team(testers, developers, domain experts) is the most effective approach. This ensures that AI outputs align with actual user expectations and system behavior. OptionCaligns exactly with this principle.

Option A is too limited and does not address the need to validate ML integration. Option B is incorrect because reusing old test cases overlooks AI-specific risks in the operating data pipeline. Option D is useful but focuses only on data representativeness, not system-level user validation. Therefore,Option Cis the best, syllabus-aligned test approach.

 Testing Autonomy:Testing for human handover when it should not be relinquishing control is the least appropriate because it contradicts the very definition of autonomous systems. The other tests are relevant to ensuring smooth operation and transitions between human and AI control.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Sections on Testing Autonomous AI-Based Systems and Testing for Human-AI Interaction​​​​. ​

The ISTQB CT-AI syllabus distinguishes between several types of bias relevant in AI testing, includingsample bias,algorithmic bias, andinappropriate bias. In Section3.3 – Bias in AI-Based Systems, the syllabus stresses the importance of identifying biases that originate fromtraining data,model development, anddecision logic. Sample bias occurs when the training data does not adequately represent the population; algorithmic bias arises when the model produces systematically skewed results due to learned patterns; inappropriate bias involves ethically or socially problematic distortions in the outcomes. All three of these bias types directly affect theoutputs of the AI modeland are therefore highly relevant when testing an industrial inspection system intended to reliably detect defects. These biases can lead to defective items being missed or false alarms being raised, which impacts quality assurance significantly .

Automation bias, however, is fundamentally different. It refers to ahuman cognitive bias, where users (e.g., inspectors) overly trust or rely on the AI system’s output. While important in user-interaction testing, it isnota biaswithin the ML model itself. Since the question asks which bias isleast important when testing the model, automation bias can be legitimately deprioritized duringmodel-level testing. Therefore, OptionBis correct.

The ISTQB CT-AI syllabus lists several characteristics that make it difficult to ensure safety in AI-based systems. Section2.8 – Safety and AIexplicitly names the characteristics that complicate safety assurance:complexity,non-determinism,probabilistic behavior,self-learning,lack of transparency, andlack of robustness. Among these,complexityis a core challenge because modern AI systems—particularly those using deep learning—have highly non-linear behavior, large numbers of parameters, and intricate interactions that are hard to predict.

Option B (Complexity) directly aligns with the syllabus and is therefore correct.

Option A (Determinism) is the opposite of AI behavior; AI is oftennon-deterministic, and determinism doesnotmake systems unsafe. Option C (Interpretability) does impact trust and explainability, but the syllabus positions it as a transparency challenge, not the primary difficulty in ensuring safety. Option D (Robustness) is a desired quality, not a reason safety is hard; alackof robustness would be a challenge, not robustness itself.

Thus,complexitybest reflects the syllabus’ explicit safety-related difficulty.

The ISTQB CT-AI syllabus explains in Section1.6 – Machine Learning Approachesthatsupervised learningis appropriate when labeled data exists and the goal is to predict an output based on known historical examples. Predicting a customer’spurchase probabilityis aclassificationtask when the output corresponds to discrete categories such as“likely to purchase”vs.“not likely to purchase.”The syllabus gives similar examples in describing classification as the process of assigning instances to predefined classes based on learned patterns in labeled data. Because the retail company wants to determine whether a customer will make a purchase based on marketing actions, classification is the most appropriate choice .

Option A (regression) predicts continuous numeric values and is less suitable because the task centers oncategorical likelihood, not estimating exact monetary values. Option C (unsupervised learning) is used when labels are not available—here, the company has labeled purchase histories. Option D (reinforcement learning) requires an interactive environment with reward-driven behavior, which is not applicable to this scenario.

Thus,supervised learning (classification)is the most suitable approach according to the syllabus.

According to the ISTQB CT-AI syllabus,metamorphic testingworks by applyingmetamorphic relations (MRs): predictable input transformations that should lead to predictable output changes. From the initial test data, clear relations emerge for defensive driving scoring. The score increases when:

v_mean decreases,

a_pos decreases,

a_neg becomes less negative,and decreases when the opposite occurs.

A valid metamorphic follow-up test must modify inputs in a direction consistent with at least one MR while keeping the expected output direction predictable.

Test Alowers v_mean compared to Test 1, with similar acceleration values. This directly satisfies the MR that lower speed → higher score.

Test Cincreases both acceleration and braking intensity compared to Test 2, making the reduced score range (30–70) consistent with more aggressive driving.

Test Dmodifies acceleration and braking magnitudes in ways consistent with Test 3’s defensive-driving scoring boundaries.

Test B, however, changes multiple variables in contradictory directions:

v_mean increases (worse)

a_pos increases (worse)

a_neg becomes less negative (better)

Because these changes conflict,the expected score trend becomes unpredictable, violating the premise of a metamorphic follow-up test.

ThusTest B cannot be considered a metamorphic follow-up, which makesOption Ccorrect.

The syllabus explains that ML models can be used to analyze reported defects and suggest which developers are best suited to fix them based on historical data about defect assignment and resolution speed:

"Assignment: ML models can suggest which developers are best suited to fix particular defects, based on the defect content and previous developer assignments."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 11.2, page 78 of 99)

AI test environments have several unique characteristics that differentiate them from traditional test environments. Let’s evaluate each option:

A. Challenges resulting from low accuracy of the models.

Low accuracy is a common challenge in AI systems, especially during initial development and training phases. Ensuring the model performs accurately in varied and unpredictable scenarios is a critical aspect of AI testing.

B. The challenge of mimicking undefined scenarios generated due to self-learning.

AI systems, particularly those that involve machine learning, can generate undefined or unexpected scenarios due to their self-learning capabilities. Mimicking and testing these scenarios is a unique challenge in AI environments.

C. The challenge of providing explainability to the decisions made by the system.

Explainability, or the ability to understand and articulate how an AI system arrives at its decisions, is a significant and unique challenge in AI testing. This is crucial for trust and transparency in AI systems.

D. Challenges in the creation of scenarios of human handover for autonomous systems.

While important, the creation of scenarios for human handover in autonomous systems is not a characteristic unique to AI test environments. It is more related to the operational and deployment challenges of autonomous systems rather than the intrinsic technology-related characteristics of AI.

Given the above points, optionDis the correct answer because it describes a challenge related to operational deployment rather than a technology-related characteristic unique to AI test environments.

The syllabus states:

“When testing probabilistic and non-deterministic systems, the same input may produce different outputs. Tests need to be run several times to produce statistically valid test results, ensuring that an appropriate number of answers are accurate.”

(Reference: ISTQB CT-AI Syllabus v1.0, Section 8.4, page 58 of 99)

The ISTQB CT-AI syllabus (Section3.2 – Model Evaluation) specifies the correct usage oftraining,validation, andtestdatasets. It emphasizes that thetest dataset must be representative of the real operational dataand must beequivalent in distribution to the training and validation sets, ensuring a fair and unbiased evaluation. Option D precisely matches this requirement.

Option A contradicts the syllabus because validation and test sets servedifferent purposes: validation is for tuning, test is for final evaluation. Combining them undermines the reliability of results. Option B is incorrect because even with limited data, the syllabus recommends maintaining a test set or using techniques such ascross-validationrather than eliminating testing. Option C is wrong because equal distribution (33/33/33) isnot recommended; typically, the training set is much larger (e.g., 70–80%).

Thus, OptionDis the only statement aligned with the syllabus' guidance.

The process of developing a machine learning model typically involves the use of three types of datasets:

Training Data:This is used to train the model, i.e., to learn the patterns and relationships in the data.

Validation Data:This is used to tune the model’s hyperparameters and to prevent overfitting during the training process.

Test Data:This is used to evaluate the final model’s performance and to estimate how it will perform on unseen data.

Let's analyze each option:

A. Training data - validation data - test data

This option correctly includes all three types of datasets used in the process of creating and validating a model. The training data is used for learning, validation data for tuning, and test data for final evaluation.

B. Training data - validation data

This option misses the test data, which is crucial for evaluating the model’s performance on unseen data after the training and validation phases.

C. Training data - test data

This option misses the validation data, which is important for tuning the model and preventing overfitting during training.

D. Validation data - test data

This option misses the training data, which is essential for the initial learning phase of the model.

Therefore, the correct answer isAbecause it includes all necessary datasets used during the process of learning and creating the model: training, validation, and test data​​.

The syllabus mentions that experience-based testing and EDA are effective for detecting biases:

“Experience-based testing can be used to verify that the training dataset is operationally relevant and identify potential sources of bias. EDA is also useful for exploring the data and understanding any relationships that might lead to bias in the model.”

(Reference: ISTQB CT-AI Syllabus v1.0, Section 8.3, page 58 of 99)

The syllabus states that while pairwise testing is effective at finding defects by reducing the number of test cases needed, the resulting test suite can still be extensive and require automation:

"Even the use of pairwise testing can result in extensive test suites… automation and virtual test environments often become necessary to allow the required tests to be run."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 9.2, Page 67 of 99)

Clustering is a type of machine learning technique used to group similar data points into clusters. It is a key concept in unsupervised learning, where the algorithm tries to find patterns or groupings in data without prior knowledge of output classes. Let's analyze each option:

A. Clustering is classification of a continuous quantity.

This is incorrect. Classification typically involves discrete categories, whereas clustering involves grouping similar data points. Classification of continuous quantities is generally referred to as regression.

B. Clustering is supervised learning.

This is incorrect. Clustering is an unsupervised learning technique because it does not rely on labeled data.

C. Clustering is done without prior knowledge of output classes.

This is correct. In clustering, the algorithm groups data points into clusters without any prior knowledge of the classes. It discovers the inherent structure in the data.

D. Clustering requires you to know the classes.

This is incorrect. Clustering does not require prior knowledge of classes. Instead, it aims to identify and form the classes or groups based on the data itself.

Therefore, the correct answer isCbecause clustering is an unsupervised learning technique done without prior knowledge of output classes​​.

The syllabus lists multiple reasons for mislabelled data, including the lack of domain knowledge:

"Lack of required domain knowledge may lead to incorrect labelling."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 4.5.2, page 38 of 99)

TheISTQB Certified Tester AI Testing Syllabus v1.0explicitly categorizes different AI implementation technologies in Section1.4 – AI Technologies. Within this section, AI methods are grouped into categories, one of which is“Reasoning techniques.”These reasoning techniques includerule engines, deductive classifiers, case-based reasoning, and procedural reasoning. Because deductive classifiers are directly listed under this set of reasoning approaches, they are recognized as a reasoning-based AI technology.

Reasoning techniques differ from machine learning approaches because they rely onstructured, predefined rules or logicto reach conclusions. Deductive classifiers use logical inference and symbolic reasoning to classify inputs by applying encoded knowledge. This makes them fundamentally different from statistical or data-driven ML algorithms.

The other options—Linear regression,Random Forest, andGenetic algorithms—are listed by the syllabus asmachine learning techniques, not reasoning methods. Linear regression performs numerical prediction, Random Forest is an ensemble decision-tree ML model, and genetic algorithms are optimization-based ML approaches inspired by evolutionary processes. None of these involve symbolic logical deduction.

Thus, based on the authoritative definitions in the syllabus,Deductive classifiers (Option A)is the only technology classified as a reasoning technique.

A. Feature selection

Feature selection is the process of selecting the most relevant features from the data. While important, it is not directly about handling excess data.

B. Data sampling

Data sampling involves selecting a representative subset of the data for training. When there is more data than needed, sampling can be used to create a manageable dataset that maintains the statistical properties of the full dataset.

C. Data labeling

Data labeling involves annotating data for supervised learning. It is necessary for training models but does not address the issue of having excess data.

D. Data augmentation

Data augmentation is used to increase the size of the training dataset by creating modified versions of existing data. It is useful when there is insufficient data, not when there is excess data.

Therefore, the correct answer isBbecause data sampling is the most relevant activity when dealing with an excess amount of data for training​​.

The syllabus defines back-to-back testing as a method to compare a modified AI system to the previous version, which is ideal in this scenario:

"Back-to-back testing is performed by comparing the outputs of two systems that are supposed to provide the same outputs, one being a known and trusted system and the other being the test system. This approach can be used to test ML systems after re-training to verify that improvements have not introduced regressions."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 9.3, page 67 of 99)

The ISTQB CT-AI syllabus describes severaldata quality aspectsthat affect ML performance. In Section2.2 – Data Preparation, it explains that datasets may suffer from issues such asincomplete data,irrelevant data,incorrect data,unbalanced data, or data lacking preprocessing. “Incomplete data” means thatportions of the required data are missing, often because some time periods, records, or sources were not captured. This aligns exactly with Option A, which correctly identifies missing intervals as incomplete data.

Option B is incorrect: “data not preprocessed” refers to data that has not undergone normalization, cleaning, or transformation—not data recorded incorrectly. Option C is wrong because irrelevant datadoesnegatively affect ML models by introducing noise and unnecessary features. The syllabus explicitly states that including irrelevant features can degrade model learning. Option D is incorrect: “unbalanced data” relates todisproportionate class distribution, not recency or freshness of data.

Thus, OptionAis the only statement that correctly matches the syllabus definition of this data quality aspect.