AI-300 Operationalizing Machine Learning and Generative AI Solutions Questions and Answers

Managed Identity is Azure ' s solution for eliminating stored secrets and API keys from application configurations. When a Microsoft Foundry resource or its dependent compute has a system-assigned or user-assigned managed identity, it can authenticate to Azure services such as Storage, Key Vault, and Cognitive Services using Azure Entra ID tokens that are automatically issued and rotated by Azure. There are no keys to store, rotate, or accidentally expose in code or configuration files, directly satisfying the requirement to authenticate without stored credentials. Role-Based Access Control (RBAC) is the Azure governance mechanism for scoping permissions. By assigning specific built-in roles to the managed identity or to user and group principals at the correct scope — subscription, resource group, resource, or project level — the organization can enforce least-privilege access across different projects and shared resources.

Microsoft Learn Reference Topic: Configure managed identities and RBAC for Microsoft Azure AI Foundry resources

Termination policy type: Bandit

Termination policy parameter: slack_factor

The correct early termination policy is Bandit because the requirement explicitly compares each training run against the best-performing run . Microsoft defines BanditPolicy as an early termination mechanism that stops runs that fall outside a permitted performance margin relative to the best run observed so far.

The correct parameter is slack_factor because the threshold is expressed as a percentage , not as an absolute difference in the metric. Microsoft defines slack_factor as the ratio representing the allowable distance from the best-performing run. Therefore, a 25% tolerance would be configured conceptually as:

slack_factor = 0.25

A trial whose primary metric falls outside that permitted 25% band is eligible for early termination at the configured evaluation interval. Microsoft provides the same pattern in SDK v2 examples, such as BanditPolicy(slack_factor=0.15, ...) for a 15% relative tolerance.

Median stopping compares runs with the median performance across completed or running trials rather than strictly against the current best run. Truncation selection terminates a specified percentage of the worst-performing trials, so neither matches this requirement.

The correct approach is to generate synthetic prompt-response interactions that reproduce the structure and behavioral patterns of the target task. Microsoft Foundry supports synthetic data generation specifically for cases where real production data is unavailable, scarce, sensitive, or subject to privacy and compliance restrictions. Generated examples can be converted into the structured conversational format required for fine-tuning, such as JSONL containing messages with system, user, and assistant roles.

A simulator or synthetic-data generator can construct representative conversations without exposing genuine customer interactions. This allows the team to create sufficient task-specific examples, review their quality, remove unsuitable samples, and then use the resulting dataset for supervised fine-tuning. Microsoft notes that synthetic data can preserve useful task structure while avoiding reliance on proprietary or sensitive production datasets.

Option A is incorrect because evaluation logs are primarily intended for assessment and observability and are not automatically suitable as training examples. Option C directly violates the requirement because it captures real production traffic. Option D confuses training data with observability artifacts: telemetry logs and traces describe execution behavior and diagnostics rather than producing the prompt-response examples required for fine-tuning.

Study Guide Reference: Optimize generative AI systems and model performance — synthetic data generation, supervised fine-tuning datasets, simulator-based interaction generation, privacy-preserving model customization, and training-data preparation.

When evaluating a fine-tuned model against the base model using the same evaluation dataset, the interpretation of each metric requires careful analysis. A statement that the fine-tuned model improves on the target task is True if and only if the target metric such as task-specific accuracy, F1 score, or ROUGE score shows a statistically meaningful improvement. A statement about regression on a complementary metric is True if the fine-tuned model ' s score on that metric is meaningfully lower than the base model ' s. In Microsoft Foundry ' s evaluation framework, both pre-fine-tuning and post-fine-tuning results are stored against the same experiment, enabling direct side-by-side comparison. The core principle is that improvement on the primary task is not sufficient if fine-tuning causes degradation on safety or coherence — this is called catastrophic forgetting, and the evaluation dataset is designed to detect it.

Microsoft Learn Reference Topic: Evaluate fine-tuned models in Microsoft Foundry – Compare base and fine-tuned model metrics

Box 1: Regenerate the keys of account1.

Azure Blob Storage support authentication through Account key or SAS token.

To authenticate your access to the underlying storage service, you can provide either your account key, shared access signatures (SAS) tokens, or service principal

Box 2: Update the authentication for store1.

For Azure Machine Learning studio users, several features rely on the ability to read data from a dataset; such as dataset previews, profiles and automated machine learning. For these features to work with storage behind virtual networks, use a workspace managed identity in the studio to allow Azure Machine Learning to access the storage account from outside the virtual network.

Note: Some of the studio ' s features are disabled by default in a virtual network. To re-enable these features, you must enable managed identity for storage accounts you intend to use in the studio.

The following operations are disabled by default in a virtual network:

Preview data in the studio.

For deployment failures when containers do not start, review deployment logs in Azure ML Studio or use the CLI get-logs command to surface dependency installation errors, missing files, or scoring script import failures. For authentication errors such as 401 or 403 responses, check that client applications are using valid authentication keys or bearer tokens, as mismatched or expired credentials cause auth failures. For performance degradation and slow responses, review compute instance scaling configuration, since a surge in traffic saturates a fixed number of replicas causing queuing and high latency without autoscaling. For internal server errors such as HTTP 500, these typically originate in the scoring script, so inspect the init and run functions for unhandled exceptions and verify the environment has all required packages installed.

Microsoft Learn Reference Topic: Troubleshoot online endpoint deployment failures – Azure Machine Learning

The correct parameter is image . In Azure Machine Learning Python SDK v2, the Environment entity supports defining an execution environment directly from an existing Docker image by specifying the image URI through the image parameter . Microsoft provides the explicit SDK v2 pattern:

Environment(image= " pytorch/pytorch:latest " , ...)

The resulting environment can then be registered by using ml_client.environments.create_or_update().

The image value can reference an image hosted in a supported container registry such as Docker Hub or Azure Container Registry. Azure Machine Learning then uses that image as the container environment for training or deployment workloads.

build is different: it is used when Azure Machine Learning must build a Docker image from a Docker build context , typically a directory containing a Dockerfile and related files. Microsoft specifies that image and build represent alternative environment-definition approaches.

conda_file specifies Python and Conda dependencies layered on top of a base Docker image; it does not identify the Docker image itself. properties stores arbitrary metadata and cannot define the execution container.

Therefore, to create an SDK v2 environment directly from an existing Docker image, use image .

Study Guide Reference: Design and implement an MLOps infrastructure — Azure Machine Learning environments, Python SDK v2, Docker images, reusable execution environments, and dependency management.

The first action should be to evaluate the model output . Fabrikam already has evaluation data containing manually defined expected responses, and the technical requirements explicitly state that the organization must use its existing evaluation datasets based on real data with input-output pairs . Microsoft Foundry supports evaluating a model against an existing curated dataset to establish measurable baseline performance before optimization. Evaluation results can identify weaknesses in accuracy, relevance, groundedness, safety, or other quality dimensions and provide objective evidence for subsequent changes.

This sequencing is critical because Fabrikam also requires that advanced fine-tuning be applied only when prompt engineering is insufficient . Therefore, immediately fine-tuning the model would bypass the required baseline evaluation and optimization process. Fine-tuning should be driven by demonstrated performance gaps, not applied automatically.

Deploying directly to production is also inappropriate because leadership requires low operational risk and gradual rollout. Production traffic should not be used as the first evaluation mechanism when curated evaluation data already exists. Synthetic interaction data is useful when evaluation coverage is insufficient, but Microsoft specifically positions synthetic generation as an option when an adequate test dataset is unavailable. Fabrikam already has suitable evaluation data.

Study Guide Reference: Optimize generative AI systems and model performance — evaluate model performance, benchmark against curated datasets, iterate on prompts, and apply advanced fine-tuning when justified.

Git commits on the main branch provide the immutable, ordered history that auditors need: every change is timestamped, attributed to a specific developer, and reversible via git revert — this is the audit trail and rollback mechanism required by the first requirement. Git branches allow developers to create and test multiple prompt variants in complete isolation from the production prompt on the main branch, addressing the second requirement. A developer on an experiment branch can run full evaluations without touching the production prompt. When a variant is approved, it is merged via pull request. You cannot use branches alone for audit history because branches can be deleted, and you cannot use main-branch commits alone for parallel variant comparison without disrupting the history. Git ' s branch-and-merge model provides both capabilities simultaneously.

Microsoft Learn Reference Topic: Version control for AI prompts – Git branch strategies for prompt management in Microsoft Foundry

A managed online endpoint can host multiple named deployments simultaneously, and you control what percentage of incoming traffic each deployment receives. With traffic splitting, you route a small percentage of live traffic to the new deployment while the majority continues to the proven existing model. You then monitor error rates, latency, and output quality for both deployments in real time under genuine production load. If the new model underperforms, you instantly route traffic back — no downtime, no user disruption. If it outperforms, you gradually increase its traffic share to 100%. Updating the registry version (option B) does not affect running deployments. A staging endpoint (option C) does not validate under real production load. An evaluation script (option D) is a pre-deployment step. Traffic splitting is the blue/green and canary deployment pattern that Microsoft recommends for safe production rollouts.

Microsoft Learn Reference Topic: Perform safe rollout of new model deployments using traffic splitting – Azure Machine Learning

Different query types require fundamentally different retrieval algorithms. For queries that include exact policy identifiers, keyword or BM25 search is the correct choice because BM25 scores documents based on term frequency and inverse document frequency — an exact match on a specific policy identifier such as POL-2024-HR-042 scores very highly regardless of semantic context. This is the right approach when semantic similarity is low but exact term matching is critical. For natural-language questions where keywords may not be an exact match, semantic or vector search is the correct choice because vector embeddings capture meaning rather than exact tokens, finding relevant documents even when the user ' s vocabulary differs from the document ' s terminology. Azure AI Search supports both modes through its hybrid search capability, and the correct configuration maps each query type to its optimal retrieval algorithm.

Microsoft Learn Reference Topic: Configure hybrid search in Azure AI Search – BM25 keyword search vs. semantic vector search

MLflow is the industry-standard open-source platform for experiment tracking, and Azure Machine Learning has first-class native integration with it. When you use MLflow within an Azure ML job, parameters, metrics, and artifacts are automatically logged to the run history of the AML workspace, making every run reproducible and comparable. Option A (AML job output logs) only captures console output and lacks structured parameter and metric logging. Option C (Application Insights logs) is designed for application-level telemetry, not ML experiment metadata. Option D (Azure Monitor alerts) is a reactive notification tool, not a tracking system. MLflow ' s autologging capability means that for common frameworks such as scikit-learn, XGBoost, and PyTorch, parameters and metrics are captured without a single line of custom code, directly answering Fabrikam ' s requirement for consistent experiment tracking.

Microsoft Learn Reference Topic: Track ML experiments with MLflow – Azure Machine Learning MLflow integration

Managed online endpoints with traffic splitting are the best fit because Azure Machine Learning supports multiple deployments behind a single online endpoint and allows production traffic to be distributed between them. This enables a blue-green or progressive rollout strategy : deploy the new model version alongside the current version, validate it independently, send a small percentage of production traffic to it, monitor behavior, and then gradually increase traffic. If problems occur, traffic can be redirected immediately to the previous deployment, providing a practical rollback mechanism.

This approach also aligns with Fabrikam’s requirement to favor managed services and automation because the team has limited Azure DevOps experience. Managed online endpoints handle much of the serving, scaling, security, and monitoring infrastructure automatically and support both traffic splitting and mirrored traffic for safer validation.

Option B can also implement blue-green deployment, but AKS introduces substantially more infrastructure and operational management than Fabrikam requires. Option A would require custom deployment, routing, scaling, and rollback logic. Option D is intended for asynchronous batch inference rather than staged real-time production serving.

Study Guide Reference: Implement machine learning model lifecycle and operations — managed online endpoints, multiple deployments, traffic splitting, blue-green deployment, staged rollout, monitoring, and rollback.

Imagine a hospital analytics firm with data scientists who kick off dozens of training jobs throughout the week. During peak hours, five jobs compete for the same GPU cluster and fail or queue for hours. On quiet nights, that cluster sits completely idle, burning money. Managed compute targets with autoscaling solve both problems: the cluster scales out automatically when multiple jobs arrive simultaneously and scales back to zero when idle. Option A (single shared cluster) is exactly the resource-contention problem Fabrikam already has. Option B (fixed-size cluster) wastes money during off-peak hours. Option C (dedicated per-experiment clusters) eliminates contention but is prohibitively expensive for a cost-conscious healthcare firm. Autoscaling managed compute is the cloud-native answer to variable workload demand.

Microsoft Learn Reference Topic: Azure Machine Learning compute targets – Configure autoscale for compute clusters

Domain specialization: Supervised fine-tuning

Poor response accuracy of a RAG-based solution: Apply prompt engineering

For domain specialization , Supervised Fine-Tuning (SFT) is the correct first strategy. Microsoft describes SFT as the foundational fine-tuning technique for training a model from labeled input-output pairs , and specifically identifies domain specialization as one of its principal use cases. Microsoft also recommends starting with SFT for most customization projects because it supports task specialization, instruction following, style, and domain-specific behavior. Fabrikam already possesses evaluation data containing input-output pairs, which aligns directly with the SFT data model.

For poor RAG response accuracy , the first action is prompt engineering . The case explicitly requires advanced fine-tuning only when prompt engineering is insufficient. In a RAG system, prompt engineering determines how the model interprets retrieved context, constrains answers to grounding information, handles missing evidence, and formats responses. Microsoft notes that inadequate RAG prompting can produce false or incomplete answers even when retrieval returns appropriate content.

DPO is primarily appropriate for alignment using preferred versus non-preferred responses, while RFT targets complex reward-based reasoning optimization. Neither is the initial technique for domain specialization in this requirement.

Study Guide Reference: Optimize generative AI systems and model performance — prompt engineering, RAG optimization, supervised fine-tuning, preference optimization, and model customization strategy.

In Azure Machine Learning, a workspace is the top-level resource that acts as a centralized hub for every ML activity. It stores experiments, pipelines, datasets, models, environments, and compute targets all in one place. Before you can register assets or deploy endpoints, the workspace itself must exist and be shared across the team. Fabrikam ' s core challenge is inconsistent experiment tracking, ad-hoc versioning, and no standardized deployment path — all symptoms of the absence of a single, governed workspace. Option A (registering assets) is a subsequent action only possible after the workspace exists. Option C (online endpoint) is a deployment concern, not an asset-management foundation. Option D (Foundry project) is appropriate for generative AI workloads, not for the traditional ML pipeline standardization described here. The workspace is the prerequisite that makes every other governance action possible.

Microsoft Learn Reference Topic: Azure Machine Learning workspaces – Microsoft Learn: Manage Azure Machine Learning workspaces