DP-750 Implementing Data Engineering Solutions Using Azure Databricks Questions and Answers

The correct answer is A. Photon is Azure Databricks ' native vectorized query engine, written in C++, designed to accelerate data ingestion and SQL-heavy workloads significantly over the standard Spark JVM path. Enabling it on a job compute cluster directly addresses Contoso ' s requirement for ' fast and consistent performance for BI workloads ' and ' production ingestion workloads that can scale automatically during telemetry spikes. '

Photon integrates transparently — no code changes are needed — and pairs well with autoscaling job clusters to handle the bursty 40,000-sensor telemetry load.

Option B contradicts the isolation requirement: Contoso explicitly needs production and development separated, not merged onto shared compute. Option C with a fixed large node gives peak capacity at all times, driving up costs even during quiet periods. Option D disabling autoscaling is the opposite of what ' s needed — telemetry spikes require elastic scaling, not a locked node count.

The right ingestion tool depends on the source characteristics:

File-based telemetry and maintenance data → Auto Loader (cloudFiles). It monitors ADLS Gen2 for new arrivals, handles schema inference and evolution for the frequent schema drift, and requires minimal operational effort.

Real-time telemetry from Event Hubs → Spark Structured Streaming with the azure-eventhubs-spark connector. This provides exactly-once semantics and checkpoint-based recovery, satisfying ' resume processing after failures without reprocessing. '

Structured maintenance data from PostgreSQL → JDBC connector. Databricks supports direct JDBC reads from relational databases with pushdown predicates.

Daily CSV ERP extracts (50–100 GB) → COPY INTO or Auto Loader. Both support idempotent incremental batch loading into Delta tables with minimal code.

The correct answer is D. Breaking the pipeline into separate tasks for ingestion, cleansing, and curation is the foundation of well-designed Lakeflow Jobs pipelines. Each task should own one responsibility — when a task does too much, debugging a failure becomes a hunt through unrelated code, and retry logic becomes expensive because you re-execute work that already succeeded.

Contoso ' s planned changes explicitly call for ' a clear execution order and dependencies ' and ' orchestrate multi-step ingestion and transformation workflows. ' Separate tasks map directly to those goals: Lakeflow Jobs tracks each task ' s status independently, so if cleansing fails, ingestion doesn ' t re-run.

Option A bundles everything into one notebook, which means a curation bug forces a full re-ingestion. Option B copies logic three times — any future change must be applied in triplicate, which is a maintenance hazard. Option C forces everything through SQL MERGE, which is the wrong tool for raw-event ingestion and doesn ' t address cleansing or schema drift.

Production ingestion: Job compute

BI: Serverless SQL warehouse

Job compute is designed for automated production workloads executed through Lakeflow Jobs. Its lifecycle can be tied to the job run, providing workload isolation and avoiding the cost of maintaining an interactive all-purpose cluster continuously. It is therefore appropriate for scheduled ingestion and transformation processing. A serverless SQL warehouse is designed for BI and Databricks SQL workloads. It provides rapid startup, automatic infrastructure management, scaling, and optimized SQL-query execution for dashboards and reporting tools. All-purpose compute is intended primarily for interactive notebook development and exploration, while shared compute does not provide the same job-specific isolation or SQL-serving experience. Consequently, job compute should support production ingestion, and a serverless SQL warehouse should serve the BI workload.

 CREATE OR REFRESH STREAMING TABLE

 city

CREATE OR REFRESH STREAMING TABLE defines a streaming table managed by Lakeflow Spark Declarative Pipelines. When the pipeline refreshes, Databricks incrementally processes newly available source data and maintains the resulting table. CREATE OR REPLACE TABLE would create a conventional table and does not provide the required streaming-table semantics. The query selects city AS city and calculates SUM(sales) AS total_sales. Because the aggregation must produce one result for each city, the GROUP BY expression must be city. Neither SUM(sales) nor total_sales belongs in the grouping clause: the former is the aggregate calculation, while the latter is only the alias assigned to its result. This produces continuously maintained city-level sales totals from the sales_order source table.

Installing a Maven package on every node requires changing Cluster1’s library configuration. The CAN MANAGE permission grants the cluster-level management capabilities required to add or remove libraries without giving User1 broader control over the entire workspace or Azure resource group. CAN RESTART allows a user to restart the cluster but does not provide sufficient authority to change its library configuration. Assigning the Azure Contributor role would grant excessive control over the resource group and does not represent the appropriate Databricks object-level permission. Making User1 a workspace administrator would provide substantially more privileges than necessary. Granting CAN MANAGE directly on Cluster1 therefore satisfies the operational requirement while limiting User1’s authority to the specific compute resource that must be modified.

The correct mapping is SCD Type 1 for equipment metadata and SCD Type 2 for IoT sensor ownership history.

SCD Type 1 overwrites the existing record whenever an attribute changes — no history is kept. Contoso ' s requirement for equipment metadata (name, manufacturer, model, commissioning date) states ' historical values are NOT required, ' which is the textbook definition of Type 1. A MERGE INTO with WHEN MATCHED THEN UPDATE handles this cleanly in Delta Lake.

SCD Type 2 creates a new row for each change, preserving the full history through effective-date or version columns. Contoso requires that ' analysts must track the full history of ownership ' as sensors change hands over time — that full audit trail is only possible with Type 2. Type 3 (keeping just the previous value in an extra column) would lose earlier ownership records, so it doesn ' t satisfy the ' full history ' requirement.

Repairing the current job run allows Lakeflow Jobs to rerun the failed task while preserving the successful status of tasks that already completed. This prevents several hours of successful processing from being repeated and reduces both recovery time and compute consumption. If downstream tasks had been skipped because of the failure, they can also be included in the repair operation as appropriate. Updating job parameters changes configuration but does not retry the failed task in the existing run. Restarting Job1 creates a new run and ordinarily executes the workflow again from its beginning. Disabling and reenabling the schedule only changes whether future scheduled runs can start; it does not repair the failed execution. Repair Run is specifically designed for recovering selected failed or skipped tasks in a multi-task workflow.

A CHECK constraint enforces a Boolean condition whenever data is inserted or updated. The constraint can require payment_amount > = 0 AND payment_amount < = 10000, causing a transaction containing an invalid value to fail instead of allowing the record into Payments. This provides storage-level data-quality enforcement regardless of which pipeline, notebook, or SQL statement performs the write. Row-level security controls which existing records users can see; it does not reject invalid writes. SELECT statements filter results only when they are executed and therefore cannot protect the underlying table. Table update triggers are not the standard Delta Lake mechanism for this requirement. Azure Databricks classifies CHECK constraints as enforced constraints and rejects transactions when their conditions are violated. Microsoft Learn

groupBy

pivot

The groupBy( " Region " ) operation establishes one aggregation group for each geographical region in the DataFrame. The subsequent pivot( " Product " ) operation converts every distinct Product value into a separate output column. Finally, agg(F.sum( " Sales " )) calculates the total sales for every product within each region. The resulting DataFrame contains one row per region and one column for each product, with the corresponding aggregated sales value. cogroup combines grouped datasets and is not used for creating a conventional pivot table. coalesce controls the number of DataFrame partitions or replaces null values, depending on its context. The sum function is already provided inside the aggregation expression and therefore is not required in either blank.

The correct answer is C — Enable schema evolution.

When new columns are added to the source data, a pipeline without schema evolution treats the unexpected columns as a schema mismatch and fails the write. Schema evolution, when enabled in an SDP pipeline, automatically adds those new columns to the target Delta table on the next pipeline run. The pipeline continues without intervention, and no historical data is lost.

Option A (disable schema enforcement) is the wrong lever — it removes all schema validation, which could allow corrupt or mistyped data into Table1. Schema evolution is a targeted, safer response.

Option B (row filters to exclude records with new columns) would silently discard valid records just because they carry extra fields — that ' s data loss. Option D (separate table per schema version) creates an explosion of tables as schemas evolve and makes downstream analytics significantly more complex. Schema evolution is the clean, built-in solution.

Branch1 must first incorporate the current changes from main. During that merge, the conflicting sections of Notebook1 can be reviewed and resolved so that the resulting feature-branch version contains the required work from both branches. After committing the resolution, Branch1 can be merged into main normally. Cloning either branch into another Git folder creates another working copy but does not resolve the conflicting histories. Applying changes directly to main bypasses the controlled feature-branch workflow and risks omitting or overwriting one contributor’s work. Resolving the conflict on Branch1 also allows the combined notebook to be tested before updating the shared main branch. Therefore, merging main into Branch1, resolving the conflict, and completing the final merge is the correct workflow.

The correct answer is B — No.

This is a common Python-to-PySpark trap. In pure Python, comparing a value to None with != works as expected. In PySpark, null comparisons follow SQL null semantics: any comparison involving NULL returns NULL (not True or False). So df.filter(df.order_amount != None) doesn ' t evaluate to True for non-null rows — the comparison itself returns NULL for null values, and Spark interprets NULL in a filter as False, effectively dropping null rows. But the behaviour is undefined in edge cases and is not the documented approach.

More practically, Python ' s None and Spark ' s SQL NULL are different concepts. PySpark Column objects don ' t support Python ' s native equality/inequality semantics for null checking. The result is typically an empty DataFrame or incorrect filtering behaviour.

Always use .isNotNull() or .isNull() for null checks in PySpark column expressions. These methods are specifically designed for SQL-null-aware comparisons and produce correct, predictable results.

The instructions must be added to Space1 because a Genie knowledge store is scoped to the individual Genie space, now called a Genie Agent. Adding synonyms and business-language instructions there teaches Space1 that “transaction,” “event,” “order,” and “invoice” refer to sales information in Sales1. The configuration affects only that Genie space, satisfying the requirement that other spaces remain unchanged. Adding instructions to Sales1 or Schema1 would modify shared Unity Catalog metadata and could affect other consumers of those objects. Notebook1 is unrelated to the semantic instructions used by Genie when converting natural-language questions into SQL. Genie knowledge stores contain space-specific definitions, synonyms, join relationships, SQL expressions, and prompt-matching guidance without changing the underlying Unity Catalog objects. Microsoft Learn

The correct answers are C and D. Together they deliver cost-efficient autoscaling:

D (Enable autoscaling) allows the cluster to grow when workloads increase and shrink when they ease off. This satisfies both scale-up and scale-down requirements without manual intervention.

C (Auto-termination after 30 minutes of inactivity) ensures the cluster stops entirely when no work is running, eliminating the cost of an idle cluster. This is the cheapest possible state.

Option A (disable Photon) reduces compute acceleration — that ' s a performance regression with no meaningful cost benefit for autoscaling. Option B (compute policy that lets users manage settings) adds governance overhead and doesn ' t address scaling behaviour. Option E (fixed number of workers) is the opposite of autoscaling — a static worker count that either over-provisions during quiet periods or under-provisions during peaks.

The correct answer is B — Repair the current job run.

Repair Run is designed for exactly this situation: a long-running job where most tasks succeeded but the final task failed due to a transient issue. Instead of restarting the entire eight-task job from the beginning — wasting hours of compute — Repair Run re-executes only the failed task and any dependents that were skipped as a result. All tasks that completed successfully are marked done and their outputs are reused.

Option A (update job parameters) changes configuration for future runs but doesn ' t re-execute the failed task in the current run. Option C (Restart Job1) re-runs every task from the start — precisely what the question says to avoid. Option D (disable and re-enable the schedule) creates a brand-new run on the next schedule trigger, again starting from the beginning rather than repairing the existing run.

Registering an ADLS Gen2 account as an external location in Unity Catalog requires three steps in a specific order:

Step 1: Create a Databricks Access Connector. This Azure resource wraps a system-assigned or user-assigned managed identity. It is the credential-less authentication bridge between Databricks and Azure Storage — no SAS tokens or access keys are stored anywhere in the workspace.

Step 2: Create a Storage Credential in Unity Catalog that references the Access Connector. This object tells Unity Catalog ' use this identity when accessing storage. ' The Storage Credential is the reusable authentication object.

Step 3: Create an External Location that maps a Unity Catalog path to the specific ADLS Gen2 container using the Storage Credential. This is the object that grants Databricks users access to files under that path, and it respects the principle of least privilege by scoping access to a specific container.