DP-750 Implementing Data Engineering Solutions Using Azure Databricks (beta) Questions and Answers

The correct answer is B — a job failure.

The requirement draws a clear line: non-critical task failures should be logged silently; notifications should only fire when a critical failure causes the whole job to stop. Configuring the alert on 'Job Failure' achieves this precisely — the notification triggers when the job itself reaches a Failed terminal state, which only happens when at least one critical task has failed and the job cannot complete.

Option A (task failure) would send a notification for every task-level failure, including non-critical ones. That's exactly the noise the question wants to avoid. Option C (job success) would never alert on failures at all. Option D (task success) confirms completion but doesn't catch failures.

Setting alerting at the job level rather than the task level is also simpler to configure — you don't need to mark individual tasks as critical or non-critical in the notification settings.

The correct answer is A — INSERT INTO.

INSERT INTO appends all rows from the source to the target. Existing rows in table2 are untouched — the historical data stays intact — and the current year's rows from table1 are added. That is exactly what the requirement asks: 'preserve any existing data and minimise processing effort.'

Option B (CREATE TABLE AS SELECT) would fail if table2 already exists, or if used with CREATE OR REPLACE it wipes table2 entirely before writing — historical data gone. Option C (INSERT OVERWRITE) replaces the entire table content with only the table1 rows, also destroying table2's existing data. Option D (CREATE OR REPLACE TABLE AS SELECT) explicitly drops and recreates the table, eliminating all historical records.

INSERT INTO is deliberately the simplest possible command here — one line, no special options, no risk of accidental data loss.

Data lineage in Unity Catalog is captured automatically — but only when jobs and notebooks run on clusters that are Unity Catalog-aware. Specifically, clusters must use 'Shared' or 'Single User' access mode. Clusters set to 'No Isolation Shared' or legacy 'High Concurrency' mode do not emit lineage events to the Unity Catalog lineage service.

No instrumentation, logging code, or external tools are required. The lineage service operates transparently, intercepting read and write operations at the Spark plan level and recording the table-to-table and column-to-column relationships.

To review captured lineage, open Catalog Explorer, navigate to the table, and select the Lineage tab. This shows the upstream sources that populate the table and the downstream consumers that read from it — all as an interactive graph, with no additional tooling needed. This built-in visibility is one of the core governance benefits Unity Catalog provides.

Two actions are needed to reclaim storage while keeping tables queryable:

Set delta.deletedFileRetentionDuration and delta.logRetentionDuration to 90 days on each table. These properties define the retention floor — VACUUM will not touch anything newer than this threshold, so no data needed for time travel within 90 days can be accidentally removed.

Run VACUUM on each table. VACUUM is the Delta Lake command that physically removes data files and transaction log entries older than the retention duration. Importantly, VACUUM runs as a background operation — it uses Delta Lake's MVCC (multi-version concurrency control) to ensure that concurrent reads against the table continue uninterrupted while cleanup happens. Tables are fully available throughout.

OPTIMIZE compacts small files for query performance but doesn't delete anything. Manually deleting files outside the Delta protocol would corrupt the table.

The correct answer is A. Photon is Azure Databricks' native vectorized query engine, written in C++, designed to accelerate data ingestion and SQL-heavy workloads significantly over the standard Spark JVM path. Enabling it on a job compute cluster directly addresses Contoso's requirement for 'fast and consistent performance for BI workloads' and 'production ingestion workloads that can scale automatically during telemetry spikes.'

Photon integrates transparently — no code changes are needed — and pairs well with autoscaling job clusters to handle the bursty 40,000-sensor telemetry load.

Option B contradicts the isolation requirement: Contoso explicitly needs production and development separated, not merged onto shared compute. Option C with a fixed large node gives peak capacity at all times, driving up costs even during quiet periods. Option D disabling autoscaling is the opposite of what's needed — telemetry spikes require elastic scaling, not a locked node count.

The correct answer is D. Breaking the pipeline into separate tasks for ingestion, cleansing, and curation is the foundation of well-designed Lakeflow Jobs pipelines. Each task should own one responsibility — when a task does too much, debugging a failure becomes a hunt through unrelated code, and retry logic becomes expensive because you re-execute work that already succeeded.

Contoso's planned changes explicitly call for 'a clear execution order and dependencies' and 'orchestrate multi-step ingestion and transformation workflows.' Separate tasks map directly to those goals: Lakeflow Jobs tracks each task's status independently, so if cleansing fails, ingestion doesn't re-run.

Option A bundles everything into one notebook, which means a curation bug forces a full re-ingestion. Option B copies logic three times — any future change must be applied in triplicate, which is a maintenance hazard. Option C forces everything through SQL MERGE, which is the wrong tool for raw-event ingestion and doesn't address cleansing or schema drift.