A data engineer manages a production Lakeflow Declarative Pipeline that processes customer transaction data. The pipeline includes several data quality expectations such as transaction_amount > 0 and customer_id IS NOT NULL. These expectations are defined using the EXPECT clause in SQL.
The engineer aims to monitor the pipeline's data quality by analyzing the number of records that passed or failed each expectation during the latest pipeline update. The Lakeflow Declarative Pipelines event logs are stored in a Delta table named event_log_table.
For the most recent pipeline update, determine a programmatically appropriate approach to extract information like the name of each expectation, associated dataset, count of records that passed the expectation, and count of records that failed the expectation.
Which method retrieves the desired data quality metrics from the Lakeflow Declarative Pipelines event log?

• A. Access the event_log_table, filter for events where event_type = 'flow_progress', and parse details.flow_progress.data_quality.expectations field to extract the required metrics.
• B. Access the event_log_table, filter for events where event_type = 'expectation_result', and extract the expectation metrics from the details field.
• C. Use the Lakeflow Declarative Pipelines UI to navigate to the specific pipeline, select the dataset, and view the Data Quality tab to manually retrieve the expectation metrics.
• D. Query the event_log_table for events with event_type = 'data_quality' and directly select the passed_records and failed_records fields.

The data engineer team is configuring environment for development testing, and production before beginning migration on a new data pipeline. The team requires extensive testing on both the code and data resulting from code execution, and the team want to develop and test against similar production data as possible.
A junior data engineer suggests that production data can be mounted to the development testing environments, allowing pre production code to execute against production data. Because all users have Admin privileges in the development environment, the junior data engineer has offered to configure permissions and mount this data for the team.
Which statement captures best practices for this situation?

• A. In environments where interactive code will be executed, production data should only be accessible with read permissions; creating isolated databases for each environment further reduces risks.
• B. Because access to production data will always be verified using passthrough credentials it is safe to mount data to any Databricks development environment.
• C. All developer, testing and production code and data should exist in a single unified workspace; creating separate environments for testing and development further reduces risks.
• D. Because delta Lake versions all data and supports time travel, it is not possible for user error or malicious actors to permanently delete production data, as such it is generally safe to mount production data anywhere.

A data engineering team needs to implement a tagging system for their tables as part of an automated ETL process, and needs to apply tags programmatically to tables in Unity Catalog.
Which SQL command adds tags to a table programmatically?

• A. COMMENT ON TABLE table_name TAGS ('key1' = 'value1', 'key2' = 'value2');
• B. ALTER TABLE table_name SET TAGS ('key1' = 'value1', 'key2' = 'value2');
• C. SET TAGS FOR table_name AS ('key1' = 'value1', 'key2' = 'value2');
• D. APPLY TAGS ON table_name VALUES ('key1' = 'value1', 'key2' = 'value2');

A data team is working to optimize an existing large, fast-growing table 'orders' with high cardinality columns, which experiences significant data skew and requires frequent concurrent writes. The team notice that the columns 'user_id', 'event_timestamp' and 'product_id' are heavily used in analytical queries and filters, although those keys may be subject to change in the future due to different business requirements. Which partitioning strategy should the team choose to optimize the table for immediate data skipping, incremental management over time, and flexibility?

• A. Cluster the table with: ALTER TABLE orders CLUSTER BY user_id, product_id, event_timestamp
• B. Partition the table with: ALTER TABLE orders PARTITION BY user_id, product_id, event_timestamp
• C. Use z-order after partitiing the table: OPTIMIZE orders ZORDER BY (user_id, product_id) WHERE event_timestamp = current date () - 1 DAY
• D. Z-order the table with OPTIMIZE orders ZORDER BY (user_id, product_id, event_timestamp)

A nightly batch job is configured to ingest all data files from a cloud object storage container where records are stored in a nested directory structure YYYY/MM/DD. The data for each date represents all records that were processed by the source system on that date, noting that some records may be delayed as they await moderator approval. Each entry represents a user review of a product and has the following schema:
user_id STRING, review_id BIGINT, product_id BIGINT, review_timestamp TIMESTAMP, review_text STRING The ingestion job is configured to append all data for the previous date to a target table reviews_raw with an identical schema to the source system. The next step in the pipeline is a batch write to propagate all new records inserted into reviews_raw to a table where data is fully deduplicated, validated, and enriched.
Which solution minimizes the compute costs to propagate this batch of data?

• A. Configure a Structured Streaming read against the reviews_raw table using the trigger once execution mode to process new records as a batch job.
• B. Use Delta Lake version history to get the difference between the latest version of reviews_raw and one version prior, then write these records to the next table.
• C. Perform a batch read on the reviews_raw table and perform an insert-only merge using the natural composite key user_id, review_id, product_id, review_timestamp.
• D. Filter all records in the reviews_raw table based on the review_timestamp; batch append those records produced in the last 48 hours.
• E. Reprocess all records in reviews_raw and overwrite the next table in the pipeline.