Databricks Databricks-Machine-Learning-Professional Dumps Questions Answers

To drop a column from a Delta table, you need to enable column mapping mode for the table and use the ALTER TABLE statement with the DROP COLUMN clause. Alternatively, you can use the spark.read.format(“delta”).table(path) method to read the Delta table as a DataFrame and then use the drop method to drop the column. This will return a new DataFrame with the column removed. You can then overwrite the originalDelta table with the new DataFrame using the write.format(“delta”).mode(“overwrite”) method. The other options are incorrect because:

• Option A: spark.read.format(“delta”).load(path) will not read the Delta table as a table, but as a directory of files. This will not preserve the schema and metadata of the table, and will not allow you to overwrite the table with the new DataFrame.
• Option C: Delta tables can be modified using the Delta Lake APIs or SQL commands. Delta Lake supports schema evolution and column mapping to enable dropping columns from Delta tables.
• Option D: spark.read.table(path) will not work for Delta tables, as it expects a table name, not a path. To read a Delta table by its path, you need to use the spark.read.format(“delta”).load(path) or spark.read.format(“delta”).table(path) methods.
• Option E: spark.sql(“SELECT * EXCEPT star_rating FROM path”) will not drop the column from the Delta table, but only return a new DataFrame with the column excluded. This will not affect the original Delta table, and will not allow you to overwrite the table with the new DataFrame. References: Drop columns, Delta Lake Quickstart, Delta Lake column mapping

Feature drift is a change in the distribution of the input data over time, which can affect the performance and accuracy of the model. Monitoring feature drift is important to ensure that the model is still valid and reliable for the current data. One simple and low-cost method of monitoring numeric feature drift is to track the summary statistics trends of the input features, such as mean, standard deviation, minimum, maximum, etc. These statistics can be computed easily and efficiently, and can provide a quick overview of the changes in the data distribution. If the summary statistics deviate significantly from the baseline values, it may indicate a feature drift. However, summary statistics trends may not capture all the nuances of the data distribution, such as outliers, multimodality, skewness, etc. Therefore, other methods, such as statistical tests or visualizations, may be needed to complement the summary statistics trends and provide a more comprehensive analysis of the feature drift123 References:

• Monitoring Feature Drift - Databricks
• Monitor feature attribution skew and drift | Vertex AI | Google Cloud
• Monitoring Model Drift - 6 Different Methodologies - Qualdo

Jensen-Shannon (JS) distance is a symmetric measure of the distance between two probability distributions. It is based on the Kullback-Leibler (KL) divergence, but unlike KL, it does not depend on which distribution is considered the reference and which is considered the comparison. JS distance is also bounded between 0 and 1, which makes it easier to interpret and compare across different features.

Kolmogorov-Smirnov (KS) test is a non-parametric test that compares the cumulative distribution functions (CDFs) of two samples. It calculates the maximum difference between the CDFs and compares it to a critical value to determine if the samples are from the same distribution. KS test is sensitive to the shape, location, and scale of the distributions, but it can also be too sensitive for large datasets, where small changes can result in significant differences.

JS distance is more robust than KS test for numeric feature drift detection because it can handle large datasets without being too sensitive to minor fluctuations. It can also capture changes in the shape and spread of the distributions, not just the location. JS distance does not require any manual threshold or cutoff determinations, unlike KS test, which requires choosing a significance level and a critical value. JS distance can also be generalized to multivariate cases, while KS test is limited to univariate cases. References:

• Which test is the best? We compared 5 methods to detect data drift on large datasets
• The Jensen-Shannon Divergence: A Measure of Distance Between Probability Distributions
• Kullback–Leibler vs Kolmogorov-Smirnov distance