Databricks Data Engineer Associate Question 98
Select 2You are working on a Databricks notebook and need to reference a large CSV file stored in a cloud storage location multiple times during a series of transformations. Which of the following approaches should you use to reference the file efficiently and according to best practices?
- A
Create a view using the SQL
CREATE VIEWcommand. - B
Create a temporary view using
DataFrame.createOrReplaceTempView(). - C
Create a Common Table Expression (CTE) in a SQL query.
- D
Directly perform transformations on the DataFrame without creating a reference.
Show answer and explanation
Correct answers: B, C
Explanation
When working with large files in Databricks, temporary views and Common Table Expressions (CTEs) are efficient ways to create reusable references. Temporary views are ideal for multiple transformations and last for the session's duration, while CTEs are useful for inline references in complex SQL queries. Permanent views are not suitable for short-lived operations, and directly transforming the DataFrame without a reference can result in less maintainable code.
- A. Incorrect.
Creating a permanent view using
CREATE VIEWis not efficient for temporary or short-lived operations in Databricks. Additionally, creating a permanent view requires writing it to the metastore, which is unnecessary for this use case. - B. Correct.
Creating a temporary view using
DataFrame.createOrReplaceTempView()is a suitable way to create a reusable reference to the file for the duration of the session. It is lightweight and does not persist beyond the session. - C. Correct.
Using a Common Table Expression (CTE) in a SQL query is a good approach for creating an inline, reusable reference to the file, especially in complex SQL transformations.
- D. Incorrect.
Directly performing transformations on the DataFrame without creating a reference is possible, but it leads to duplicated code and can reduce readability and maintainability in the notebook.