Databricks Data Engineer Professional Question 261
Select 2You are working on a shared Databricks environment where multiple notebooks require the same custom functions from a Python module. Instead of copying the code into each notebook, you decide to refactor this setup to use a Python file dependency. Which of the following steps are required to implement this pattern correctly?
- A
Save the Python file locally on your machine and manually upload it to each notebook.
- B
Store the Python file in a shared location, such as the Databricks FileStore or a cloud storage bucket.
- C
Use the
%runmagic command to reference the Python file in your notebook. - D
Use Databricks Utilities (
dbutils) to import the Python file into the notebook session. - E
Use the
sys.path.appendfunction to add the path of the Python file to the Python module search path. - F
Package the Python file as a PyPI library and install it on the cluster.
Show answer and explanation
Correct answers: B, E
Explanation
To adapt a notebook dependency pattern to use Python file dependencies, the Python file should first be stored in a shared location accessible by all intended notebooks (e.g., Databricks FileStore or cloud storage). Then, the path to this file must be added to the Python module search path using sys.path.append, allowing the file to be imported as a module. This approach ensures reusability, maintainability, and efficient code management.
- A. Incorrect.
Incorrect: Manually uploading the Python file to each notebook defeats the purpose of reusability and creates maintenance overhead.
- B. Correct.
Correct: Storing the Python file in a shared location ensures that it can be accessed by multiple notebooks and users in a streamlined manner.
- C. Incorrect.
Incorrect: The
%runmagic command is used to run other notebooks, not standalone Python files. - D. Incorrect.
Incorrect: Databricks Utilities (
dbutils) are used for tasks like file system operations, but they cannot be used to import Python files as dependencies directly. - E. Correct.
Correct: Adding the path of the Python file to the Python module search path using
sys.path.appendensures that the file can be imported as a module. - F. Incorrect.
Incorrect: While packaging as a PyPI library is a valid approach, it is unnecessary for a single Python file dependency and involves extra complexity.