Databricks Data Engineer Professional Question 263
Select 3You are working on a Databricks notebook that requires the reuse of utility functions defined in a Python file (e.g., utils.py). Previously, you used %run in the notebook to include dependencies from another notebook. However, for better dependency management and reusability, you decide to refactor the code to use Python file dependencies. What steps should you take to properly implement this change?
- A
Place the
utils.pyfile in the Databricks FileStore and use%runto include it in your notebook. - B
Upload the
utils.pyfile to a dedicated folder in the Databricks workspace and use theimportstatement to include it in your notebook. - C
Package the utility functions into a Python module, upload the package to a Databricks cluster as a library, and use the
importstatement to include it in your notebook. - D
Store the
utils.pyfile in a shared location, such as DBFS, and usesys.path.appendto modify the Python path before importing the file. - E
Use the Databricks REST API to dynamically load the
utils.pyfile into the notebook at runtime and execute its content.
Show answer and explanation
Correct answers: B, C, D
Explanation
To replace %run with Python file dependencies, you can upload the Python file to the workspace and use the import statement, package the utility functions into a Python module and upload it as a library, or store the file in a shared location like DBFS and modify the Python path using sys.path.append. These approaches enable efficient dependency management and reusability, aligning with best practices in Databricks. %run is specific to including notebooks, and using the REST API for this purpose is neither standard nor recommended.
- A. Incorrect.
Using
%runis specific to including other notebooks, not Python files. This approach is not suitable for Python file dependencies. - B. Correct.
This is a valid approach. Uploading the
utils.pyfile to the Databricks workspace allows you to use theimportstatement to include it in your notebook. - C. Correct.
This is a valid approach. Packaging the utility functions into a Python module and uploading it to the cluster as a library allows for efficient dependency management and reusability.
- D. Correct.
This is a valid approach. By storing
utils.pyin DBFS and modifying thesys.path, you can import the file and make its functions available in the notebook. - E. Incorrect.
Using the Databricks REST API to dynamically load a Python file at runtime is not a standard or recommended practice for managing dependencies in Databricks.