Databricks Data Engineer Professional Question 268
Select 3A data engineering team has been maintaining a Python library as a Wheel file in their Databricks environment. They decide to refactor the project to use direct imports with relative paths instead. What steps should the team take to correctly adapt their code for this change?
- A
Update the import statements in the codebase to use relative paths, such as
from .module_name import function_name. - B
Ensure that the
__init__.pyfiles are present in all directories to define Python packages. - C
Manually copy the library files to each Databricks cluster to ensure compatibility.
- D
Remove any previous references to the Wheel file from the cluster library configuration.
- E
Use the
%pip installcommand to install the library directly into the Databricks workspace.
Show answer and explanation
Correct answers: A, B, D
Explanation
To migrate from using a Wheel file to direct imports with relative paths, the team must first update the import statements in their codebase to use relative paths. They also need to ensure that __init__.py files exist in all directories that are intended to be Python packages. Finally, it is essential to remove the old Wheel file references from the cluster configuration to avoid conflicts. Manual copying and %pip install are not applicable or recommended in this scenario.
- A. Correct.
Correct: When switching from a Wheel-based library to direct imports, the import statements need to be updated to use relative paths.
- B. Correct.
Correct: Python uses
__init__.pyfiles to recognize directories as packages, which is required for relative imports to work correctly. - C. Incorrect.
Incorrect: Manually copying files to clusters is not scalable and not recommended in Databricks. Instead, proper project packaging or workspace structuring should be used.
- D. Correct.
Correct: Removing the Wheel file from the cluster library configuration ensures no conflicts occur between the old and new versions of the library.
- E. Incorrect.
Incorrect: The
%pip installcommand is used to install external libraries, not for adapting or using relative imports in a custom library.