Databricks Data Engineer Professional Question 265
Select 3You are maintaining a Databricks project where a shared Python library was previously packaged and distributed as a Wheel (.whl) file. To simplify development and avoid repackaging, you want to adapt the project to use direct imports with relative paths. After making the necessary changes, you notice that the Databricks notebook fails to import a module from the shared library. What steps should you take to resolve this issue?
- A
Ensure the shared library's directory is added to the
sys.pathin the notebook's initialization code. - B
Update the notebook's cluster configuration to include the shared library's folder in the
PYTHONPATHenvironment variable. - C
Convert all absolute imports in the shared library to relative imports based on the new project directory structure.
- D
Revert to using the Wheel file instead, as relative imports are not supported in Databricks.
- E
Verify that the shared library's directory is correctly uploaded to the Databricks workspace or mounted to DBFS.
Show answer and explanation
Correct answers: A, B, E
Explanation
When transitioning from using Wheels to direct imports via relative paths, you must ensure that the shared library is accessible to the Databricks environment. Adding the library's directory to the sys.path and setting the PYTHONPATH environment variable ensures Python can locate the library during runtime. Additionally, the shared library must be available in the Databricks workspace or DBFS so it can be accessed by the cluster.
- A. Correct.
Adding the shared library's directory to the
sys.pathin the notebook ensures that Python can locate the library for imports. This is a common practice when using relative paths for shared code. - B. Correct.
Setting the
PYTHONPATHenvironment variable for the cluster allows Databricks to recognize the shared library's location, making it available for import in notebooks. - C. Incorrect.
Converting absolute imports to relative imports is unnecessary unless the library’s internal structure has changed. The issue described is about Databricks failing to locate the library, not an import type mismatch.
- D. Incorrect.
Reverting to the Wheel file is not required, as Databricks supports using relative paths for shared libraries. Reverting would contradict the goal of simplifying development.
- E. Correct.
Ensuring the shared library's directory is accessible in the Databricks workspace or DBFS is essential for making the library available to the cluster and notebooks.