Databricks Data Engineer Professional Question 259
Select 3A data engineering team is working on a Databricks project that involves multiple Python files for modularity. These Python files include utility functions and business logic, and they need to be reused across multiple notebooks in the project. The team previously used %run to include dependencies directly from other notebooks, but they want to adapt this pattern to use Python file dependencies instead. Which of the following steps are required to ensure the Python file dependencies are correctly set up and reusable across notebooks?
- A
Save the Python files into the Databricks File System (DBFS) or a shared location accessible to all cluster nodes.
- B
Use the
sys.path.append()method to add the directory containing the Python files to the Python path. - C
Use the
%pip installmagic command to directly install the Python files into the notebook environment. - D
Import the specific modules or functions from the Python files using standard Python import statements.
- E
Manually copy and paste the content of the Python files into each notebook that requires them.
Show answer and explanation
Correct answers: A, B, D
Explanation
To adapt a notebook dependency pattern to use Python file dependencies, you need to make the Python files accessible across all cluster nodes (e.g., by saving them to DBFS or a shared location), ensure Python can locate them using sys.path.append(), and then import the necessary modules or functions using standard Python imports. This approach enhances reusability, maintainability, and scalability compared to methods like %run or manual copying.
- A. Correct.
Correct: Saving the Python files into DBFS or a shared location ensures that they are accessible across all cluster nodes, which is essential for reuse.
- B. Correct.
Correct: Using
sys.path.append()allows Python to locate the Python files by adding their directory to the Python path. - C. Incorrect.
Incorrect: The
%pip installcommand is used for installing packages from PyPI or other repositories, not for installing local Python files. - D. Correct.
Correct: Once the Python files are accessible and on the Python path, you can import specific functions or modules using standard Python import statements.
- E. Incorrect.
Incorrect: Manually copying and pasting the content is not a scalable or maintainable approach for managing dependencies.