Databricks Data Engineer Associate Question 525
Single answerYou are designing a Databricks job that needs to connect to an Azure Data Lake Storage Gen2 account to read and write data. The organization mandates secure and scalable access management practices. Which of the following is the BEST approach to configure the connection?
- A
Use a service principal with the necessary permissions and configure it in the Databricks secret scope.
- B
Embed the Azure Data Lake Storage Gen2 account key directly in the job code for authentication.
- C
Use the personal credentials of the data engineer running the job to access the storage account.
- D
Allow access to the storage account by setting it to public and restricting IP addresses.
Show answer and explanation
Correct answer: A
Explanation
Using a service principal for authentication ensures secure, application-specific access to Azure resources. Service principals can be granted least-privilege access and are centrally managed, making them a best practice for connecting Databricks to external storage systems. By storing the credentials in a Databricks secret scope, the sensitive information is securely managed and not exposed in the code.
- A. Correct.
This is the best practice as service principals are designed for secure, application-level authentication and can be managed centrally. By storing the service principal credentials in a Databricks secret scope, you ensure sensitive data is not exposed in the code.
- B. Incorrect.
Embedding storage account keys in the job code is a poor security practice as it can expose sensitive information and is difficult to maintain if keys need to be rotated.
- C. Incorrect.
Using personal credentials violates security best practices, does not scale well, and creates dependency on individual users.
- D. Incorrect.
Setting the storage account to public violates security principles and exposes the data to potential unauthorized access, even with IP restrictions.