Databricks Data Engineer Associate Question 522
Single answerYou are configuring a Databricks workspace to allow a data pipeline to securely write data to an Azure Data Lake Storage account. As a best practice, you want to avoid using user personal access tokens or hardcoding credentials in notebooks. What should you do to ensure a secure and scalable authentication mechanism?
- A
Use a service principal with appropriate permissions to authenticate the connection and store its credentials securely in Databricks Secrets.
- B
Use a shared access signature (SAS) token and hardcode it in the notebook that runs the pipeline.
- C
Use an individual user’s personal access token for authentication and store it in Databricks Secrets.
- D
Enable anonymous access to the Azure Data Lake Storage account and manage access through network rules.
Show answer and explanation
Correct answer: A
Explanation
Service principals are a best practice for secure and scalable connections in Databricks as they provide a dedicated, non-user-specific identity for authentication. Storing their credentials in Databricks Secrets ensures sensitive information is managed securely and can be easily updated without modifying code. This approach avoids the risks associated with hardcoding credentials or using user-specific tokens.
- A. Correct.
Using a service principal with appropriate permissions and storing its credentials securely in Databricks Secrets aligns with best practices for secure and scalable connections. Service principals provide a non-user-specific identity for authentication, ensuring better management and security.
- B. Incorrect.
Hardcoding a SAS token in a notebook is insecure as it exposes sensitive credentials in source code and can lead to unauthorized access if the code is shared or compromised.
- C. Incorrect.
Using an individual user’s personal access token ties the pipeline to a specific user, which is not scalable and can lead to issues if the user leaves the organization or the token expires.
- D. Incorrect.
Enabling anonymous access to the Azure Data Lake Storage account is highly insecure and should never be used in production scenarios as it allows unrestricted access to the data.