Azure - Accessing Blob storage from Data bricks cluster using account key

Azure - Accessing Blob storage from Data bricks cluster

container = "raw"
storageAccount = "arunacc"
accessKey = "<<>>"
accountKey = "fs.azure.account.key.{}.blob.core.windows.net".format(storageAccount)
# Set the credentials to Spark configuration
spark.conf.set(
  accountKey,
  accessKey)
spark._jsc.hadoopConfiguration().set(
  accountKey,
  accessKey)

# Mount the drive for native python
inputSource = "wasbs://{}@{}.blob.core.windows.net".format(container, storageAccount)
mountPoint = "/mnt/" + container
extraConfig = {accountKey: accessKey}
print("Mounting: {}".format(mountPoint))
try:
  dbutils.fs.mount(
    source = inputSource,
    mount_point = str(mountPoint),
    extra_configs = extraConfig
  )
  print("=> Succeeded")
except Exception as e:
  if "Directory already mounted" in str(e):
    print("=> Directory {} already mounted".format(mountPoint))
  else:
    raise(e)


Read file
df = spark.read.text("/mnt/raw/data-01.csv")

and EDA starts from here....................



Setup details on azure portal for databricks


Downaload and install databricks CLI

reference
https://docs.databricks.com/user-guide/dev-tools/databricks-cli.html#databricks-cli

step 1: pip3 install databricks-cli


Set up authentication
https://docs.databricks.com/api/latest/authentication.html#authentication

To configure the CLI to use the access token
databricks configure --token


Create a Databricks-backed secret scope
databricks secrets create-scope --scope <scope-name>

databricks secrets create-scope --scope <scope-name> --initial-manage-principal users

List secret scopes
databricks secrets list-scopes

Create a secret
databricks secrets put --scope <scope-name> --key <key-name>

List secrets
databricks secrets list --scope <scope-name>

Read a secret
DBUtils secret utilities are available only on clusters running Databricks Runtime 4.0 and above.



Databricks Utilities
https://docs.databricks.com/user-guide/dev-tools/dbutils.html#dbutils-secrets



-------------------------
https://uksouth.azuredatabricks.net#secrets/create

xxxxxxxxxxxx


./databricks secrets create-scope --scope arunscope --initial-manage-principal users


fs.azure.sas.raw.testarunaccount..blob.core.windows.net


./databricks secrets put --scope arunscope --key key1


storage account - testarunaccount
storage container - raw
scope name - arunscope