Azure - Accessing Blob storage from Data bricks cluster using account key


Azure - Accessing Blob storage from Data bricks cluster

container = "raw"
storageAccount = "arunacc"
accessKey = "<<>>"
accountKey = "fs.azure.account.key.{}.blob.core.windows.net".format(storageAccount)
# Set the credentials to Spark configuration
spark.conf.set(
  accountKey,
  accessKey)
spark._jsc.hadoopConfiguration().set(
  accountKey,
  accessKey)

# Mount the drive for native python
inputSource = "wasbs://{}@{}.blob.core.windows.net".format(container, storageAccount)
mountPoint = "/mnt/" + container
extraConfig = {accountKey: accessKey}
print("Mounting: {}".format(mountPoint))
try:
  dbutils.fs.mount(
    source = inputSource,
    mount_point = str(mountPoint),
    extra_configs = extraConfig
  )
  print("=> Succeeded")
except Exception as e:
  if "Directory already mounted" in str(e):
    print("=> Directory {} already mounted".format(mountPoint))
  else:
    raise(e)


Read file
df = spark.read.text("/mnt/raw/data-01.csv")

and EDA starts from here....................



Setup details on azure portal for databricks


Downaload and install databricks CLI

reference
https://docs.databricks.com/user-guide/dev-tools/databricks-cli.html#databricks-cli

step 1: pip3 install databricks-cli


Set up authentication
https://docs.databricks.com/api/latest/authentication.html#authentication

To configure the CLI to use the access token
databricks configure --token


Create a Databricks-backed secret scope
databricks secrets create-scope --scope <scope-name>

databricks secrets create-scope --scope <scope-name> --initial-manage-principal users

List secret scopes
databricks secrets list-scopes

Create a secret
databricks secrets put --scope <scope-name> --key <key-name>

List secrets
databricks secrets list --scope <scope-name>

Read a secret
DBUtils secret utilities are available only on clusters running Databricks Runtime 4.0 and above.



Databricks Utilities
https://docs.databricks.com/user-guide/dev-tools/dbutils.html#dbutils-secrets



-------------------------
https://uksouth.azuredatabricks.net#secrets/create

xxxxxxxxxxxx


./databricks secrets create-scope --scope arunscope --initial-manage-principal users


fs.azure.sas.raw.testarunaccount..blob.core.windows.net


./databricks secrets put --scope arunscope --key key1


storage account - testarunaccount
storage container - raw
scope name - arunscope

Comments

Post a Comment

Popular posts from this blog

Qlik Sense Important Links

Qlik Sense learning https://learning.qlik.com/ https://www.qlik.com/us/services/training/self-paced-learning Qlik sense pricing   https://www.capterra.com/p/145530/QlikSense/#pricing https://www.qlik.com/us/pricing/qlik-sense-client-managed Qlik Sense Server Installation Document preview of Qlik Sense Server Installation Step By Step Guide.pdf https://community.qlik.com/t5/Documents/Qlik-Sense-Server-Installation-Step-By-Step-Guide/ta-p/1480562 https://community.qlik.com/t5/New-to-Qlik-Sense/how-to-download-Qlik-Sense-server/td-p/869140 Support Portal https://support.qlik.com/QS_Home_Page Qlik FIPS Support FIPS mode is not supported out of the box for Qlik Sense, QlikView, and Qlik NPrinting. Workarounds exist for NPrinting and Sense.  For general information about FIPS and Microsoft's statement around it, see  Microsoft's Security Guidance blog. https://community.qlik.com/t5/Knowledge-Base/Qlik-Software-and-FIPS-mode/ta-p/1717239 https://docs.microsoft.com/en-US/windows/...
Read more

Cloud Architecture Notes

Image
The architectural priorities and needs of every app are different, but the four pillars of architecture are an excellent guidepost you can use to make sure that you have given enough attention to every aspect of your application: Security : Safeguarding access and data integrity and meeting regulatory requirements Performance and scalability : Efficiently meeting demand in every scenario Availability and recoverability : Minimizing downtime and avoiding permanent data loss Efficiency and operations : Maximizing maintainability and ensuring requirements are met with monitoring A layered approach to security Defense in depth is a strategy that employs a series of mechanisms to slow the advance of an attack aimed at acquiring unauthorized access to information The common principles used to define a security posture are confidentiality, integrity, and availability, known collectively as CIA. Confidentiality - Principle of least privilege. Restricts access to informa...
Read more

AWS Rout53 NS records do not match with whois dns records OR Your site NOT working with registered domain name? Check this...

  Log in to your AWS web console Select Route53 service Select Hosted Zones in the left pane Select domain name (but do not click on the domain name, just select the radio button) Notice the 4 name servers in the right pane. Next, go to domain registrar site (in your case stay in Route53), log in and configure name servers for your domain. In Route53: select 'Registered domains' in the left pane, click on your domain name, verify that in the upper right you have name servers from step 5, if they don't exactly match, click 'Add/Edit Name servers' and enter name servers from step 5) After this you will receive email on your registered email id with AWS. Once the sync os done you will get notified. After notification you can try your ULR <some-name>.com or www.<some-name>.com "A" records created during route 53
Read more