Data Engineering

Git and Azure DevOps Integration in Azure Databricks: Setup, Workflow, and CI/CD

By Vinit Patel November 20, 2026 10 min read

When several engineers work in the same Databricks workspace, notebooks written directly in the workspace quickly become a problem: 

  • Someone overwrites another person’s notebook 
  • Nobody knows who changed a piece of logic, or why 
  • Code is copied manually from Dev to Prod 
  • Nothing is reviewed before it reaches production 

Azure Databricks solves this with Git folders (formerly Databricks Repos), which link the workspace directly to an Azure DevOps repository. Combined with Azure Pipelines, the same code can then be deployed automatically to Test and Production. 

In this article, we’ll set up the integration step by step, look at the day-to-day developer workflow, and build a CI/CD pipeline. 

What are Databricks Git Folders? 

A Git folder is a workspace folder linked to a remote Git repository. From inside Databricks you can clone, create and switch branches, view diffs, commit and push, pull, merge, rebase, and resolve conflicts. 

Azure Repos (Azure DevOps) 
        │ 
        │  clone / pull / push 
        ▼ 
Databricks Workspace 
└── Users 
      └── abc@company.com 
            └── sales-data-platform     <- Git folder 
                  ├── notebooks/ 
                  ├── sql/ 
                  └── README.md 

Each developer clones the repository into their own user folder, so everyone has a private working copy. 

Overall Architecture 

   Developer (Databricks Git folder) 
                 │ 
                 │ 1. Commit & Push (feature branch) 
                ▼ 
    Azure Repos 
                │ 
                 │ 2. Pull Request → Review → Merge to main 
                ▼ 
   Azure Pipelines (CI/CD) 
                 │ 
                 │ 3. Validate & Deploy 
                ▼ 
      ┌─────────┴─────────┐ 
      ▼                   ▼ 
Databricks Test     Databricks Prod 
  Workspace           Workspace 

Step-by-Step Setup 

Step 1 – Configure Git Credentials in Databricks 

Databricks needs permission to read from and write to your Azure DevOps repository. There are two options. 

Option A – Microsoft Entra ID (Recommended) 

If Azure DevOps and Azure Databricks are linked to the same Microsoft Entra ID tenant, authentication happens automatically. No token is created or stored. 

  • In Databricks, click your user name → Settings → Linked accounts. 
  • Set the Git provider to Azure DevOps Services and choose the Microsoft Entra ID option. 
  • Click Save. 

Entra ID tokens are short-lived and rotate automatically, which is why Databricks recommends them over personal access tokens. 

Option B – Personal Access Token (PAT) 

Use a PAT only when the two services are in different Entra ID tenants. 

  • In Azure DevOps: User settings → Personal access tokens → New Token, with scope Code → Read & write. 
  • In Databricks: Settings → Linked accounts, choose Azure DevOps Services with the Personal access token option, and paste the token. 

Note: PATs expire. When one does, every Git operation fails until the token is renewed in Linked accounts. 

Step 2 – Create a Git Folder 

  • In Azure DevOps, open Repos → Clone and copy the HTTPS URL (https://dev.azure.com/<org>/<project>/_git/<repo>). 
  • In Databricks, go to Workspace → your user folder → Create → Git folder. 
  • Paste the URL, confirm the provider shows Azure DevOps Services, and click Create Git folder. 

Step 3 – Create a Feature Branch 

Click the branch name button in the Git folder (it shows main), then Create branch, and enter a name such as feature/bronze-sales. Never develop directly on main. 

Step 4 – Commit and Push 

Add or edit a notebook inside the Git folder, for example: 

# Bronze layer – raw sales ingestion 
from pyspark.sql.functions import current_timestamp 
 
raw_df = spark.read.format(“json”).load(“/Volumes/retail/raw/sales/”) 
 
(raw_df.withColumn(“ingestion_time”, current_timestamp()) 
    .write.format(“delta”).mode(“append”) 
    .saveAsTable(“retail.bronze.sales”)) 

Open the Git dialog, review the changes in the Changes tab, enter a commit message, and click Commit & Push. 

Step 5 – Pull Request and Merge 

In Azure DevOps, go to Repos → Pull requests → New pull request, set feature/bronze-sales → main, add reviewers, and create it. After approval, click Complete to merge. Other developers then click Pull in their Git folders to get the latest code. 

Daily Developer Workflow 

Pull latest main 
      │ 
      ▼ 
Create feature branch → Develop & test 
      │ 
      ▼ 
Commit & Push → Pull Request → Review 
      │ 
      ▼ 
Merge to main → CI/CD deploys automatically 

What Gets Stored in Git? 

Notebooks, Python/SQL files, YAML/JSON configs, and dependency files are all supported. By default, notebooks are saved in source format, which stores code only, not cell outputs: 

# Databricks notebook source 
df = spark.table(“retail.bronze.sales”) 
 
# COMMAND ———- 
 
display(df.limit(10)) 

This keeps the repository small and prevents query results from being committed by accident. 

Protecting the main Branch 

In Azure DevOps, go to Repos → Branches → … next to main → Branch policies, and enable: 

  • Minimum number of reviewers 
  • Check for linked work items 
  • Build validation (a pipeline must pass before the PR can be completed) 

CI/CD with Azure Pipelines 

There are two common ways to deploy Databricks code from Azure DevOps. 

Approach 1 – Update a Production Git Folder 

Production jobs point to notebooks in a dedicated Git folder (for example, under /Workspace/Production). After a merge, the pipeline updates that folder to the latest commit: 

databricks repos update /Workspace/Production/sales-data-platform –branch main 

Simple to set up, but it deploys code only. Jobs, compute, and permissions must still be managed separately. 

Approach 2 – Declarative Automation Bundles (Recommended) 

Declarative Automation Bundles (formerly Databricks Asset Bundles) define the whole project as code: notebooks, jobs, schedules, compute, and permissions, in YAML files stored in Git. The same bundle is deployed to each environment with one command. 

sales-data-platform 
├── databricks.yml              <- bundle definition 
├── resources/ 
│     └── sales_job.yml         <- job definition 
├── src/ 
│     ├── bronze_sales.py 
│     └── silver_sales.py 
└── azure-pipelines.yml         <- CI/CD pipeline 

databricks.yml 

bundle: 
  name: sales_data_platform 
 
include: 
  – resources/*.yml 
 
targets: 
  dev: 
    mode: development 
    default: true 
    workspace: 
      host: https://adb-1111111111111111.11.azuredatabricks.net 
 
  prod: 
    mode: production 
    workspace: 
      host: https://adb-2222222222222222.22.azuredatabricks.net 
      root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target} 
    run_as: 
      service_principal_name: <service-principal-application-id> 

resources/sales_job.yml 

resources: 
  jobs: 
    sales_daily_job: 
      name: sales_daily_job 
      tasks: 
        – task_key: bronze_load 
          notebook_task: 
            notebook_path: ../src/bronze_sales.py 
 
        – task_key: silver_transform 
          depends_on: 
            – task_key: bronze_load 
          notebook_task: 
            notebook_path: ../src/silver_sales.py 

azure-pipelines.yml 

The pipeline authenticates through an Azure Resource Manager service connection backed by a Microsoft Entra ID service principal that has access to the Databricks workspace. 

trigger: 
  branches: 
    include: 
      – main 
 
pool: 
  vmImage: ubuntu-latest 
 
steps: 
  – script: | 
      curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh 
    displayName: Install Databricks CLI 
 
  – task: AzureCLI@2 
    displayName: Validate and deploy bundle 
    inputs: 
      azureSubscription: databricks-prod-connection 
      scriptType: bash 
      scriptLocation: inlineScript 
      inlineScript: | 
        export DATABRICKS_AUTH_TYPE=azure-cli 
        databricks bundle validate -t prod 
        databricks bundle deploy -t prod 

Every merge to main validates the bundle and deploys it to Production. To require a manual approval first, move the deploy step into a deployment job that targets an Azure DevOps environment with an approval check. 

Common Issues 

  • Authentication fails: the PAT has expired or lacks Code (Read & write) scope. Renew it, or switch to Entra ID. 
  • Entra ID access denied: the tenants differ, or a conditional access policy blocks access. Use a PAT for cross-tenant setups. 
  • Merge conflicts: two people edited the same lines. Pull often, keep branches short-lived, and split large notebooks. 
  • Production job runs old code: the job points to a personal folder, or the pipeline did not run. Point jobs only to deployed paths. 
  • On-premises Azure DevOps Server not reachable: use the Git proxy server option for Git folders. 

Best Practices 

  • Develop only in your personal Git folder; production changes only through pipelines. 
  • Prefer Microsoft Entra ID over PATs. 
  • Run pipelines and production jobs as service principals, not users. 
  • Keep secrets out of Git; use Databricks secret scopes (dbutils.secrets.get). 
  • Make small commits with meaningful messages. 
  • Use bundles so jobs and compute are version-controlled, not just notebooks. 

Key Takeaways 

  • Git folders connect the Databricks workspace directly to Azure DevOps repositories. 
  • Use Microsoft Entra ID authentication when both services share a tenant; use a PAT only for cross-tenant setups. 
  • Developers work on feature branches in personal Git folders, and changes reach main only through reviewed pull requests. 
  • Azure Pipelines with Declarative Automation Bundles deploy notebooks, jobs, and settings consistently across environments. 

Leave a Comment

Your email address will not be published. Required fields are marked *

Your email will not be published.

Chat on WhatsApp Call Us Now