Introduction
Managing cloud costs efficiently is a critical aspect of operating in Databricks, especially when leveraging serverless compute. Budget Policies provide a structured way to track, monitor, and manage serverless usage across different workloads, such as Notebooks, Jobs, DLT Pipelines, and Model Serving. While Budget Policies do not directly enforce spending limits, they enable organizations to attribute costs accurately and integrate with Databricks Budget for more advanced cost control mechanisms. This guide walks you through the steps to create, manage, and attach Budget Policies in Databricks, ensuring greater visibility and governance over your cloud spending.
Budget Policy is a Databricks feature that is in Public Preview (as of February 10, 2025). Budget Policies are basically sets of tags that you can apply to a serverless workload: the workload could be a notebook that runs on serverless compute, a job, or a DLT pipeline that runs on serverless compute. Currently, Budget Policies cannot be attached to SQL Queries in Databricks. So Budget Policies help attribute serverless usage to a specific budget, which is especially important for tracking your Databricks spending.
Enable Budget Policy Feature
Since the feature is not in General Availability, you need to go to the Databricks Account Console and check the Previews section on the left menu. There, you will find the Budget Policy, and you have an option of turning this feature on or off. There, you will find a link to the documentation about the Budget Policy and have an option of sending feedback to the Databricks team about the feature.
Enable Billing Admin Permission
Budget Policy requires you to be an admin in a Databricks workspace to create a budget policy. Another way to create a budget policy is to be assigned the Budget Policy: Manager permission. A Databricks admin can grant you this permission. Even if you are a Databricks admin, it is likely that you do not have the Budget Policy: Manager permission.
To enable it for yourself or a team member, go to the Databricks Account Console, select the User Management section on the left menu, and search for the user (yourself or a team member) under the Users tab. Click on the user’s name and navigate to the Roles tab. Under the Roles tab, you have the option to toggle on or off the Account Admin, Marketplace Admin, and Billing Admin roles. Toggle on the Billing Admin role to enable yourself or a team member to create budget policies.
You can also assign the Billing Admin role to a service principal. To do this, search for and find the service principal under the Service Principals tab on the User Management page, and toggle on the Billing Admin role on the service principal’s profile page.
Groups are not eligible to be assigned these roles.
Create a Budget Policy
Now that you have all the necessary permissions, you can create a budget policy. Go to the Databricks workspace and click on your profile located in the top left corner. Click on Settings and choose Compute from the menu on the left. In the Policies section, you will find Budget Policies. Click on the Manage button. It will take you to the Budget Policies page. Click on the Create button and provide a name for your budget policy. Add tags that will be used for attribution and finalize it by clicking on the Create button.
It states that Budget Policies only support fixed tags, which I interpret to mean that the tag needs to have both a key and a value.
You can edit an existing policy or delete it from this page. You can add multiple tags to a budget policy and create multiple budget policies.
Although a budget policy is created in the workspace settings, it is available across all workspaces you have access to within your Databricks account.
Once you create a budget policy, you will have the Budget Policy: Manager role automatically assigned. Now you should grant other users permission to use or manage the budget policy. To do that, click on the budget policy you created. It will show two tabs: the Configuration tab, where you can see the name of the budget policy, who created it, and the tags associated with it. Click on the Permissions tab, and you will see your name (or the person who created the budget policy or the people who have permissions) and their roles. To grant access to other users (or service principals or user groups), click on the Grant Access button. Find the principal from the search box and assign the desired role.
(User) Principal refers to a user, a service principal, or a user group. There are two types of roles that can be assigned to a principal: Budget Policy: User and Budget Policy: Manager. The User role is only able to attach the budget policy to the workload (a notebook, DLT pipeline, job, or model serving). The Manager role is able to edit and delete the budget policy.
Attach a Budget Policy
Notebook
To attach a Budget Policy to a workload, a user needs to have permission to use the Budget Policy. If the Budget Policy does not appear in the notebook Environment tab or the Job Details UI, ensure the user has the appropriate permissions to use the Budget Policy.
Create a notebook or open an existing notebook in the Databricks workspace. Click on the Connect button on the top right and select Serverless as the compute option. Click on the Environment icon from the menu on the right (Environment is the last icon). It will open the Serverless Environment window on the right side of the screen. Click on the drop-down for Budget Policy and select your budget policy.
When you select a budget policy, it will automatically display and update Budget Policy tags underneath the budget policy name. Click the Apply button at the bottom right of the side window. Once you are done, you can minimize the right-side window by clicking on the X button.
You can additionally set up the environment version, the base environment, and install dependency libraries and packages. However, these are not covered in this document.
Jobs
To attach a Budget Policy to a job, select the relevant policy on the Job Details page. Ensure that the user principal in the Runs As section has the appropriate permissions to use the Budget Policy. If the job is set up to run as a service principal, the service principal must have permission to use the Budget Policy.
Budget policies can be attached to Notebooks, Jobs, DLT Pipelines, and Model Serving. Currently, Databricks does not support attaching a budget policy to an SQL query or an SQL Warehouse (endpoint).
Limitations
- Only one budget policy can be attached per workload (a notebook, job, DLT pipeline, or a Model Serving) at a time.
- Budget policies by themselves do not enforce any restrictions. They are primarily used for tracking and monitoring serverless usage. However, they can be used for the enforcement of a certain rule or action when used in conjunction with Databricks Budget (Databricks Budget is not covered in this article).
Tagging Guidelines for Budget Policies
When creating the Budget Policies, I had to carefully consider what we wanted to track and how to categorize it to ensure accurate tracking of serverless costs. The fact that only one budget policy can be applied to a workload makes this challenging. To provide an accurate breakdown of serverless usage costs, the budget policies you create should be mutually exclusive.
Initially, I considered development and production budget policies. However, every workflow in our workspaces falls into one of these categories, making any additional budget policies redundant. I also thought about categorizing usage by users versus service principals, but this approach felt too broad.
Ultimately, I decided to create dbt, development, and ingestion budget policies. Tracking serverless usage for dbt workflows is crucial because they run in Databricks workflows and primarily handle the data transformation layer. Similarly, I wanted to track ingestion workloads (jobs) separately. In my case, dbt and ingestion costs would account for almost all serverless job costs. Any user workloads—such as notebooks, testing jobs, and ad hoc executions—can be attributed to the development budget policy.
This list is by no means exhaustive. There are several areas where we use serverless that won’t be captured due to the current limitations of Budget Policies—for example, they do not track SQL queries, dashboards, or API calls to the warehouse.
Conclusion
Budget Policies in Databricks serve as a powerful tool for tracking and managing serverless usage across multiple workloads. While they do not enforce spending limits by themselves, they play a crucial role in cost attribution, monitoring, and potential enforcement when used in conjunction with Databricks Budget. By correctly setting up and managing Budget Policies, organizations can gain better control over cloud expenses, optimize resource allocation, and enhance overall cost governance within their Databricks environment. As Databricks continues to evolve, future enhancements may expand the capabilities of Budget Policies, further strengthening cost management strategies.
Author: Dilorom Abdullah