Databricks Lakeflow Jobs add AI Function tasks in Beta that process only new inputs

Databricks has added a new task type to Lakeflow Jobs that runs an AI function over every file in a Unity Catalog volume or every row of a table, and then keeps the results current on each scheduled run. The feature appeared in the Databricks platform release notes on November 4, 2026, under the name AI Function tasks for jobs, and it is in Beta. Three functions are supported at launch: ai_parse_document for turning documents into structured text and tables, ai_extract for pulling fields defined in a schema, and ai_classify for assigning labels you provide.

The practical change for data teams is that document parsing, field extraction and classification no longer have to be written as SQL queries wrapped in a job. The task is configured in the Jobs UI with an input, an output location and a function, and Databricks documents that each run processes only the inputs that were added, changed or deleted since the previous run. Results land in a Unity Catalog streaming table that the task creates and manages, so a downstream task, a dashboard query or a Genie space can read them like any other governed table.

The same week brought two related notices that affect anyone planning to use AI Functions in production. From November 1, 2026, spend on task-specific AI Functions counts toward Unity Gateway budgets, and from January 2027 AI Functions in SQL queries will stop working on Databricks Runtime versions below 17.3 LTS.

··········

WHAT THE DATABRICKS AI FUNCTION TASK DOES.

According to the Databricks documentation, an AI Function task applies one AI function to each input and writes one result per file or per row to an output table. In the Jobs UI the three task types appear under a group called Document intelligence: Document Parsing runs ai_parse_document, Information Extraction runs ai_extract, and Classification runs ai_classify. In the Jobs API and in Declarative Automation Bundles the task type is ai_function_task, and the function setting decides which of the three it performs.

Inputs depend on the function. Document Parsing reads files from a volume, or from a FILE or BINARY column in a table. Information Extraction and Classification read a text or VARIANT column, which can be the response column produced by an earlier parsing task. Databricks gives three example uses: parsing new contracts added to a volume each night, extracting invoice fields from documents that a parse task has already processed, and classifying support tickets by urgency before a later task routes them.

Tasks can be chained. A later task can reference the output with the dynamic value reference shown below, and when a new AI Function task is placed after another one, the documentation says its input defaults to the earlier output table and its response column. A typical pipeline therefore parses PDFs in one task, extracts fields from the parsed text in a second task and classifies the result in a third, each step incremental.

{{tasks.<task_name>.output.table_name}}

··········

AVAILABILITY AND REQUIREMENTS FOR AI FUNCTION TASKS IN LAKEFLOW JOBS.

The feature is in Beta and has to be switched on. A workspace admin enables Lakeflow AI Function Task on the Previews page; the documentation notes that if Document intelligence does not appear in the task list, the preview is most likely off. The AI functions must also be available in the workspace region, and Databricks points to its feature regional availability list instead of naming regions on the task page. The output schema has to exist before the first run.

The job's Run as identity needs a specific set of Unity Catalog privileges, which the documentation lists in full.

........

Privilege	Needed on	When
USE CATALOG, USE SCHEMA	Input and output catalogs and schemas	Every run
SELECT or READ VOLUME	Input table or input volume	Every run
CREATE TABLE	Output schema	First run
MODIFY	Output table	Later runs
EXECUTE on the function in system.ai	AI function	If AI Functions access is restricted

........

Table inputs have two further prerequisites. Change data feed must be enabled on the input table, and row tracking is also required when no ID column is configured and the input column is not of FILE type. The documented statements are these:

ALTER TABLE <table-name> SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
ALTER TABLE <table-name> SET TBLPROPERTIES (delta.enableRowTracking = true);

The task page does not state the compute type, file size limits, page limits or row limits, and Databricks has not published a separate price for the task itself. Billing follows the general AI Functions rules: the compute running the work is billed, and task-specific functions with model inference carry an additional charge for Databricks-managed serverless GPU inference. In the system.billing.usage table, ai_classify, ai_extract and ai_parse_document appear under AI_FUNCTIONS.

··········

HOW INCREMENTAL PROCESSING WORKS IN THE OUTPUT TABLE.

The output is a streaming table with three documented parts: an ID column that identifies the input (for example the file path), a response column holding the full function result as VARIANT, and an error column that is NULL on success. Optional values such as confidence scores, rationales and citations are kept inside response and selected from it in a query, never split into separate columns. Joining on the ID column brings results back together with the source data.

For table inputs an optional ID column matches results to source rows across runs and is copied to the output. Without one, rows are identified by their row tracking ID. For a FILE column, rows are identified by file URI, which means row tracking is not needed and rows that reference the same file share one result. For volume inputs, file events on the volume's external location are optional but speed up detection of new and changed files.

To show what incremental processing means in practice, the table below follows an illustrative volume of contracts across three nightly runs. The numbers are invented for the example and the arithmetic was checked in Python.

........

Run (illustrative)	New files	Changed files	Deleted files	Files processed	Rows in output table
Run 1	1,200	0	0	1,200	1,200
Run 2	85	12	4	97	1,281
Run 3	60	0	9	60	1,332

........

Across the three runs the task sends 1,357 files to the function. Reprocessing the full volume every night would have meant 3,813 function calls for the same final state, which is the cost difference incremental processing is designed to avoid. Deleted files are handled too: the documentation states that each run deletes results for inputs that were removed.

··········

SETTINGS CHANGES, REPROCESSING AND CONCURRENCY LIMITS.

Changes to task settings, such as the extraction fields, the classification labels or the instructions, apply only to inputs processed after the change. Existing rows keep the result produced under the old settings. To apply new settings to every input, the documentation says to enter a new Table name, which starts a fresh output table. Changing the input or the output table makes the next run reprocess all inputs, and Databricks warns that this adds cost.

The output table name is optional. If it is left blank, the task generates a name that begins with parsed, extracted or classified and stays the same across runs. A run fails if a table with the chosen name already exists and was not created by the task, which protects existing tables from being overwritten.

Two operational limits are documented. A task allows only one active run, so starting a new run while one is in progress makes the new run fail. Runs also share the workspace's AI Functions capacity and slow down when many run at once, and Databricks recommends keeping about 20 or fewer runs active at a time. During a run, the task graph shows the percent processed and the run output shows Rows, Succeeded, Failed and ETA; after the run, Sample Data shows output rows.

For teams already working with documents in Databricks, the task sits alongside features covered earlier on Data Studios, such as Genie Agents answering from files in Unity Catalog volumes and native XML support in Auto Loader, read_files and COPY INTO. Comparable document functions exist on other platforms, for example BigQuery's AI.PARSE_DOCUMENT and Snowflake's AI_EXTRACT.

··········

BUDGETS AND THE RUNTIME 17.3 LTS DEADLINE FOR AI FUNCTIONS.

Starting November 1, 2026, Unity Gateway budgets include spend from task-specific AI Functions such as ai_parse_document, ai_extract and ai_classify, in both batch and real-time execution. Budgets can set shared limits, per-user limits and per-user overrides, and can either send an alert or block further usage until the budget resets or an admin raises the threshold. Databricks describes blocking as approximate because it relies on near-real-time cost estimates, and states that the feature should not be used to guarantee an absolute spend cap. Task-specific AI Function requests count toward budgets but are not recorded in system.ai_gateway.usage; their billed usage is in system.billing.usage.

The runtime notice was published on November 5, 2026. Starting January 2027, AI Functions used in SQL queries will no longer work on Databricks Runtime versions below 17.3 LTS. The change does not affect ai_similarity or the AI Functions REST API. The AI Functions documentation currently lists Databricks Runtime 15.4 LTS as the minimum and recommends 18.2 or above, and it states that AI Functions are not available on Classic SQL warehouses. Teams running notebooks or jobs on older clusters should plan the upgrade before January.

··········

DATABRICKS AI FUNCTION TASKS: WHAT TO CONFIRM BEFORE A PRODUCTION PIPELINE.

Databricks has confirmed the three supported functions, the incremental behavior for new, changed and deleted inputs, the output table structure, the permissions, the one-active-run limit and the recommendation of about 20 concurrent runs. The feature is Beta, enabled per workspace from the Previews page, and depends on AI Functions being available in the workspace region.

What depends on the customer's setup is regional availability, whether AI Functions access is restricted through EXECUTE on system.ai, whether input tables already have change data feed and row tracking enabled, and how Unity Gateway budgets are configured. The documentation does not disclose file size, page or row limits for the task, the compute type it uses, or a task-specific price, so cost estimates should come from system.billing.usage on a pilot run. Any change to extraction fields or labels after go-live needs a deliberate decision, because only new inputs pick it up unless the output table is rebuilt under a new name.

··········

FOLLOW US FOR MORE.

·····

DATA STUDIOS

·····

datastudios.org