Enterprise generative AI engineering is no longer limited to writing prompts and connecting an application to a large language model. A production AI system must combine software engineering, governed data access, model routing, retrieval, tool execution, evaluation, monitoring, security, cost control and continuous delivery.

Databricks addresses these requirements through an integrated set of capabilities covering the complete GenAI lifecycle: querying foundation models and agents, building custom and low-code agents, connecting agents to governed tools, preparing structured and unstructured data, implementing retrieval with AI Search, deploying agents and applications, governing traffic through Unity AI Gateway, tracing with MLflow, evaluating and monitoring quality, operationalising through LLMOps, and delivering governed experiences through Genie One.

Data, models, prompts, tools, traces, vector indexes and application assets can be governed through Unity Catalog, while MLflow provides the quality and observability layer. The sections below turn that platform story into an engineering playbook.

1. The Databricks Generative AI Capability Model

The Databricks GenAI platform can be understood as six connected layers.

1.1 Experience layer​

This is where developers, reviewers and business users interact with AI.

It includes:

• AI Playground for prompt experimentation and agent prototyping.
• Databricks Apps for deploying custom user interfaces and APIs.
• Review App for expert evaluation and feedback.
• Genie One for business-facing analytics and AI interaction.
• External applications that call Databricks APIs.

AI Playground is designed for comparing models, experimenting with parameters and testing tool-calling behaviour before moving into application code. Databricks Apps provides a more flexible production surface for custom agent applications.

1.2 Agent and application layer​

This layer contains the application logic:

• Prompts and system instructions.
• Conversation management.
• Agent decision logic.
• Retrieval orchestration.
• Tool selection.
• Multi-agent coordination.
• Validation and response formatting.
• Human approval workflows.

Databricks supports no-code agent prototyping, guided Knowledge Assistant development, supervisor agents and fully custom agents written in Python. Custom agents can use frameworks such as LangGraph, LangChain, OpenAI Agents SDK and LlamaIndex.

1.3 Model and inference layer​

This layer provides access to:

• Databricks-hosted foundation models.
• External models from providers such as OpenAI, Anthropic and Google.
• Custom models.
• Fine-tuned open-source models.
• Agents deployed through Databricks Apps or Model Serving.

Applications can access these resources through the Databricks OpenAI client, OpenAI-compatible REST APIs, Databricks SDKs, SQL and AI Functions.

1.4 Data and tool layer​

This layer enables an AI system to move beyond text generation.

Agents can:

• Search documents through AI Search.
• Query structured data through Genie or Databricks SQL.
• Execute Unity Catalog functions.
• Connect to managed or external MCP servers.
• Call REST APIs through governed Unity Catalog connections.
• Access SaaS platforms such as Slack, GitHub, Google Drive and SharePoint.

This is where a chatbot becomes an operational agent capable of obtaining information and performing controlled actions.

1.5 Observability and quality layer​

MLflow 3 provides:

• End-to-end tracing.
• Prompt and application versioning.
• Evaluation datasets.
• Built-in LLM judges.
• Custom judges.
• Deterministic scorers.
• Human feedback.
• Development-time evaluation.
• Production monitoring.

Tracing is the foundation of this layer because evaluation and production monitoring operate on recorded application traces.

1.6 Governance layer​

The governance layer consists primarily of:

• Unity Catalog for asset governance.
• Unity AI Gateway for traffic governance.
• Service policies for behavioural governance.
• System tables and inference tables for usage, cost and audit analysis.

Together, these services control who can access an AI asset, what requests can be made, which tools can be called, what content is permitted and how much the organisation can spend.

2. Python and Software Engineering for Databricks AI Systems

Python remains the primary engineering language for custom Databricks agents, MLflow evaluation, model serving integration and AI Search clients. Expert Python capability in this context means considerably more than creating notebooks.

2.1 Modular agent architecture​

A production agent should separate concerns rather than placing the prompt, retrieval, tools and deployment code in one file.

A maintainable project might use the following structure:

This structure allows model access, retrieval, tool calling, policy enforcement and evaluation to evolve independently.

2.2 The ResponsesAgent abstraction​

Databricks recommends the MLflow ResponsesAgent interface for custom agent development. It acts as a framework-neutral compatibility layer around agents built with Python, LangGraph, LangChain, OpenAI Agents SDK or another orchestration library.

The interface provides support for:

• Streaming responses.
• Multi-agent systems.
• Intermediate tool-call history.
• Tool confirmation.
• Long-running tools.
• Typed Python authoring.
• Automatic trace aggregation.
• Integration with evaluation and monitoring.
• Compatibility with the OpenAI Responses schema.

The architectural value is decoupling. Your internal orchestration framework can change without requiring the surrounding Databricks evaluation, tracing and deployment systems to be redesigned.

2.3 MLflow AgentServer​

The Databricks agent templates include an asynchronous FastAPI-based MLflow AgentServer. It handles request routing, observability, errors and the standard response endpoint.

This prevents the engineering team from having to implement basic serving infrastructure repeatedly. The team can focus on business-specific agent logic while retaining a consistent serving contract.

2.4 Local development and Git-based delivery​

Databricks Apps supports local IDE development and Git-based versioning. The official agent template can be developed locally and deployed declaratively using Databricks Asset Bundles.

A disciplined workflow would normally include:

Databricks Asset Bundles define application code, experiments, serving dependencies and permissions in databricks.yml. Databricks distinguishes between deploying the bundle and starting or restarting the application, so release automation should explicitly execute both operations.

2.5 Testing strategy​

Traditional software tests are necessary but insufficient for GenAI systems.

Unit tests​

Unit tests should cover deterministic components:

• Prompt construction.
• Input validation.
• Output parsing.
• Retrieval filters.
• Tool parameter validation.
• Policy evaluation.
• Exception handling.
• Cost calculations.
• Model-routing logic.

Contract tests​

Contract tests verify integration boundaries:

• Model API request and response schemas.
• MCP tool discovery.
• Unity Catalog function signatures.
• AI Search result schemas.
• Agent streaming event structures.
• Authentication and permission propagation.

Evaluation tests​

Evaluation tests measure probabilistic behaviour:

• Relevance.
• Correctness.
• Groundedness.
• Safety.
• Tool-selection accuracy.
• Retrieval quality.
• Conversation completeness.
• Policy compliance.

Adversarial tests​

These should cover:

• Prompt injection.
• Retrieval poisoning.
• PII leakage.
• Unauthorised tool calls.
• Excessive tool loops.
• Malformed tool responses.
• Requests to bypass approval.
• Cross-tenant data access attempts.

Performance tests​

Performance testing should measure:

• End-to-end latency.
• Time to first token.
• Retrieval latency.
• Tool latency.
• Tokens per request.
• Concurrent-user behaviour.
• Failure and retry rates.
• Cost per successful interaction.

3. Querying LLMs and Agents

Databricks supports several model-consumption patterns because development, analytics and production application workloads have different needs.

3.1 AI Playground​

AI Playground is the interactive prototyping environment.

It can be used to:

• Compare models side by side.
• Test prompts.
• Change temperature and token limits.
• Prototype question-answering systems.
• Test tool-calling agents.
• Validate model behaviour before writing application code.

It is particularly useful during discovery because product owners and domain experts can participate without needing to understand SDKs or deployment infrastructure.

Playground should not replace automated evaluation. It is best treated as an exploratory environment for generating hypotheses that later become evaluation cases.

3.2 System-provided model services​

Preconfigured model services are available through the system.ai catalog. These are appropriate for experimentation and straightforward use cases because the organisation does not need to create its own routing configuration.

3.3 Custom model services​

Custom model services are more appropriate where the organisation needs:

• Multiple underlying models.
• Traffic distribution.
• Provider fallback.
• Central governance.
• Controlled model upgrades.
• Business-unit-level cost attribution.
• A stable endpoint name independent of the underlying provider.

The model-service abstraction makes it possible to change the backing model without forcing every consuming application to change its code.

3.4 Query interfaces​

Databricks supports three primary production query patterns.

Databricks OpenAI Client​

This is the recommended interface for new applications. It supports native streaming and provides access to the complete Databricks feature set.

OpenAI-compatible REST API​

This is appropriate when:

• The application is not written in Python.
• An existing application already uses an OpenAI-compatible interface.
• A platform team wants a language-neutral integration contract.

SQL and ai_query​

ai_query allows SQL-based workloads to invoke supported models. This is particularly useful for batch enrichment and data engineering, rather than interactive conversational applications.

4. Building Enterprise AI Agents

4.1 Choosing the right agent-development approach​

Databricks supports several levels of abstraction.

AI Playground​

Use AI Playground when the goal is rapid prototyping with limited custom orchestration.

Knowledge Assistant​

Use Knowledge Assistant for domain-specific knowledge experiences where guided development and optimisation are preferable.

Supervisor Agent​

Use a supervisor when the application must coordinate multiple governed agents and tools, such as Genie Agents, custom endpoints, MCP servers and Unity Catalog functions.

Custom agent​

Use a custom agent when the system requires:

• Complex decision logic.
• Custom state management.
• Multi-agent coordination.
• Non-standard API behaviour.
• Custom approval workflows.
• Advanced streaming.
• Git-based delivery.
• Local development.
• Framework-specific functionality.

Databricks Apps provides full control over application code, server behaviour and deployment workflows for this pattern.

4.2 Core custom-agent architecture​

A production custom agent commonly contains the following stages:

The model should not be treated as the entire application. It is one decision-making component surrounded by deterministic controls.

4.3 Streaming​

Streaming improves perceived responsiveness by returning partial output before the complete answer is ready.

In the ResponsesAgent model, streaming consists of incremental output events followed by a final completion event. The final event is important because Databricks uses it to:

• Complete the MLflow trace.
• Aggregate the response for inference tables.
• Display the complete response in supported user interfaces.

Applications should also handle errors that occur after streaming begins. A request can successfully return several tokens and then fail during a tool call, policy check or downstream operation.

4.4 Custom inputs and outputs​

Enterprise applications often need additional fields beyond chat messages.

Custom inputs might include:

• Tenant identifier.
• User role.
• Session identifier.
• Communication channel.
• Region.
• Consent status.
• Risk classification.
• Client application.

Custom outputs might include:

• Source references.
• Confidence indicators.
• Approval status.
• Tool-execution summaries.
• Escalation reason.
• Policy decision.
• Audit metadata.

The ResponsesAgent interface supports custom input and output fields for these scenarios.

4.5 Authentication models​

Databricks Apps supports two important authorisation patterns.

App authorisation​

The application runs using a service principal. All users effectively share the application’s configured resource permissions.

This is suitable when:

• Users should have the same data access.
• The application accesses a controlled service dataset.
• The application itself is the security boundary.

User authorisation​

The application acts on behalf of the current user.

This is suitable when:

• Data visibility differs by user.
• The agent should respect each user’s Unity Catalog privileges.
• User-level auditability is required.
• An action must be attributed to an individual.

User credentials must be obtained inside the request-handling path rather than during application startup because they exist only in the context of an authenticated request.

5. Connecting Agents to Tools

An LLM produces language. A tool-enabled agent can retrieve information, query systems and perform operations.

Databricks currently recommends MCP for most agent integrations, while also supporting direct REST access and Unity Catalog functions.

5.1 Managed MCP servers for Databricks​

Databricks provides managed MCP servers for:

• Genie.
• AI Search.
• Databricks SQL.
• Unity Catalog functions.

These services give agents access to governed Databricks resources without requiring the engineering team to build and host a separate MCP server.

Each managed service has a dedicated endpoint and OAuth scope. Access remains controlled through Unity Catalog.

Genie MCP​

This allows an agent to ask natural-language questions about structured data through a governed Genie Agent.

Example use case:

A customer-service agent receives a question about a delayed order. It uses Genie to query order, warehouse and delivery tables without embedding SQL-generation logic directly into the main agent.

AI Search MCP​

This enables document search across governed AI Search indexes.

Example use case:

An HR assistant retrieves relevant policy sections before answering an employee’s leave question.

Databricks SQL MCP​

This provides controlled SQL execution against Unity Catalog data.

Unity Catalog Functions MCP​

This exposes approved Python or SQL functions as callable tools.

Example functions might include:

• Calculate a refund.
• Validate customer eligibility.
• Create a support case.
• Retrieve a risk score.
• Check product availability.

5.2 External MCP servers​

External MCP servers can connect Databricks agents to services such as Slack, GitHub, Google applications and other SaaS systems.

Registering an external MCP server as a Unity Catalog MCP Service adds:

• Governed discovery.
• Access grants.
• Tool filtering.
• Per-user authentication.
• Service policies.
• Traffic visibility through Unity AI Gateway.

This is materially safer than allowing each development team to embed API credentials and unrestricted tool definitions in application code.

5.3 Managed OAuth​

Databricks provides managed OAuth for selected providers, including read-only integrations with Google Drive, Gmail, Google Calendar and Microsoft services through SharePoint and Microsoft Graph.

Managed OAuth is especially useful during development because the engineering team does not need to create and maintain its own OAuth application. The documentation recommends considering custom provider credentials where production requirements demand greater control.

5.4 Unity Catalog connections proxy​

The Unity Catalog connections proxy allows an agent to use an external provider’s normal SDK while routing requests through Databricks.

The application authenticates to Databricks. Databricks then injects the external service credentials associated with the governed connection. The application code does not directly handle the external provider token.

Structurally, this provides:

• Central credential management.
• Reduced secret exposure.
• Easier credential rotation.
• Unity Catalog access controls.
• A consistent outbound integration pattern.

5.5 Unity Catalog function tools​

Unity Catalog functions are useful when a tool can be represented as a controlled Python or SQL operation.

They are appropriate for:

• Deterministic calculations.
• Governed data queries.
• Controlled API wrappers.
• Reusable business rules.
• Functions requiring explicit grants.

A tool description should clearly define:

• What the tool does.
• When it should be used.
• Required parameters.
• Allowed values.
• Expected output.
• Side effects.
• Whether approval is required.

Poorly described tools increase the probability that the model selects the wrong operation or supplies invalid parameters.

5.6 Local Python tools​

Local tools run inside the agent process.

They are suitable for:

• Date calculations.
• Text transformations.
• Schema conversion.
• Validation.
• Lightweight deterministic calculations.
• Formatting.

They should not be used to bypass Unity Catalog governance for operations that access protected data or external systems.

6. Databricks AI Search

Databricks AI Search, previously called Databricks Vector Search, provides retrieval for RAG systems, recommendations and other similarity-based applications. It creates governed search indexes from Delta tables and can synchronise indexes as source data changes.

6.1 How AI Search works​

A typical RAG flow is:

AI Search uses the HNSW approximate-nearest-neighbour algorithm and L2 distance for vector similarity. Where cosine-style ranking is required, embeddings should be normalised before indexing.

6.2 Hybrid search​

Pure semantic search can struggle with exact identifiers such as:

• Product codes.
• Policy numbers.
• Error codes.
• Customer IDs.
• Abbreviations.
• Legal clause references.

Hybrid search combines semantic similarity with keyword matching. Databricks uses BM25 for keyword relevance and Reciprocal Rank Fusion to combine keyword and vector rankings.

For enterprise RAG, hybrid search is often preferable when documents contain a mixture of natural language and exact business terminology.

6.3 Index options​

Delta Sync with Databricks-managed embeddings​

The source Delta table contains text. Databricks computes embeddings using a selected embedding model and keeps the index synchronised with the source table.

Use this when:

• The organisation wants minimal embedding infrastructure.
• The embedding model is supported.
• Operational simplicity is more important than custom embedding logic.

Delta Sync with self-managed embeddings​

The source Delta table contains embeddings generated by your own pipeline. Databricks synchronises the index as the table changes.

Use this when:

• A specialist embedding model is required.
• Embeddings are produced elsewhere.
• The organisation needs complete control over preprocessing and embedding versions.

A self-managed index cannot be converted in place to a Databricks-managed embedding index; a new index and new embeddings are required.

Direct Vector Access Index​

The application updates the index directly through APIs.

Use this when:

• Data does not originate from a Delta table.
• Updates are controlled by an external application.
• The organisation requires explicit upsert and delete behaviour.

The engineering team becomes responsible for consistency between the source system and index.

Dedicated full-text index​

Storage-optimised endpoints can provide a dedicated BM25 full-text index without vector embeddings. This feature is currently documented as Beta.

Use it where the dominant requirement is exact keyword retrieval rather than semantic similarity.

6.4 Standard versus storage-optimised endpoints​

Standard endpoints are appropriate for high sustained query throughput and support high-QPS configurations.

Storage-optimised endpoints support considerably larger indexes and faster indexing, but introduce somewhat higher query latency and several operational limitations. Databricks documents capacity beyond one billion 768-dimensional vectors, indexing that can be 10–20 times faster and approximately 250 milliseconds of additional query latency for storage-optimised configurations.

The decision should therefore be based on:

• Corpus size.
• Update frequency.
• Query volume.
• Latency target.
• Synchronisation requirements.
• Embedding dimensions.
• Cost profile.

6.5 Retrieval-quality engineering​

A production retrieval system should evaluate more than whether a document was returned.

Important metrics include:

• Recall at K.
• Precision at K.
• Mean reciprocal rank.
• Normalised discounted cumulative gain.
• Context relevance.
• Context coverage.
• Retrieval groundedness.
• Citation correctness.
• Empty-result rate.
• Filter-rejection rate.
• Reranking improvement.
• Retrieval latency.

The evaluation dataset should include:

• Normal user questions.
• Exact identifier searches.
• Ambiguous queries.
• Questions with no answer.
• Questions requiring multiple documents.
• Questions involving outdated documents.
• Permission-sensitive questions.

6.6 Security considerations​

AI Search indexes and endpoints are governed resources. Requests are authenticated, authorised and encrypted.

However, the current documentation notes that native row-level and column-level permissions are not supported directly by AI Search. Application-level access control can be implemented through metadata filters.

In a multi-tenant application, every indexed chunk should therefore contain security metadata such as:

tenant_id
business_unit
country
document_classification
user_group
effective_from
effective_to
access_level

Retrieval filters must be derived from trusted server-side identity information—not from values supplied by the user or generated by the LLM.

7. Databricks AI Functions

AI Functions bring GenAI operations into SQL, notebooks, Lakeflow pipelines and Workflows.

They are designed for large-scale data transformation and enrichment rather than only interactive chat.

7.1 Task-specific functions​

Task-specific functions implement predefined operations using Databricks-managed systems.

Important capabilities include:

ai_parse_document​

Parses text, tables, layout and figure descriptions from unstructured documents.

This can be the first stage of an intelligent document-processing pipeline.

ai_extract​

Extracts structured fields using a schema.

Example:

SELECT ai_extract(
    contract_text,
    array(
        'supplier_name',
        'contract_start_date',
        'contract_end_date',
        'termination_notice_period'
    )
)
FROM contracts;

ai_classify​

Classifies text into a supplied set of labels.

Example use cases:

• Customer-support routing.
• Risk classification.
• Complaint identification.
• Product categorisation.
• Document classification.

ai_prep_search​

Transforms parsed document output into chunks designed for AI Search and RAG ingestion. It is currently documented as Beta.

7.2 Text transformation and analysis​

Other task-specific functions include:

• ai_fix_grammar
• ai_translate
• ai_summarize
• ai_mask
• ai_analyze_sentiment
• ai_similarity
• ai_gen
• ai_forecast
• ai_top_drivers
• vector_search

7.3 General-purpose ai_query​

Use ai_query where the task requires:

• A custom prompt.
• A specific model.
• Flexible response schemas.
• Application-specific generation behaviour.
• A model not covered by a task-specific function.

Task-specific functions should generally be preferred where they directly match the requirement because they reduce prompt engineering and operational complexity.

7.4 Batch inference architecture​

AI Functions are especially useful for:

• Enriching millions of records.
• Processing document backlogs.
• Running scheduled classification.
• Generating summaries for analytics.
• Preparing RAG corpora.
• Producing structured metadata.

An example production flow might be:

7.5 Monitoring and cost​

Progress can be reviewed through the SQL query profile, including completed inferences, failed inferences and total processing time.

AI Function consumption can also be analysed through Databricks system billing tables. Batch model workloads are represented through model-serving batch-inference usage, while some document and extraction functions use the AI Functions billing product classification.

8. MLflow Tracing

MLflow Tracing provides end-to-end observability for GenAI applications and agents.

A trace records:

• Application input.
• Model requests.
• Model responses.
• Retrieval steps.
• Tool calls.
• Intermediate decisions.
• Latency.
• Token usage.
• Errors.
• Metadata.
• Final output.

8.1 Trace and span model​

A trace represents one complete application interaction.

Spans represent individual operations within it.

For example:

Without tracing, the team sees only the final answer. With tracing, the team can determine whether the failure came from retrieval, tool selection, model reasoning, validation or infrastructure.

8.2 Instrumentation approaches​

MLflow supports:

• Automatic tracing through framework autologging.
• Manual tracing for custom functions and workflows.
• A combined approach.

Automatic tracing supports more than 20 GenAI frameworks and libraries. Manual spans remain important for business-specific logic that framework-level instrumentation cannot understand.

import mlflow

@mlflow.trace(name="validate_refund_request")
def validate_refund_request(order: dict, policy: dict) -> dict:
    eligible = (
        order["status"] == "delivered"
        and order["days_since_delivery"] <= policy["refund_window_days"]
    )

    return {
        "eligible": eligible,
        "reason": "Within refund period" if eligible else "Outside refund period",
    }

8.3 Trace storage​

Databricks provides two storage patterns.

Unity Catalog trace storage​

This is recommended for new production workloads.

Benefits include:

• No documented trace-count cap.
• Delta-table storage using OpenTelemetry-compatible structures.
• SQL access.
• Unity Catalog permissions.
• Analysis through Spark, dashboards and Genie.
• Interoperability with OpenTelemetry tooling.

Experiment storage​

Traces can also be stored directly in an MLflow experiment, but this option has a documented limit of 100,000 traces per experiment and does not provide the same SQL and OpenTelemetry capabilities.

8.4 Operational use cases​

Tracing supports:

• Debugging.
• Root-cause analysis.
• Cost optimisation.
• Latency analysis.
• Auditability.
• Compliance evidence.
• Evaluation dataset creation.
• Production-quality monitoring.
• User-feedback investigation.

A mature team should attach standard metadata to each trace:

application_version
prompt_version
model_service
retrieval_index_version
environment
tenant
region
user_role
release_id
experiment_id
feature_flags

This makes failures attributable to a specific release or configuration.

9. Evaluating and Monitoring GenAI Applications

Traditional model metrics are not sufficient for agentic applications because outputs are free-form, non-deterministic and produced through multiple interacting components.

MLflow evaluation measures applications such as:

• LLM applications.
• RAG systems.
• Tool-calling agents.
• Multi-agent systems.
• Multi-turn conversational systems.

9.1 Evaluation harness​

mlflow.genai.evaluate() provides a structured evaluation harness.

Conceptually, it receives:

mlflow.genai.evaluate(
    data=evaluation_dataset,
    predict_fn=application,
    scorers=scorers,
    model_id=application_version,
)

The harness:

1. Runs the application against evaluation inputs.
2. Captures traces.
3. Applies judges and scorers.
4. Stores results in an evaluation run.
5. Provides aggregate metrics and trace-level results.

The same evaluation logic can later be applied to production traces.

9.2 Direct evaluation​

In direct evaluation, MLflow calls the application.

This is the preferred mode when the application can be executed in the evaluation environment.

Use cases include:

• Pull-request quality gates.
• Prompt comparison.
• Model comparison.
• Regression testing.
• Release-candidate validation.
• Nightly evaluation.

9.3 Answer-sheet evaluation​

In answer-sheet evaluation, the outputs already exist.

This is useful when:

• The application is hosted externally.
• Historical production responses must be evaluated.
• Running the application is expensive.
• Outputs were produced by a batch process.
• A third party supplied the results.

Existing traces can also be passed directly into the evaluation harness.

9.4 Scorers and LLM judges​

A scorer receives a trace, extracts relevant fields, calculates an assessment and attaches feedback to the trace.

Scores may be:

• Boolean.
• Pass or fail.
• Numeric.
• Categorical.
• Structured feedback.

MLflow supports three broad approaches.

Built-in judges​

These evaluate common dimensions such as:

• Correctness.
• Relevance.
• Safety.
• Retrieval groundedness.
• Other general quality dimensions.

Custom LLM judges​

Custom judges are appropriate for domain-specific criteria.

For example, a financial-services judge might assess:

• Whether required risk disclosures are present.
• Whether financial advice was avoided.
• Whether the agent correctly escalated.
• Whether a regulated claim was supported by an approved source.

Code-based scorers​

These are appropriate for deterministic rules.

Examples:

• The answer must include a source.
• A refund amount must match the calculator output.
• An escalation must occur for a vulnerable customer.
• The response must not expose internal identifiers.
• A specific tool must be called before a decision.

The same scorers can be reused in development and production monitoring.

9.5 Multi-turn evaluation​

Conversational agents require evaluation at the conversation level.

Important measures include:

• Conversation completeness.
• Dialogue coherence.
• User frustration.
• Repetition.
• Escalation timing.
• Resolution rate.
• Context retention.
• Contradiction across turns.

MLflow provides specialised support for multi-turn judges and conversation simulation. Synthetic conversation generation can test behaviours that are too expensive or rare to collect manually.

9.6 Production monitoring​

Production monitoring applies selected scorers to sampled production traces.

An effective monitoring design should combine:

Continuous operational metrics​

• Request count.
• Error rate.
• Latency percentiles.
• Token consumption.
• Tool-call failure rate.
• Retrieval latency.
• Cost per request.

Scheduled quality metrics​

• Relevance.
• Safety.
• Groundedness.
• Correctness.
• Policy compliance.
• Conversation completion.

Event-driven evaluation​

Run additional evaluation when:

• A new prompt is deployed.
• The model is changed.
• The retrieval index is rebuilt.
• A new tool is added.
• Safety rules change.
• A quality alert is triggered.

9.7 Human feedback​

Domain experts and end users can provide feedback through review experiences and APIs.

Human feedback should be used to:

• Build evaluation datasets.
• Identify missing scenarios.
• Validate automated judges.
• Align custom judges.
• Prioritise application improvements.
• Investigate production failures.

MLflow allows expert feedback and automated scoring to participate in the same iterative quality loop.

10. Unity AI Gateway and AI Governance

Unity AI Gateway is the central traffic-control layer for model and MCP requests. As of the referenced July 2026 documentation, Unity AI Gateway and service policies are Beta features and require preview enablement.

Databricks separates AI governance into three dimensions.

10.1 Asset governance​

Unity Catalog governs:

• Models.
• MCP services.
• Functions.
• Connections.
• Tools.
• Other AI assets.

These are securable objects subject to grants and attribute-based access-control policies.

Asset governance answers:

Who is allowed to use this model, tool or connection?

10.2 Traffic governance​

Unity AI Gateway routes model and MCP requests through a central control plane.

It can support:

• Usage tracking.
• Rate limits.
• Spend caps.
• Central provider access.
• Cost attribution.
• Model routing.
• Controlled external AI access.

Traffic governance answers:

How much can this principal consume, and where should the request be routed?

10.3 Behaviour governance​

Service policies inspect individual requests and responses.

A service policy can:

• Allow an interaction.
• Deny an interaction.
• Require approval.

Policies can consider:

• The calling principal.
• Request content.
• Response content.
• The selected tool.
• The operation being attempted.

Behaviour governance answers:

Is this specific interaction permitted?

Databricks documents built-in policy capabilities for risks including PII, prompt injection and unsafe content, while also supporting custom service policies.

10.4 Example governed request​

Consider a coding agent attempting to call a GitHub MCP tool.

A policy might permit repository reading but require approval before creating a pull request or modifying a protected branch.

10.5 Cost and usage governance​

Gateway governance should be designed at multiple levels:

• Account.
• Workspace.
• Business unit.
• Application.
• User group.
• Model service.
• Environment.

Useful controls include:

• Requests per minute.
• Tokens per minute.
• Daily or monthly spend.
• Maximum output tokens.
• Approved model list.
• Approved tools.
• Environment-specific restrictions.

10.6 Monitoring governance activity​

Databricks provides usage and inference records for analysing:

• Who called a model or service.
• When it was called.
• Which model handled the request.
• Cost by model, principal or tag.
• Request and response payloads.
• MCP usage.
• Policy outcomes.

Access to full request and response payloads should itself be tightly governed because those records may contain customer data, confidential prompts or model outputs.

11. MLflow 3 for GenAI Lifecycle Management

MLflow 3 combines observability, evaluation, feedback and version tracking across development and production.

Its role extends beyond traditional experiment tracking. It provides a quality-management system for entire GenAI applications.

11.1 Prompt Registry​

Prompt Registry provides central prompt versioning and sharing.

A production prompt record should include:

• Prompt template.
• Version.
• Owner.
• Intended use case.
• Supported model.
• Required variables.
• Expected output schema.
• Evaluation results.
• Risk classification.
• Approval status.
• Deployment aliases.

Prompt changes should be treated like code changes because they can alter behaviour, safety, latency and cost.

11.2 Application versioning​

A GenAI application version is broader than a model version.

It may include:

Application version =
    orchestration code
  + prompt version
  + model service configuration
  + tool definitions
  + retrieval index
  + embedding model
  + reranking configuration
  + validation rules
  + service policies

MLflow app and prompt versioning allows the team to compare these configurations and attribute quality changes to a specific release.

11.3 Evaluation as a release gate​

A release pipeline should reject a candidate when:

• Safety falls below the accepted threshold.
• Retrieval groundedness regresses.
• Tool-call accuracy declines.
• Cost rises beyond tolerance.
• Latency violates the service-level objective.
• A critical adversarial case fails.
• Human-review requirements are not met.

Example release policy:

Safety pass rate:                     100% on critical dataset
Groundedness:                         ≥ 0.90
Tool selection accuracy:             ≥ 0.95
P95 latency:                          ≤ 4 seconds
Average cost per successful request:  ≤ approved budget
Critical policy violations:          0

Thresholds should be set using business risk rather than generic benchmark targets.

12. LLMOps on Databricks

LLMOps extends MLOps to account for prompts, retrieval, external models, tools, human feedback and non-deterministic evaluation.

12.1 What remains the same as MLOps​

Databricks recommends retaining familiar engineering practices:

• Separate development, staging and production environments.
• Git-based version control.
• MLflow lifecycle management.
• Unity Catalog model governance.
• Delta-table data storage.
• Existing CI/CD infrastructure.
• Modular pipelines.

12.2 What changes for LLM applications​

The production artifact may not be a model​

The deployable artifact may be:

• A prompt pipeline.
• A RAG chain.
• An agent.
• A multi-agent graph.
• A tool-calling application.
• A wrapper around an external model.

Evaluation becomes multidimensional​

There may be several acceptable answers, meaning accuracy alone is insufficient.

External providers become part of the architecture​

The system may depend on:

• Provider availability.
• Provider latency.
• Model-version changes.
• Token pricing.
• Regional availability.
• Data-processing terms.

Central model services and AI Gateway reduce the amount of provider-specific logic embedded in applications.

Retrieval becomes a first-class dependency​

The application’s quality depends on:

• Source data.
• Parsing.
• Chunking.
• Embeddings.
• Index synchronisation.
• Filtering.
• Retrieval.
• Reranking.

Human feedback becomes operational data​

Feedback should be stored, governed and incorporated into evaluation, monitoring and future development.

12.3 LLMOps environments​

A mature environment model might look like this:

12.4 Model and prompt promotion​

Promotion should rely on aliases rather than hard-coded version numbers.

Example:

The same concept should apply to prompts, models, retrieval configurations and complete application versions.

12.5 Deployment patterns​

Blue-green deployment​

Deploy the new version separately and switch traffic after validation.

Canary deployment​

Send a small percentage of traffic to the new version and compare:

• Quality.
• Errors.
• Latency.
• Cost.
• User feedback.

A/B testing​

Use concurrent variants to measure business outcomes such as:

• Resolution rate.
• Conversion.
• Task completion.
• User satisfaction.
• Escalation rate.

Shadow deployment​

Send production requests to a candidate version without returning its responses to users. Compare the candidate’s traces and scores against the production version.

These patterns should be implemented together with versioned prompts, stable model-service interfaces, trace metadata and evaluation scorers.

13. ML System Architecture Considerations

The supplied Databricks documentation primarily focuses on agent development, retrieval, governance, evaluation and serving integration. Production architects must additionally optimise the system as a complete distributed application.

13.1 Latency optimisation​

Total latency can be approximated as:

Total latency =
    authentication
  + policy checks
  + planning
  + retrieval
  + reranking
  + tool execution
  + model generation
  + output validation
  + network overhead

Optimisation techniques include:

• Streaming responses.
• Reducing unnecessary agent loops.
• Parallelising independent tool calls.
• Caching stable retrieval results.
• Using metadata filters before reranking.
• Selecting smaller models for routing and classification.
• Limiting retrieved context.
• Choosing the appropriate AI Search endpoint.
• Setting tool timeouts.
• Avoiding serial judge execution in the user-facing path.

13.2 Batching​

Batching is appropriate for:

• Document processing.
• Classification.
• Summarisation.
• Embedding creation.
• Offline evaluation.
• Scheduled quality scoring.
• Bulk content enrichment.

AI Functions and Lakeflow pipelines are more appropriate for these workloads than repeatedly invoking an interactive agent endpoint.

13.3 Multi-model routing​

A multi-model system might use:

• A small model for classification.
• A specialised model for extraction.
• A stronger model for complex reasoning.
• An embedding model for retrieval.
• A separate model for evaluation.
• A fallback model for provider failure.

Unity AI Gateway and custom model services provide the governance and routing layer needed to centralise this design.

13.4 Model compression and quantisation​

Quantisation and compression primarily apply when the organisation hosts its own model.

They can reduce:

• Memory consumption.
• GPU requirements.
• Serving cost.
• Inference latency.

The trade-off is potential degradation in:

• Accuracy.
• Tool-calling reliability.
• Structured output.
• Reasoning quality.
• Long-context behaviour.

A compressed model should therefore be evaluated using the application’s complete dataset rather than only generic model benchmarks.

13.5 Autoscaling​

Autoscaling policies should consider:

• Requests per second.
• Concurrent streaming sessions.
• Average generation time.
• Tool-call duration.
• Cold-start tolerance.
• Model size.
• Index size.
• Peak versus average traffic.

Scaling the model endpoint alone may not solve bottlenecks in retrieval, downstream APIs or approval workflows.

14. Model Monitoring and Drift

Traditional feature drift remains relevant for predictive models, but GenAI monitoring introduces additional forms of change.

14.1 Data drift​

The distribution of user requests may change.

Examples:

• New products.
• New terminology.
• Seasonal requests.
• Changes in customer demographics.
• New languages.

14.2 Retrieval drift​

Retrieval quality can change because:

• Documents were added or removed.
• Chunking changed.
• An embedding model changed.
• The index became stale.
• Metadata filters became incorrect.
• Source documents became outdated.

14.3 Behaviour drift​

Behaviour can change because:

• The prompt changed.
• The model provider updated a model.
• Tool descriptions changed.
• A new tool was added.
• Service policies changed.
• Agent-routing logic changed.

14.4 Quality drift​

Monitor trends in:

• Groundedness.
• Relevance.
• Correctness.
• Safety.
• Tool-selection accuracy.
• Escalation quality.
• User frustration.
• Human-feedback scores.

14.5 Statistical monitoring​

For numeric operational metrics, statistical process-control techniques can detect abnormal movement.

Examples include:

• Control charts for latency.
• EWMA for gradual cost increases.
• CUSUM for small but persistent shifts.
• Population Stability Index for request categories.
• Distribution-distance measures for embeddings.
• Multivariate monitoring across cost, latency and quality.

Automated retraining or reconfiguration should not be triggered solely because a drift statistic crossed a threshold. The workflow should confirm:

1. The drift is material.
2. It affects a business or quality outcome.
3. The likely cause is understood.
4. A new model, prompt or retrieval configuration improves the system.
5. Governance and approval requirements are met.

15. Genie One

Genie One is the simplified Databricks interface for business users. It was previously named Databricks One.

Its purpose is to allow users to interact with analytics and AI without navigating technical concepts such as notebooks, compute, models and SQL infrastructure.

15.1 Business-user capabilities​

Users can:

• View AI/BI dashboards.
• Track KPIs.
• Ask natural-language questions.
• Interact with Genie Agents.
• Use Databricks Apps.
• Search shared assets.
• Access personalised recommendations.
• Browse content through business domains.
• Schedule chat tasks.
• Draft documents from chat conversations.

15.2 Workspace and account levels​

Workspace-level Genie One is scoped to a single workspace.

Account-level Genie One provides a cross-workspace discovery experience, while still showing users only assets they are permitted to access. Assets open in their originating workspace.

15.3 Governed access​

Genie One does not create a separate security model. Users interact only with assets shared with them.

This allows platform teams to expose:

• Certified dashboards.
• Approved Genie Agents.
• Curated Databricks Apps.
• Governed documents and data experiences.

Business users gain a simple interface without receiving unnecessary authoring or infrastructure permissions.

15.4 Data residency considerations​

The documentation identifies specific account-level metadata-processing considerations, including possible US processing of Databricks-generated metadata and Geo-dependent processing behaviour.

Organisations with strict residency or compliance requirements should review these settings carefully. Account administrators can disable the account-level feature, and workspaces using the compliance security profile are excluded from account-level aggregation.

16. End-to-End Reference Architecture

Consider an enterprise customer-service agent.

16.1 Data ingestion​

Customer policies, product documents and support articles are ingested into Delta tables.

AI Functions parse, classify and prepare the documents for search.

16.2 Retrieval​

AI Search indexes the prepared chunks.

Metadata includes:

• Product.
• Country.
• Customer type.
• Effective date.
• Document classification.
• Access group.

16.3 Structured data​

A Genie Agent or Databricks SQL tool provides access to:

• Orders.
• Payments.
• Refunds.
• Deliveries.
• Customer subscriptions.

16.4 Agent​

A custom ResponsesAgent:

1. Classifies the request.
2. Determines whether retrieval or structured data is required.
3. Selects approved tools.
4. Collects evidence.
5. Generates a response.
6. Validates the response.
7. Escalates when required.

16.5 Model access​

The agent calls a model service through Unity AI Gateway.

The gateway applies:

• Permissions.
• Rate limits.
• Spend controls.
• PII policies.
• Prompt-injection policies.
• Tool restrictions.

16.6 Observability​

MLflow records:

• User request.
• Retrieved documents.
• SQL or Genie requests.
• Tool calls.
• Model input and output.
• Latency.
• Tokens.
• Policy decisions.
• Final answer.

16.7 Evaluation​

Development evaluation tests:

• Correctness.
• Groundedness.
• Safety.
• Escalation.
• Tool accuracy.
• Citation accuracy.
• Tone.

Production monitoring applies the same scorers to sampled traces.

16.8 Business experience​

The capability can be delivered through:

• A Databricks App for service agents.
• An external customer portal.
• Genie One for internal analytics and operational questions.

The complete architecture is:

17. What Expert-Level Databricks GenAI Capability Looks Like

An expert Databricks AI engineer should be able to:

Software engineering​

Design typed, modular Python applications with clear separation between orchestration, retrieval, tools, validation and serving.

Agent engineering​

Build custom agents using ResponsesAgent, implement streaming, manage state, support tool confirmation and design multi-agent workflows.

Tool engineering​

Select appropriately between managed MCP, external MCP, Unity Catalog functions, local tools, Managed OAuth and connection-proxy integrations.

Retrieval engineering​

Design document ingestion, chunking, embeddings, hybrid retrieval, filtering, reranking and retrieval evaluation.

Governance​

Apply Unity Catalog privileges, Gateway rate limits, spend caps, service policies, tool filtering and audit controls.

Observability​

Instrument application and business logic using automatic and manual MLflow tracing.

Evaluation​

Create representative datasets, built-in judges, custom judges, deterministic scorers and multi-turn conversation tests.

LLMOps​

Version prompts and applications, build CI/CD quality gates, implement staged deployment and continuously evaluate production traces.

Business delivery​

Expose governed analytics and AI experiences through Databricks Apps, APIs and Genie One.

Conclusion

The main value of the Databricks GenAI ecosystem is not any single model, agent framework or vector-search feature. Its value is the integration of data, application engineering, model access, tools, evaluation, governance and business consumption.

A production Databricks AI system should therefore be designed as a governed software product:

• Unity Catalog governs the assets.
• Unity AI Gateway governs the traffic and behaviour.
• AI Search and Genie provide governed knowledge access.
• AI Functions operationalise batch intelligence.
• Databricks Apps and custom agents provide the application layer.
• MLflow provides tracing, evaluation, versioning and monitoring.
• LLMOps provides the delivery and improvement process.
• Genie One provides a simplified governed experience for business users.

The resulting system is not merely an LLM endpoint. It is an observable, testable, governable and continuously improving enterprise AI platform.