Databricks rethinks: Where data and AI finally come together

In today's business world, one thing is certain: companies that effectively use AI and generative AI will become winners in their industries. What was once considered a peripheral issue has now become a strategic imperative. But many organizations face a fundamental problem: their data and AI systems exist in separate worlds.

Imagine: On the one hand, your valuable company data - on the other, the powerful AI models that could transform this data into competitive advantages. In between? A chasm of technical hurdles, governance problems and inefficient processes.

In this article, we'll show you how Databricks bridges this gap and why this approach is so valuable for forward-thinking companies. We highlight:

• The current challenges of fragmented data and AI landscapes
• The revolutionary Lakehouse concept as a bridge builder
• How Unity Catalog enables unified governance
• The role of GenAI agents in production
• Practical use cases for SAP users

The pain points of the separated data world

The separation of data and AI systems causes significant problems in companies:

1. Loss of control over your data

If data has to be moved to external model providers, a dangerous loss of control occurs. Questions about data sovereignty, compliance and the protection of sensitive information often remain unanswered. There is also a risk of an ecosystem lock-in that limits strategic flexibility.

2. The silos in your data landscape

The typical corporate landscape resembles a patchwork quilt:

• Data warehouses for structured data
• Data lakes for unstructured information
• Separate ETL systems for data transformations
• Isolated BI systems for analytics

The result? Multiple data copies, inconsistent governance and opaque data silos that make informed decisions difficult.

3. The complexity of GenAI agents

The development of GenAI agents poses challenges for even experienced teams:

• Lack of standards for robust evaluations
• Ambiguity in selecting appropriate techniques
• Difficult balance between cost and quality

4. The burden of legacy database technologies

Traditional databases come with their own problems:

• High vendor lock-in
• Costly licensing models
• Focus on on-premise instead of cloud-native solutions
• Inflexible scaling

5. The gap between data engineers and business analysts

While data engineers work with professional tools, business analysts often use Excel or similar applications. The result: discrepancies in data, production problems and unclear governance.

The Databricks solution: A data intelligence platform

Databricks has developed a comprehensive solution based on a fundamentally new approach: the models come to the data, not the other way around. This philosophy manifests itself in a well-thought-out platform with several key components:

The Lakehouse concept: The best of both worlds

The Databricks Lakehouse combines the strengths of data lakes and data warehouses:

• Flexibility and scalability of a data lake
• Structure and performance of a data warehouse
• Open formats such as Delta and Iceberg as a basis
• Accessible to any data engine, without proprietary restrictions

This architecture makes it possible to efficiently manage both structured and unstructured data and make it usable for AI applications.

Unity Catalog: Governance across all data assets

The Unity Catalog goes far beyond traditional data catalogs:

• Unified governance layer across all data formats
• Manage all data assets: tables, unstructured files, models, tools, notebooks and dashboards
• Comprehensive security controls and access management
• Detailed auditing and lineage tracking
• Cost controls for efficient data management
• Federation functions for integrating external data sources

With Unity Catalog, companies finally get a “single point of truth” for their entire data landscape.

AI-driven interactions for every user

Databricks democratizes access to data through intuitive, AI-powered interfaces:

• Genie Spaces (Text-to-SQL): Enables natural language data queries - already used by over 81% of Databricks users
• Databricks Assistant: A continuously learning assistant that supports critical workflows and suggests code fixes, for example

These tools dramatically lower the barrier to entry for data analysis and enable even non-experts to gain valuable insights.

Agent Bricks: Easily bring GenAI agents into production

The new product Agent Bricks (currently in beta) revolutionizes the development and use of GenAI agents:

• Governance-centric platform for trusted agents
• Robust evaluation mechanisms with LLM judges
• Intelligent selection of suitable techniques (fine tuning, vector DBs etc.)
• Optimization of costs and quality

Particularly noteworthy is the multi-agent supervisor, which can coordinate different agents. For example, the power of Genie (for structured data) can be combined with a Knowledge Assistant (for unstructured data) to answer complex questions across all data sources.

Lakebase: The new generation of databases

Lakebase applies a traditional database engine to a data lake and provides:

• Low-cost, open storage as a basis
• Pay-per-use pricing model for maximum flexibility
• Direct consumption through AI models
• Fresh and synchronized real-time data
• Consistent governance and access controls
• Separation of compute and storage for optimal scaling
• Ultra-low latencies for demanding applications

Built on a PostgreSQL base with support for common extensions, Lakebase databases can be started in less than a second - a quantum leap over traditional systems.

The bridge to SAP data: Practical integration

The Databricks approach offers particular advantages for companies with SAP landscapes. Integrating SAP data into the Lakehouse enables:

• Unified view of SAP and non-SAP data
• Real-time analyzes across system boundaries
• AI-powered forecasts based on operational SAP data
• Automated workflows between SAP and other systems

Use case: Integrated supply chain management

A particularly valuable use case is integrated supply chain management. Here companies can:

• Combine SAP ERP data with external market data
• Detect delivery bottlenecks early through AI-supported forecasts
• Automatically identify alternative suppliers
• Realize cost and time savings through optimized processes

Use case: Financial reporting and planning

In finance, the Databricks platform enables:

• Consolidation of financial data from SAP and other sources
• AI-powered anomaly detection for fraud detection
• Automated creation of financial reports
• What-if analyzes for strategic planning

LakeFlow and AIBI: The final pieces of the puzzle

LakeFlow: Seamless data integration and management

LakeFlow helps companies integrate and manage their data efficiently:

• LakeFlow Connect: Ingest structured and unstructured data from various systems
• LakeFlow Declarative Pipelines: Simplified transformation, cleansing and quality assurance
• LakeFlow Jobs: Orchestration of workloads for always up-to-date data
• LakeFlow Designer: Bridges the gap between data engineers and business analysts through natural language and collaborative development

AIBI: AI-powered business intelligence for everyone

AIBI (AI-powered Business Intelligence) extends data intelligence to all BI teams:

• Scalable, managed text-to-visualization capabilities
• Derived graphs for complex relationships
• Genie Databases for natural language data queries