Confluent, a company that’s part of IBM, has introduced new capabilities for building Streamhouse, an innovative data architecture designed to keep business information current. Its goal is to let AI agents, applications, and data-analytics tools make decisions and take action based on up-to-date information.
With the general availability of Lightning Tables on AWS, Confluent adds a new capability to its real-time data-streaming platform, which now brings together five core functions: stream, connect, process, govern, and query data. In doing so, the company brings together, on a single platform, the tools needed to build a Streamhouse architecture.
Confluent also plays a key role in IBM’s data strategy, combining real-time stream processing with IBM’s integration, governance, and artificial intelligence (AI) solutions, across both hybrid environments and different technology infrastructures.
Why businesses need a Streamhouse architecture
Data lakes, data warehouses, and lakehouse architectures were designed mainly to generate reports and run other batch processes. As AI agents take on more tasks within businesses, they need to make decisions based on information spread across different systems, each with its own update frequency. When those systems offer different views of the current state of the business, agents can act on outdated or contradictory information — and those actions can ripple into other workflows.
A Streamhouse architecture addresses this problem by continuously capturing, processing, governing, and making available up-to-date business context for agents, applications, and analytics tools.
Take the following example of how an airline handles a canceled flight:
- Today, it relies on different systems that hold conflicting views of the same operation. The flight-operations system tracks where the plane is; the scheduling system tracks when it’s due to take off; and the reservations system tracks which seats are available. When these systems update at different times, an autonomous agent handling new bookings can build an itinerary that looks viable in each one individually but doesn’t work in practice: a seat is available, but the plane won’t land in time for the passenger to make their connection. Customer-service teams and notification systems can also act on that contradictory information, compounding the fallout from the disruption.
- With a Streamhouse, every change can update a shared view of the airline’s operations. When a plane’s location changes, a departure is delayed, or a seat is booked, the source system sends that update to a continuous stream of events. Streamhouse brings all those updates together to provide up-to-date business context and make it available to the agents handling new bookings, customer-service teams, and applications as they work. Instead of pulling information from systems that update separately, everyone can jointly check the plane’s location, departure time, and seat availability before offering the passenger a new itinerary.
Companies are already seeing results. Accenture uses a Streamhouse architecture powered by Confluent to connect data across all of its global operations.
“With a presence in more than 120 countries, our company was running on legacy batch-processing architectures and point-to-point connections, which created duplication, increased operational load, and delayed data availability for analytics work. Adopting a Confluent-based Streamhouse architecture lets us turn large volumes of events into real-time, governed, reusable data products that connect our operational and analytical systems. Thanks to Confluent’s infrastructure automation and enterprise connectivity, our developers work in a standardized environment, without having to build custom integrations or manually manage data movement. This continuous data foundation has cut the time needed to get useful insight from months to days and has shortened reporting times, letting our teams focus on building high-value AI solutions for our 800,000 employees.”
What sets a Streamhouse architecture apart
A Streamhouse rests on three architectural principles: continuous updates, business-critical reliability, and decentralization. It keeps business context constantly up to date, is built to meet the service levels production environments demand, and lets teams access data where it lives, without having to centralize it first. It also coexists with existing data warehouses and lakehouse architectures, adding a real-time operational layer on top to support decision-making and action.
“As an AI-native commerce platform delivering personalized shopping experiences and operational and financial advantages to merchants, Extend needs a high degree of adaptability to handle rapid merchant onboarding, real-time price adjustments, and instant fraud detection. We initially ran into the limits of traditional message queues and static data lakes, which lacked the ability to replay events, guarantee data persistence, and enforce strict schemas at scale. Implementing a 100% serverless Streamhouse architecture built on Confluent Cloud turns Kafka into our system of record for operations. This lets downstream microservices consume and query data in real time asynchronously, without breaking established contracts. This inherently decoupled streaming foundation gives us the flexibility and responsiveness we need to scale our operations, adapt to real-time market signals, and power agentic AI workflows.”
Confluent’s data-streaming platform now offers a complete Streamhouse architecture
Confluent’s data-streaming platform (DSP) provides a unified foundation for building a Streamhouse. At its core is Apache Kafka®, which captures and streams business events. On its own, though, Kafka requires teams to integrate other capabilities to turn those events into up-to-date, useful business information. The DSP brings all of these capabilities together on a single platform, now joined by Query as a fifth pillar alongside Stream, Connect, Process, and Govern. Key capabilities and updates for each of these pillars include:
- Stream, built on Apache Kafka and the Kora engine, provides a durable, scalable foundation for capturing and streaming business events between systems, with sub-millisecond latency in Confluent’s internal tests. Actual latency can vary depending on configuration and workload.
- Connect, with more than 150 fully managed and custom connectors, lets teams bring data from enterprise systems into Kafka and distribute it to other systems. Hybrid connector management expands the options for administering these connections.
- Process uses Apache Flink to combine, enrich, and derive business context close to the source as data changes. It also includes batch-processing capabilities that let teams bootstrap streaming workloads from existing data.
- Govern helps teams discover, classify, and protect data through lineage, catalog, quality, encryption, schema management, and access-control capabilities.
- Query makes up-to-date business context available to analytics tools, agents, and applications.
Query lets teams tap streaming data for analytics, agents, and applications, through whichever interface best fits each use case:
- Confluent Tableflow makes streaming data available to lakehouses and data warehouses in open table formats, ready for analysis.
- Real-Time Context Engine gives AI agents up-to-date business context through the Model Context Protocol (MCP).
- Lightning Tables gives applications access to the current state of the business through low-latency SQL and APIs, letting them respond as conditions change.
Lightning Tables: fast access to the current state of the business
Now generally available on AWS, Lightning Tables in Confluent Cloud is a read-only query layer that persistently stores Apache Kafka data and materializes it into tables, keeping them current as changes occur. This lets AI agents, applications, and analytics tools instantly query the current state of the business. Its key features include:
- Data freshness and query performance: in Confluent’s internal tests, the time from new data being created to its availability for querying is around 100 milliseconds, with query latency of 10 to 100 milliseconds and the capacity to handle more than 1,000 queries per second.
- Lower cost and complexity: delivers data continuously through a fully managed serving layer, without needing to build separate ETL pipelines, databases, or caching systems.
- Multiple query interfaces: offers access through MCP, built into Real-Time Context Engine for AI applications, as well as through SQL and a direct API.
- Unified governance: lets teams apply schemas, lineage, and access policies from the moment data is captured through to when it’s made available.
- Schema evolution: makes it easier to manage changes to data schemas so they don’t break downstream applications that use them.
- Feature Store: lets teams compute features using Flink and store them for machine-learning use cases, including real-time personalization.
“The next era of enterprise AI will be defined by agents that know what’s happening in the business at the exact moment they act,” said Shaun Clowes, General Manager of Confluent and Data at IBM. “That’s what a Streamhouse can deliver. With Lightning Tables, our data-streaming platform gives every agent and application an up-to-date, governed view of the business. This lets companies catch fraud signals earlier, serve customers at the right moment, and run their operations based on what’s actually happening right now, rather than what was happening the day before.”
Frequently asked questions
What is a Streamhouse architecture?
It’s a data architecture that continuously captures, processes, governs, and delivers up-to-date business context to AI agents, applications, and analytics tools, instead of relying on the batch updates typical of data lakes and warehouses.
What is Lightning Tables, and what’s new about it?
It’s a read-only query layer in Confluent Cloud, now generally available on AWS, that materializes Apache Kafka data into continuously updated tables, with around 100-millisecond data freshness and 10-to-100-millisecond query latency in Confluent’s internal tests.
How does Streamhouse relate to data lakes and warehouses?
It doesn’t replace them. Streamhouse coexists with existing data warehouses and lakehouse architectures, adding a real-time operational layer on top so agents and applications can act on current business context.
What role does Confluent play in IBM’s data strategy?
Confluent, part of IBM, combines real-time stream processing with IBM’s integration, governance, and AI solutions across hybrid environments and different technology infrastructures.
This article is also available in Spanish: «Confluent presenta su arquitectura Streamhouse para impulsar la IA con contexto de negocio en tiempo real», on revistacloud.com.
