Blogs

Real-Time Data Processing with Eventstream in Microsoft Fabric​

Inkey Solutions, July 8, 20253634 Views

In today’s data-driven world, the ability to process data in real time is no longer a luxury—it’s a necessity. Whether you’re tracking website traffic, monitoring IoT sensors, or responding to financial market fluctuations, real-time analytics can give your organization a critical edge.

With Eventstream in Microsoft Fabric, real-time data ingestion, transformation, and analytics have become simpler and more scalable. In this blog, we’ll explore what Eventstream is, its key capabilities, how to connect sources (including custom endpoints), apply transformations, and route data to meaningful destinations.

Overview

Eventstream is a real-time data ingestion and processing component within Microsoft Fabric. It allows you to stream data from various sources into destinations like Lakehouse, Eventhouse, Power BI, and more—without needing to write code.

It allows you to:

• Ingest data from multiple streaming platforms like Azure Event Hubs, IoT Hub, and custom APIs.
• Apply real-time transformations using a graphical interface.
• Route data to various Fabric destinations like Lakehouse, Eventhouse, or Power BI.
• Trigger real-time workflows and alerts.

Key Features

• No-Code/Low-Code Experience

You don’t need to write complex code or scripts. The UI-based interface lets you drag, drop, and configure your data flow visually, making it accessible even for users with minimal development experience.

• Multiple Source Integrations

You can ingest data from:

Azure Event Hubs

Azure IoT Hub

Custom Webhooks

Real-Time Analytics (KQL databases)

A sample dataset is also available for exploration purposes.

 

• Real-Time Transformation

Apply filtering, enrichment, or transformation logic using Power Query-style interfaces to clean and shape data on the fly.

• Multiple Destinations

Push real-time data to:

Lakehouse for long-term storage

KQL Databases for real-time query processing etc.

• Scalability & Reliability

Eventstream supports high-throughput, low-latency data processing with built-in failover and reliability—ideal for mission-critical workloads.

Setting up an Eventstream flow:

First create the EventStream in Fabric.

This will be the Interface.

We have these options for Source.

• From Connect data sources we can connect to many Microsoft Sources, Database CDC, Azure Events, Fabric Events as shown below and many more.

• From Sample Data We can connect to these sources.

• A Custom Endpoint allows you to bring real-time data into Fabric from external platforms that act as streaming producers. Unlike traditional file imports or REST-based APIs, this feature is designed for event-driven streaming, where messages are continuously published to Fabric in near real-time.

It supports 3 protocols for custom source integration:

• Kafka: A high-throughput distributed messaging system
  • Common tools to connect: Apache Kafka, Confluent, or Redpanda.
• AMQP: Advanced Message Queuing Protocol
  • Common tools to connect: RabbitMQ, Azure Service Bus.
• Event Hub: Azure’s big data streaming platform
  • Common tools to connect: Azure Event Hubs.

 You can connect to any external system that:

• Publishes data in Kafka, AMQP, or Event Hub
• It has proper network access to reach the Fabric Eventstream endpoint.
• Sends data as structured events (e.g., JSON messages).

You cannot connect to:

• REST APIs: Because Eventstream does not accept HTTP POST/GET.
• Webhooks: Because they are designed to send data when something happens, like a one-time notification. But Eventstream is built for continuous, real-time streaming, not one-off messages or callbacks.
• File uploads (CSV, JSON etc.): These are batch-based, not real-time.

How Custom EndPoint will work:

• Your external system (e.g., Kafka, RabbitMQ) acts as a publisher/producer.
• Fabric Eventstream is configured with the custom endpoint to act as a
• As events are emitted by your system, Fabric ingests them in real-time.
• You can then route those events to destinations like Lakehouse, Eventhouse, or Power BI.

After you select the Custom Endpoint, only the name of the configured source will be displayed.

After adding the source, you need to publish the flow to view the remaining configurations.

Once the flow is published, the configuration options will become visible:

Here, the Entra ID Authentication tab is configuring Azure Active Directory (Entra ID) authentication.

If your producer (like Kafka or Event Hub) is secured with Entra ID, you’ll use this to   authorize access to the custom endpoint by:

• Registering your app in Entra ID (Azure AD)
• Providing the necessary client ID, tenant ID, and client secret

This ensures that only trusted apps or services can send data to Fabric.

And SAS Key Authentication tab is used for Shared Access Signature (SAS) key-based authentication, typically for Azure Event Hubs.

It allows you to connect by pasting a SAS connection string, which grants permission to send data securely to the Eventstream via Event Hub.

 

Now, we have added Sample Data as a Source.

From here, we can change the format and time settings of the data we want to retrieve.

Then we can perform these transformation steps:

• Aggregate: Summarizes data by applying functions like count, sum, average, etc., over a time window.
• Expand: Flattens nested records or arrays into separate fields.
• Filter: Filters incoming events based on specified conditions.
• Group by: Groups events based on one or more fields and performs calculations on each group.
• Join: Combines data from multiple streams based on a matching key or condition.
• Manage fields: Lets you add, remove, or rename fields in your stream schema.
• Union: Merges multiple streams into a single stream.

And these are the destinations you can use:

• Cutom EndPoint: We can use a Custom Endpoint as the destination, just like we did for the source.
• LakeHouse: Lakehouse destination stores your streaming data in Delta tables in Microsoft Fabric OneLake. It offers both structured storage and analytical capability by combining the best of data lakes and data warehouses. With Lakehouse, you can later analyze this data using SQL, Spark, or Power BI.
• Eventhouse: Eventhouse is designed for storing high volumes of event-driven data, especially telemetry or log-type data, optimized for low-latency streaming and fast querying. It’s ideal when your use case involves real-time dashboards, anomaly detection, or time-series analytics.
• Activator: The Activator destination triggers real-time actions when specific conditions are met in your data stream. It doesn’t just store data—it reacts. You can define logic to send alerts, call downstream systems, or kick off workflows like Power Automate or Azure Functions.