Stream service

The stream service is created on Pega Platform™ to enable asynchronous data flow between Processes. The stream service is a multi-node component based on Apache Kafka. Apache Kafka is a distributed data store optimized for ingesting and processing streaming data in real time. You must enable at least one stream node for the stream service to function.

The stream service ingests, routes, and delivers high volumes of low-latency data, for example, web clicks, transactions, sensor data, work objects, Data Pages, and customer interaction histories. As a resiliency mechanism for operations, the same data is replicated to other nodes in the cluster. Distribution and replication of the stream data records ensure the scalability and fault tolerance of the Stream Service. The service operates as a cluster on one or more servers. You can distribute the servers among multiple data centers to minimize planned or unplanned downtime.

You can use the stream service, for example, to pass correspondence data to delivery channels in Pega Customer Decision Hub™, process customer responses for the Adaptive Decision Manager (ADM), or initiate background processes in any Pega application. Queue processors and Data Flows use stream service to complete their tasks. 

Monitor the status of your stream nodes, partitions, disk space, CPU usage, and database availability regularly to keep your stream service operating without errors.
 

When the stream service operates in external mode, node details are not visible on the stream landing page. Instead, a NORMAL status message appears, indicating that your Pega Platform installation can connect with external Kafka and use it for the stream service. In Pega Cloud® environments, Pega configures and maintains your streaming service. No action is required by you.

You can perform a simple, manual status check of the stream service on the stream landing page (in the header of Dev Studio, click Configure > Decisioning > Infrastructure > Services > Stream). The stream service relies on the availability of the Pega Platform database. A slow or unavailable database might have a serious impact on the stream service.

To understand the health of the Stream service, monitor the queries to the following three databases:

• pr_data_stream_sessions : represents the currently active Kafka sessions
• pr_data_stream_nodes : contains all meta-information about the Kafka cluster
• pr_data_stream_node_updates : keeps the list of recent updates to the pr_data_stream_nodes table. The table does not play a significant role in stream service operations.

If queries exceed 1 second, think about tuning your database. In the event of planned or unplanned database unavailability, consider restarting your stream nodes, particularly if they appear unhealthy. By default, the stream service maintains two replicas of each record. If you increase the number of stream nodes from two to three or four, adjust the data replication setting to match the number of stream nodes.

The stream service replicates each record across a configurable number of servers. This replication enables automatic failover to these replicas when a server in the cluster fails, ensuring messages remain available despite failures.

Check your knowledge with the following interaction: