Hemanta Ghosh
Hemanta Ghosh
Engineering Manager, Best Buy Inc., USA
Adaptive Control for Streaming Systems: AI-Driven Dynamic Aggregation
"Real-time streaming systems commonly rely on fixed window sizes for aggregation, but this approach performs poorly when data patterns change over time. Fixed windows create a trade-off between detection latency and output stability, where large windows delay insights while small windows produce noisy results. In this paper, we propose an AI-driven control plane (ADCP) that dynamically selects aggregation window sizes based on real-time stream behavior. The system extracts lightweight features such as event rate, growth, and variability, and uses a combination of a gradient boosting model and a Proximal Policy Optimization (PPO) agent to select the most appropriate window size. Instead of modifying window definitions at runtime, the system evaluates multiple window sizes in parallel and selects the best output with minimal overhead. The approach is implemented using Apache Beam on Google Cloud Dataflow and evaluated on four pre-processed real-world-derived bench- marks spanning periodic traffic, bursty e-commerce, adversarial network intrusion, and low-rate IoT sensing, all resampled and normalized to a synthetic per-second format. Experiments demonstrate up to 50% reduction in detection latency, along with 8–12% improvement in Top-K accuracy and 20–25% reduction in output variance, compared to fixed and rule-based approaches. Adversarial evaluations under noisy telemetry, delayed arrivals, and concept drift confirm that these gains are robust across diverse streaming conditions. These results are obtained on pre- processed benchmarks; validation on live event streams remains an open direction."