High-Availability Algorithms for Distributed Stream Processing

Jeong-Hyon Hwang, Magdalena Balazinska, Alexander Rasin, Ugur Cetintemel, Mike Stonebraker, Stan Zdonik
The 21st International Conference on Data Engineering (ICDE 2005), Tokyo, Japan, April 2005

Stream-processing systems are designed to support an emerging class of applications that require sophisticated and timely processing of high-volume data streams, often originating in distributed environments. Unlike traditional data-processing applications that require precise recovery for correctness, many stream-processing applications can tolerate and benefit from weaker recovery guarantees. In this paper, we study various recovery guarantees and pertinent recovery techniques that can meet the correctness and performance requirements of stream-processing applications. We discuss the design and algorithmic challenges associated with the proposed recovery techniques and describe how each can provide different guarantees with proper combinations of redundant processing, checkpointing, and remote logging. Using analysis and simulations, we quantify the cost of our recovery guarantees and examine the performance and applicability of the recovery techniques. We also analyze how the knowledge of query network properties can help decrease the cost of high availability.

[PDF (206KB)]

Bibtex Entry:

@inproceedings{hwang2005high-availability,
   author =       "Jeong-Hyon Hwang and Magdalena Balazinska and Alexander Rasin and Ugur Cetintemel and Mike Stonebraker and Stan Zdonik",
   title =        "{High-Availability Algorithms for Distributed Stream Processing}",
   booktitle =    {The 21st International Conference on Data Engineering (ICDE 2005)},
   year =         {2005},
   month =        {April},
   address =      {Tokyo, Japan}
}