Stream Processing Engines (SPEs) are a new class of system software, designed to process "firehoses" of real time data with very low latency. They are applicable today in feed processing and electronic trading applications on Wall Street, network monitoring, fraud detection and military environments. As the sea change caused by cheap micro-sensor technology takes hold, we expect to see many more applications with high-volume low-latency requirements. Of course, other system software technologies, such as main memory data bases and rule engines, are being repurposed by marketing departments to address streaming applications. In this talk we describe 10 characteristics that a good SPE must possess, so that consumers can tell real SPEs from "wannabees". We will use examples from actual StreamBase applications to help make the discussion concrete. We then close with a collection of open research issues in stream processing, some of which are being worked on in the Borealis project, joint research between MIT, Brown and Brandeis.
Dr. Stonebraker has been a pioneer of data base research and technology for more than a quarter of a century. He was the main architect of the INGRES relational DBMS, and the object-relational DBMS, POSTGRES. These prototypes were developed at the University of California at Berkeley where Stonebraker was a Professor of Computer Science for twenty five years. More recently at M.I.T. he was a co-architect of the Aurora/Borealis stream processing engine. He is the founder of three venture-capital backed startups, which commercialized these prototypes. Presently he serves as Chief Technology Officer of StreamBase Systems, Inc., which is commercializing Aurora/Borealis. Professor Stonebraker is the author of scores of research papers on data base technology, operating systems and the architecture of system software services. He was awarded the ACM System Software Award in 1992, for his work on INGRES. Additionally, he was awarded the first annual Innovation award by the ACM SIGMOD special interest group in 1994, and was elected to the National Academy of Engineering in 1997. He was awarded the IEEE John Von Neumann award in 2005, and is presently an Adjunct Professor of Computer Science at M.I.T., where he is working on a variety of future-generation data-oriented projects.
