Realtime Systems Monitoring Using Time Series Databases and Visualization Tools

Overview

With the pervasiveness of cloud services, hybrid solutions and an explosion of data from ever increasing sources, it is easy for most systems managers to feel overwhelmed with the rising rush of data.

One of the proven ways of making sense and digesting complex data flows is through timely and pertinent visualization. It allows both specialists and laymen to instantly make sense of data, either in realtime or over a specific timespan, and to help in identifying issues and pinpointing problems in a fairly consistent manner, once properly set up.

Two challenges stem from dealing with an increasing amount of realtime or near-realtime data; one pertaining to storage and processing, the other to presentation and visualization.

Storing and Manipulating Time-dependent Data

Traditional relational database management systems are generally designed with particular emphasis on data integrity, atomic operations and either read or write optimization. They are also geared to work particularly well with large amounts of data with unique keys and related data. Working with time sensitive information is possible with RDBMS; however, due to their very architecture, working with single keys spread across time negatively impacts the way data is both stored requirements and accessed as data grows.

To remedy this issue, a new class of database systems has emerged in the past few years with particular emphasis on storing and manipulating time series data. The very architecture of these databases was geared towards efficient storage and retrieval of time based numeric data in particular and are akin to NoSQL type databases than relational ones. Popular time-series databases include InfluxDB, Graphite and Prometheus.

The most prominent features of time-series databases include time-stamp data storage and compression, data lifecycle management, data summarization and aggregation, ability to handle large time series dependent scans of many records and time series aware queries. For a more detailed information you can refer to following links here and here.

Data Acquisition and Presentation

Data acquisition and presentation represent respectively data collection and data consumption. We will focus mainly on the presentation aspect of this pipeline, which includes in the middle the time-series DB component, but it’s essential to mention the data acquisition aspect, which generally consists of an agent or system, that is able to generate time-stamped data from distinct systems to be monitored and send them to the time-series DB for storage. In time-series parlance, these are referred to as metrics which are discreet data sets that reflect the state of a particular component in time.

Visualization tools such as Graphite Web, Chronograf and Grafana provide the necessary front-end web interface and graphing capabilities and data connectivity tools to process, present, and most importantly, act on collected metrics based on specific criteria. The choice of data visualization is crucial when presenting time based metrics, and most platforms offer a vast array of options to make the most out of incoming data from presenting state-in-time single stats and gauges to complex tabular metrics, passing through the usual bar or line graphs. It is both an art and a science to select the right presentation of metrics in order to gleam and spot any outliers or odd spikes that diverge from baseline collected metrics.

Data visualizations platforms have now gone further than just presenting collected data and waded into data correlation wether with other metrics and more importantly with logs and events through complementary tools such as Loki and scripting solutions like Flux.

Where to go from here

This was meant as a quick overview of the current state of realtime data collection, processing and presentation. These emerging tools are becoming extremely necessary and useful in keeping track and alerting technical teams on the state of their internal and cloud systems. The provided links provide an excellent primer on the various components discussed in this article. We can expect many more features and integrations to be included as these tools evolve, which most assuredly have a bright future ahead.