Spotify has developed Heroic, an open-source time series database, to address the challenges of monitoring large-scale systems. Heroic’s inception was driven by Spotify’s need to handle a vast amount of operational data, generated by its rapidly growing infrastructure. The existing solutions were unable to scale to meet this demand, leading to the creation of Heroic.
The design of Heroic is based on two key principles: scalability and flexibility. The database can handle a huge volume of data without compromising on performance. It is also flexible, allowing for different types of queries and data aggregation methods to be used.
Heroic has a distributed architecture, which means it can be spread across multiple machines. This design allows it to handle large amounts of data and provide high availability. It uses Elasticsearch as its metadata backend and Cassandra or Bigtable as its data backend.
Heroic also features an API that enables users to interact with it programmatically. This API is based on Google’s Protocol Buffers, which allows for efficient serialization of structured data.
Spotify has made Heroic open-source to allow others in the community to benefit from it. The company hopes that others will contribute to the project, enhancing its capabilities and helping to solve more complex monitoring problems.
Go to source article: https://labs.spotify.com/2015/11/17/monitoring-at-spotify-introducing-heroic/