Grafana

Topic

What is grafana? what makes it different from a tool like sentry and how can I use it as effectively as possible?

Conjecture

Ok, so as I understand it, Grafana is an open source tool and UI for parsing data. In general, this data is a live feed coming from an application or server. The most common uses for it are to allow users to query log data coming from an application or visualize metrics that are being generated from an application. Essentially, though, Grafana is a data store that allows data from the feeds to be parsed, visualized, and queried.

Grafana is the overall piece of software, and then inside of it, there are sub-modules: Prometheus for data visualization and recording metrics, and then Loki for parsing log data.

Research

So basically, my conjecture was correct. The purpose of grafana is to “Collect, correlate, and visualize data.”

It allows you to query, visualize, alert on, and explore metrics, logs, and traces from data sources that you configure.

So I was kind of wrong about the submodules. Technically, Prometheus and Loki are data source plugins. Prometheus is a time series database and loki is a logging tool. So they are data sources that grafana can use. However, grafana isn’t limited to this it also has plugins for NoSQL/SQL, CI/CD tooling and more.

At its core Grafana is an open source project, but the company that develops it offers a cloud hosted version of the app called Grafana Cloud.

The main connector between this cloud-deployed version and your deployment/data source is called Grafana Alloy.

The most important data plugin for Grafana is arguably prometheus.

Prometheus is a core technology for monitoring and observability of systems. It is essentially a data model and a query language, and it was first implemented at SoundCloud, so technically it is independent of Grafana.

Observability of a system is mostly about understanding the state of a system at a specific instant. Thus prometheus is essentially a technology for storing time series data. The core data model for prometheus is composed of:

1. Metrics - Essentially a timestamp and a metric or sample

1. Label sets - There are called dimensions for example job or device.

Thus, you can associate your metrics with specific labels in a set. You can technically store time series data in any DB, but Prometheus is optimized specifically for time series data. It also has its own query language, PromQL for efficiently querying.

Final Summary

Grafana is the query and visualization layer, not the data store — that is the piece I had muddled going in. You bring the data; Grafana connects to it, queries it, draws it, and alerts on it. Prometheus and Loki aren't modules inside Grafana, they are two of the sources it plugs into, and both run perfectly well without it.

That also settles the Sentry question I opened with, which I never actually got to in the research. They aren't really competitors. Sentry is opinionated about one job: catch an exception, tie it to a stack trace and a release, group it into an issue somebody can own. It answers what broke, and where in the code. Grafana is deliberately unopinionated and answers what is the state of the system, over time. Sentry tells you a request threw; Grafana tells you p99 latency has been climbing for three hours and nothing threw at all. You want both, and they barely overlap.

On using it well — the thing I want to carry into our setup is that the label sets are the part to be careful with. A metric labelled with anything high-cardinality (a user ID, a request ID) quietly becomes a separate time series per distinct value, which is the standard way people melt a Prometheus instance. Keep labels to bounded dimensions: job, route, status code, environment. And point alerts at symptoms like latency and error rate rather than at causes, so the alert keeps making sense after you rearrange whatever is underneath it.