Prometheus, Grafana, GitOps and SRE practices

Why and what

Prometheus scrapes labelled time-series metrics. Grafana visualizes data from configured sources. Logs and traces complement metrics. GitOps controllers reconcile declared configuration from a repository into an environment; Git becomes the reviewed source of desired deployment state, not a place to commit plaintext secrets.

Metrics lab

1. In an isolated lab, install a reviewed Prometheus/Grafana stack following its supported method.
2. Expose a sample application's metrics endpoint only to intended collectors.
3. Configure a scrape target and verify it is up before building dashboards.
4. Graph request rate, error rate and latency using correctly typed metrics.

rate(http_requests_total[5m])
sum(rate(http_requests_total{status=~"5.."}[5m]))

These queries require an application exporting that metric and label schema. A chart returning no data can mean the metric name is wrong, not that the error count is zero. Avoid unbounded labels such as user IDs and raw URL paths; high cardinality increases resource use.

Alerting and SLOs

Define a user-facing service-level indicator, such as successful requests over valid requests. Set an SLO over a time window and use its error budget to guide release risk. Alert on actionable symptoms and sustained burn, not every noisy fluctuation. Include a runbook and owner in alerts.

GitOps lab

Create a repository containing the academy Deployment/Service manifests. Connect a reviewed Argo CD or Flux installation with minimum necessary repository/cluster access. Sync a tagged image change, inspect the diff and observe rollout. Make a harmless manual replica change in the lab to see drift/reconciliation behaviour. Avoid two controllers fighting over the same field, for example replicas managed by HPA and forced by another tool.

Incident exercise

Simulate an unhealthy application version, detect it through a health signal, revert the desired version and record the timeline. Distinguish rollback of code from restoration of data. Write a short blameless review covering detection delay, impact, cause and a concrete prevention action.

Official references

Prometheus getting started Grafana documentation Argo CD getting started

Interview and revision check

Why can two reconcilers cause unstable replica counts?

If both force different desired values for the same field, they can repeatedly undo each other. Establish clear field ownership.