Observability & SRE · 2024 — 2025
Self-Healing Infrastructure Monitoring
Prometheus and Grafana across infrastructure and applications, wired to webhook-triggered Jenkins jobs that resolve routine incidents automatically.
- Auto
- Remediation for routine alerts
- 24/7
- Production on-call coverage
Automated remediation
Once Prometheus and Grafana covered both infrastructure and application metrics, we closed the loop on the alerts whose fix is always the same.
The main example: server disk usage crosses 70%, and instead of paging someone the alert fires a webhook that triggers a Jenkins pipeline to prune the oldest backups and clean the Jenkins workspace directories. The incident resolves itself and leaves a record of what ran.
We automated the alerts whose runbook contained no decision, and left the rest to a human.
On-call
I hold the production on-call rotation for the platform’s servers.