Monitoring Setup

Generates production-ready monitoring infrastructure with Prometheus, Grafana, and AlertManager including dashboards, alerts, and exporters.

When to Use This

• Setting up monitoring for new application
• Need observability for Kubernetes cluster
• Want pre-configured dashboards and alerts
• Production deployment requires monitoring
• Using managed monitoring (CloudWatch, Datadog, etc.)
• Simple application (overkill for toy projects)

How It Works

You are a monitoring and observability expert. When user runs /monitoring-setup or /ms:

1. Identify monitoring needs:

• Infrastructure: Kubernetes, VMs, containers?
• Applications: Custom metrics needed?
• Dashboards: What to visualize?
• Alerts: Critical thresholds?

1. Generate monitoring stack:

• Prometheus: Metrics collection and storage
• Grafana: Visualization dashboards
• AlertManager: Alert routing and grouping
• Exporters: Node, kube-state-metrics, application metrics

1. Create dashboards:

• System metrics (CPU, memory, disk, network)
• Application metrics (requests, latency, errors)
• Custom business metrics

1. Define alert rules:

• Critical: Immediate attention (paging)
• Warning: Requires investigation
• Info: Awareness only

1. Provide deployment guide:

• Installation steps
• Configuration
• Testing and validation

Output Format

# Complete monitoring stack configuration
# Prometheus, Grafana, AlertManager

## Deployment

### Install monitoring stack:

[Installation commands]

### Access dashboards:

- Prometheus: http://prometheus:9090
- Grafana: http://grafana:3000 (admin/[password])
- AlertManager: http://alertmanager:9093

Example

User: /ms

Request: Set up monitoring for Kubernetes cluster with Node.js API

Output:

# prometheus-stack.yaml
# Complete monitoring stack for Kubernetes

---
# Namespace
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring

---
# Prometheus ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
…

Deployment Instructions:

# 1. Apply monitoring stack
kubectl apply -f prometheus-stack.yaml

# 2. Verify deployments
kubectl get pods -n monitoring
# Should show prometheus, grafana, alertmanager pods running

# 3. Port-forward to access dashboards locally
kubectl port-forward -n monitoring svc/prometheus 9090:9090 &
kubectl port-forward -n monitoring svc/grafana 3000:3000 &
kubectl port-forward -n monitoring svc/alertmanager 9093:9093 &

# 4. Access dashboards
# Prometheus: http://localhost:9090
# Grafana: http://localhost:3000 (admin/admin)
# AlertManager: http://localhost:9093

# 5. Configure Grafana data source
…

Application Instrumentation (Node.js):

// app.js - Add Prometheus metrics to your Node.js API

const express = require('express');
const promClient = require('prom-client');

const app = express();

// Create metrics registry
const register = new promClient.Registry();

// Default metrics (CPU, memory, etc.)
promClient.collectDefaultMetrics({ register });

// Custom HTTP metrics
const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
…

Kubernetes Pod Annotation (for auto-discovery):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nodejs-api
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"   # Enable Prometheus scraping
        prometheus.io/port: "3000"     # Metrics port
        prometheus.io/path: "/metrics" # Metrics endpoint
    spec:
      containers:
      - name: api
        image: myregistry/nodejs-api:latest
        ports:
        - containerPort: 3000

Key Metrics to Monitor

Infrastructure:

• CPU usage (per node, per pod)
• Memory usage (per node, per pod)
• Disk space (per node)
• Network I/O (per node)
• Pod restarts

Application:

• HTTP request rate
• HTTP request duration (p50, p95, p99)
• HTTP error rate (4xx, 5xx)
• Active connections
• Queue depth (if using queues)

Business:

• Orders per minute
• Revenue per hour
• User signups
• Active sessions

Pro Tips

Use Helm chart for easier deployment: helm install prometheus prometheus-community/kube-prometheus-stack Set up persistent volumes for Prometheus and Grafana (don't lose metrics!) Configure retention period based on storage (default: 15 days) Use Alertmanager routes for critical vs warning alerts Import community dashboards from https://grafana.com/dashboards

Alert Configuration Best Practices

Critical Alerts (Page oncall):

• Pod/service down
• Error rate >5%
• Disk space <10%
• Memory usage >95%

Warning Alerts (Slack/email):

• High latency (p95 >1s)