Ai Monitoring Setup

Generate complete LLM monitoring infrastructure with cost tracking, performance metrics, error logging, and alerting for production AI applications.

What You'll Get

When you run this command, you'll receive:

1. Cost tracking with real-time budget monitoring
2. Performance metrics (latency, throughput, success rate)
3. Token usage analytics per model, user, endpoint
4. Error logging and alerting (Sentry, Slack, PagerDuty)
5. Prometheus metrics for Grafana dashboards
6. Custom dashboards with pre-built visualizations
7. Alerting rules for cost spikes, errors, latency
8. OpenTelemetry integration for distributed tracing

Usage

/ai-monitoring-setup <monitoring_stack>

Monitoring Stacks: prometheus, datadog, newrelic, comprehensive

Examples:

• /ams prometheus - Prometheus + Grafana stack
• /ams datadog - DataDog integration
• /ams comprehensive - Full observability stack (Prometheus + Sentry + Logging)

Generated Output

Example: Comprehensive Monitoring Stack

Input:

/ams comprehensive

Output:

1. Project Structure

llm-monitoring/
├── monitoring/
│   ├── __init__.py
│   ├── metrics.py              # Prometheus metrics
│   ├── cost_tracker.py         # Cost tracking and budgets
│   ├── logger.py               # Structured logging
│   ├── tracer.py               # OpenTelemetry tracing
│   └── alerting.py             # Alert management
├── dashboards/
│   ├── grafana/
│   │   ├── llm_overview.json
│   │   ├── cost_analysis.json
│   │   └── performance.json
│   └── prometheus/
│       └── alerts.yml
├── docker/
│   ├── prometheus.yml
│   ├── grafana-datasources.yml
…

2. Cost Tracker (monitoring/cost_tracker.py)

from dataclasses import dataclass, field
from datetime import datetime, timedelta
from typing import Dict, List, Optional
import json

@dataclass
class CostTracker:
    """Track LLM costs with budget alerts."""

    monthly_budget: float = 1000.0  # USD
    alert_thresholds: List[float] = field(default_factory=lambda: [0.5, 0.75, 0.9])

    # Pricing per 1M tokens (as of 2024)
    PRICING = {
        "gpt-4-turbo": {"input": 10.00, "output": 30.00},
        "gpt-3.5-turbo": {"input": 0.50, "output": 1.50},
        "claude-3-opus": {"input": 15.00, "output": 75.00},
        "claude-3-sonnet": {"input": 3.00, "output": 15.00},
…

3. Prometheus Metrics (monitoring/metrics.py)

from prometheus_client import Counter, Histogram, Gauge, Info
import time
from functools import wraps

# Metrics
llm_requests_total = Counter(
    'llm_requests_total',
    'Total LLM API requests',
    ['model', 'endpoint', 'status']
)

llm_request_duration = Histogram(
    'llm_request_duration_seconds',
    'LLM request duration in seconds',
    ['model', 'endpoint'],
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0, 30.0, 60.0]
)
…

4. Structured Logging (monitoring/logger.py)

import logging
import json
from datetime import datetime
from typing import Any, Dict

class StructuredLogger:
    """JSON structured logging for LLM operations."""

    def __init__(self, name: str = "llm-app"):
        self.logger = logging.getLogger(name)
        self.logger.setLevel(logging.INFO)

        # JSON formatter
        handler = logging.StreamHandler()
        handler.setFormatter(self.JSONFormatter())
        self.logger.addHandler(handler)

    class JSONFormatter(logging.Formatter):
…

5. Alerting (monitoring/alerting.py)

import requests
from typing import Dict, Optional
from enum import Enum

class AlertLevel(Enum):
    INFO = "info"
    WARNING = "warning"
    CRITICAL = "critical"

class AlertManager:
    """Send alerts to various channels."""

    def __init__(
        self,
        slack_webhook: Optional[str] = None,
        pagerduty_key: Optional[str] = None,
        email_config: Optional[Dict] = None
    ):
…

6. Grafana Dashboards (dashboards/grafana/llm_overview.json)

{
  "dashboard": {
    "title": "LLM Overview",
    "panels": [
      {
        "title": "Total Requests",
        "targets": [{
          "expr": "sum(rate(llm_requests_total[5m]))"
        }],
        "type": "graph"
      },
      {
        "title": "Cost (Last 24h)",
        "targets": [{
          "expr": "sum(increase(llm_cost_total_usd[24h]))"
        }],
        "type": "singlestat"
      },
…

7. Docker Compose (docker/docker-compose.yml)

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./alerts.yml:/etc/prometheus/alerts.yml
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'

  grafana:
    image: grafana/grafana:latest
    volumes:
…

Features Included

Cost Management:

• Real-time cost tracking per request
• Budget alerts (50%, 75%, 90% thresholds)
• Cost breakdown by model, user, endpoint
• Monthly cost reset automation

Performance Monitoring:

• Request latency (P50, P95, P99)
• Throughput (requests per second)
• Success rate tracking
• Active request count

Resource Tracking:

• Token usage (input/output)
• Model usage distribution
• Per-user analytics

Alerting:

• Slack notifications
• PagerDuty integration (critical)
• Custom alert rules