# Observability

Llumnix provides built-in observability at four levels: request performance, component diagnostics, fine-grained instance state, and engine-native metrics. Each component exposes a `/metrics` HTTP endpoint. Prometheus Operator CRDs (ServiceMonitor / PodMonitor) scrape these endpoints, and pre-built Grafana dashboards in `deploy/observability/` visualize the collected data.

## Monitoring Setup

`deploy/base/monitoring.yaml` defines Prometheus Operator resources to scrape metrics from Llumnix components:

- **ServiceMonitor** (`llumnix-control-plane`): scrapes Gateway (port 8089) and Scheduler (port 8088) via `/metrics` every 10s.
- **PodMonitor** (`llumnix-engine-neutral` / `llumnix-engine-prefill` / `llumnix-engine-decode`): scrapes engine pods managed by LeaderWorkerSet (neutral, prefill, decode) via `/metrics` every 10s, with relabeling to extract `infer_type` and `model` labels.

`monitoring.yaml` is included in `deploy/base/kustomization.yaml`. All deployment configurations under `deploy/` reference this base and inherit monitoring automatically. Note that the engine PodMonitors match only LeaderWorkerSet-based pods; deployment examples using plain Deployments (e.g., `traffic-mirror/`, `traffic-splitting/`) are not covered by the engine PodMonitors — only Gateway and Scheduler metrics are collected for those examples.

## Grafana Dashboards

Pre-built Grafana dashboard JSON files are located in `deploy/observability/`.

### Llumnix Request Dashboard

`llumnix-request-dashboard.json` — end-user request-level metrics.

| Panel | Key Metrics | Type | Description |
|---|---|---|---|
| Request Rate | `request_total` | Counter | Total inference requests processed, partitioned by status code |
| Request Rate by Status | `request_total` | Counter | Request rate broken down by HTTP status code |
| Request Retry & Fallback Rate | `request_retry_total`, `request_fallback_total`, `request_fallback_retry_success_total` | Counter | Retry count, fallback count, and successful retries after fallback rate-limit (429) |
| Input / Output Token Throughput | `request_input_tokens_total`, `request_output_tokens_total` | Counter | Cumulative input (prompt) and output (completion) token counts |
| Input / Output Token Distribution | `request_input_tokens`, `request_output_tokens` | Histogram | Per-request token count distribution |
| E2E Latency | `request_e2e_latency_seconds` | Histogram | End-to-end request latency in seconds |
| TTFT | `request_ttft_milliseconds` | Histogram | Time to first token in milliseconds |
| TPOT | `request_tpot_milliseconds` | Histogram | Time per output token: (E2E − TTFT) / (output_tokens − 1) in milliseconds |
| ITL | `request_itl_milliseconds` | Histogram | Inter-token latency between consecutive output tokens in milliseconds |
| Prefix Cache Hit Ratio | `request_prefix_cache_hit_percent` | Histogram | Prefix cache hit ratio on the selected instance per request (0–100%) |
| Max Prefix Cache Hit Ratio | `request_max_prefix_cache_hit_percent` | Histogram | Maximum prefix cache hit ratio across all instances per request (0–100%) |

### Llumnix Component Dashboard

`llumnix-component-dashboard.json` — internal component-level metrics for Gateway, Scheduler, and system runtime.

| Panel | Key Metrics | Type | Description |
|---|---|---|---|
| Queue Duration | `request_queue_duration_milliseconds` | Histogram | Request queue waiting duration in milliseconds |
| Preprocess Duration | `request_preprocess_duration_milliseconds` | Histogram | Request preprocessing duration in milliseconds |
| Schedule Duration | `request_schedule_duration_milliseconds` | Histogram | Request scheduling phase duration in milliseconds |
| Postprocess Duration | `request_postprocess_duration_milliseconds` | Histogram | Response postprocessing duration in milliseconds |
| Gateway Requests | `gateway_pending_requests`, `gateway_current_requests` | Gauge | Pending and total in-flight requests in the Gateway |
| Scheduling Events | `scheduler_scheduling_total`, `scheduler_scheduling_failed_total` | Counter | Total scheduling attempts and failures |
| Rescheduling Events | `scheduler_rescheduling_total`, `scheduler_rescheduling_failed_total` | Counter | Total rescheduling operations and failures |
| CMS Refresh Metadata Duration | `scheduler_cms_refresh_metadata_duration_milliseconds` | Histogram | CMS instance metadata refresh duration |
| CMS Refresh Status Duration | `scheduler_cms_refresh_status_duration_milliseconds` | Histogram | CMS instance status refresh duration |
| Full-Mode Schedule Duration | `request_full_mode_schedule_duration_milliseconds` | Histogram | Full-mode scheduling decision duration per request |
| Query Prefix Cache Hit Duration | `request_query_prefix_cache_hit_duration_milliseconds` | Histogram | Duration of querying KVS for prefix cache hit |
| Calc Prefix Cache Hit Duration | `request_calc_prefix_cache_hit_duration_milliseconds` | Histogram | Duration of calculating prefix cache hit length |
| Uptime / Goroutines / Go Memory | `uptime_seconds`, `go_goroutines`, `go_memstats_alloc_bytes` | Gauge | System runtime diagnostics |

### Llumnix CMS Dashboard

`llumnix-cms-dashboard.json` — per-instance CMS status, split into Prefill and Decode sections. Each section includes:

| Panel | Key Metrics | Type | Description |
|---|---|---|---|
| CMS Requests | `instance_cms_running_requests`, `instance_cms_waiting_requests`, `instance_cms_loading_requests`, `instance_cms_scheduler_waiting_to_decode_requests`, `instance_cms_scheduler_running_to_decode_requests`, `instance_cms_hybrid_scheduler_waiting_to_decode_requests` | Gauge | Per-instance request counts by state |
| CMS Used Tokens | `instance_cms_used_gpu_tokens` | Gauge | GPU tokens currently used per instance |
| CMS Prefill Tokens | `instance_cms_uncomputed_tokens_all_waiting_prefills`, `instance_cms_uncomputed_tokens_scheduler_running_prefills`, `instance_cms_unallocated_tokens_scheduler_running_prefills` | Gauge | Uncomputed/unallocated tokens for prefill requests |
| CMS Decode Tokens | `instance_cms_unallocated_tokens_hybrid_scheduler_waiting_decodes`, `instance_cms_hybrid_scheduler_waiting_to_decode_tokens`, `instance_cms_scheduler_waiting_to_decode_tokens`, `instance_cms_scheduler_running_to_decode_tokens`, `instance_cms_tokens_loading_requests` | Gauge | Tokens for decode and loading requests |
| CMS Inflight Dispatch Requests | `instance_cms_inflight_dispatch_requests`, `instance_cms_inflight_dispatch_prefill_requests`, `instance_cms_inflight_dispatch_decode_requests` | Gauge | Inflight dispatch request counts |
| CMS Inflight Dispatch Tokens | `instance_cms_uncomputed_tokens_inflight_dispatch_prefill_requests`, `instance_cms_tokens_inflight_dispatch_decode_requests` | Gauge | Tokens for inflight dispatch requests |
| CMS KV Cache Usage Ratio | `instance_cms_kv_cache_usage_ratio_projected` | Gauge | Projected KV cache usage ratio per instance |
| CMS Decode Batch Size | `instance_cms_decode_batch_size` | Gauge | Decode batch size per instance |
| CMS All Prefill/Decode Tokens | `instance_cms_all_prefills_tokens_num`, `instance_cms_all_decodes_tokens_num` | Gauge | Total tokens for all prefill/decode requests per instance |

The dashboard also includes a **Selected Instance Scheduling Metrics** section showing the scheduling decision context for the chosen instance:

| Panel | Key Metrics | Type | Description |
|---|---|---|---|
| Selected Instance KV Cache Usage Ratio | `selected_instance_kv_cache_usage_ratio_projected` | Histogram | Projected KV cache usage ratio on the selected instance |
| Selected Instance Decode Batch Size | `selected_instance_decode_batch_size` | Histogram | Decode batch size on the selected instance |
| Selected Instance Prefill/Decode Tokens | `selected_instance_all_prefills_tokens_num`, `selected_instance_all_decodes_tokens_num` | Histogram | Total prefill/decode tokens on the selected instance |
| Selected Instance Predicted TTFT | `selected_instance_predicted_ttft` | Histogram | Predicted TTFT for the selected instance in milliseconds |
| Selected Instance Predicted TPOT | `selected_instance_predicted_tpot` | Histogram | Predicted TPOT for the selected instance in milliseconds |

### Llumnix LRS Dashboard

`llumnix-lrs-dashboard.json` — per-instance Local Real-time State (LRS), split into Prefill and Decode sections.

| Panel | Key Metrics | Type | Description |
|---|---|---|---|
| LRS Requests | `instance_lrs_running_requests`, `instance_lrs_waiting_requests`, `instance_lrs_total_requests` | Gauge | Running, waiting, and total requests on a backend endpoint |
| LRS Tokens | `instance_lrs_running_tokens`, `instance_lrs_waiting_tokens`, `instance_lrs_total_tokens` | Gauge | Running, waiting, and total token counts on a backend endpoint |

### vLLM Dashboard

`vllm-dashboard.json` — engine-native vLLM metrics.

| Panel | Key Metrics | Type | Description |
|---|---|---|---|
| E2E Request Latency | `vllm:e2e_request_latency_seconds` | Histogram | End-to-end request latency from vLLM |
| Time To First Token Latency | `vllm:time_to_first_token_seconds` | Histogram | TTFT from vLLM |
| Inter-Token Latency | `vllm:inter_token_latency_seconds` | Histogram | ITL from vLLM |
| Scheduler State | `vllm:num_requests_running`, `vllm:num_requests_waiting` | Gauge | vLLM internal scheduler state (running/waiting) |
| Cache Utilization | `vllm:kv_cache_usage_perc` | Gauge | KV cache utilization ratio |
| Token Throughput | `vllm:prompt_tokens_total`, `vllm:generation_tokens_total` | Counter | Token generation throughput |
| Finish Reason | `vllm:request_success_total` | Counter | Request completion reason distribution |
| Queue Time | `vllm:request_queue_time_seconds` | Histogram | Time spent in vLLM queue |
| Prefill and Decode Time | `vllm:request_prefill_time_seconds`, `vllm:request_decode_time_seconds` | Histogram | Per-request prefill and decode time |
| Request Prompt/Generation Length | `vllm:request_prompt_tokens`, `vllm:request_generation_tokens` | Histogram | Prompt and generation length distributions (heatmap) |