Connecting your service to MONSTER
MONSTER is push-model: your service reports in, MONSTER never polls you (except the optional active pinger). There are four integration layers — adopt them in order. Base URL: https://monster-sre.taxstreem.com (use your staging deployment's URL in staging).
1 · Core concepts
Service & service_key. Everything hangs off a registered service. Register one in Settings (or POST /api/services) and MONSTER generates a service_key. Every payload you push carries that key. Pushing with an unregistered key returns 404 — register first.
The status ladder is computed from heartbeat age, then demoted by payload health:
| Field | Type | Meaning |
|---|---|---|
| PENDING | — | Registered but no heartbeat ever received |
| UP | — | Last heartbeat < 32 minutes ago |
| DEGRADED | — | Heartbeat 32–38 min ago (late), OR on-time but metrics breached a soft threshold (error_rate > 5%, memory > 1800 MB), OR service self-reported "degraded" |
| DOWN | — | No heartbeat in the last 38 minutes → missed_heartbeat incident opens |
Incidents. Missed heartbeats, breached payload thresholds, degraded routes, and queue failures all open incidents. Incidents opened by different services within a 3-minute window are correlated into one incident group, and each gets an AI-generated ops summary (Gemini) for the on-call engineer.
2 · Quick start (5 minutes)
Step 1 — register the service (dashboard Settings, or API):
curl -X POST https://monster-sre.taxstreem.com/api/services \
-H "Content-Type: application/json" \
-d '{"name": "my-service", "description": "What it does"}'
# → { "service": { "serviceKey": "my-service", ... } }Step 2 — send your first heartbeat:
curl -X POST https://monster-sre.taxstreem.com/api/heartbeat \
-H "Content-Type: application/json" \
-d '{
"service_key": "my-service",
"status": "healthy",
"version": "1.4.2",
"environment": "staging",
"response_time_ms": 42
}'Step 3 — verify: the service flips from PENDING to UP on the dashboard.
Step 4 — schedule it every ≤ 30 minutes. A cron, a background goroutine, a Cloud Scheduler job — anything. At 32 minutes without a beat you are DEGRADED; at 38, DOWN, and an incident opens.
Step 5 — add the structured metrics block (section 3) so MONSTER can detect problems in a service that is alive but unhealthy.
3 · Heartbeat — liveness + windowed health
POST /api/heartbeat every ≤ 30 minutes. Heartbeat answers “is the process alive and broadly healthy?” — never put per-route telemetry here (that is section 4).
| Field | Type | Meaning |
|---|---|---|
| service_key | string · required | Key from MONSTER Settings |
| status | string | "healthy" | "degraded" — self-reported |
| response_time_ms | number | Overall latency sample for the latency graph |
| version | string | Semver tag |
| environment | string | "staging" | "production" |
| metrics | object | Structured telemetry block — see below |
| meta | object | { version, commit_sha, region } — shown in incident AI summaries |
The metrics block — send what you have; every field is optional:
| Field | Type | Meaning |
|---|---|---|
| error_rate | number 0–1 | Fraction of requests ≥ 400 in the window. > 0.05 → DEGRADED · > 0.10 → incident |
| requests_per_window | number | Requests since the last heartbeat |
| avg_latency_ms | number | Mean request latency |
| latency_p99_ms | number | 99th percentile latency |
| memory_mb | number | Total memory (OS). > 1800 → incident |
| memory_pct | number 0–100 | Memory utilisation. > 90 → incident |
| heap_mb / goroutines / gc_cycles / uptime_seconds | number | Runtime snapshot (Go-flavoured, optional) |
| db_pool_saturation | number 0–1 | DB pool usage. > 0.90 → incident |
| active_requests | number | In-flight requests at beat time |
| queue_failures_per_window | number | Retryable queue-handler failures (nacked) in the window |
| poison_messages_per_window | number | Non-retryable messages acked+logged. > 0 → incident |
| queue_breaker_open | boolean | A queue consumer is paused by its circuit breaker. true → incident |
Go (this is how numen does it — windowed atomics, swap-to-zero on send):
// Count every request into windowed atomic counters, then every 30 min:
payload := map[string]any{
"service_key": "numen-staging",
"status": status, // "healthy" | "degraded"
"version": version,
"environment": env,
"response_time_ms": avgLatencyMs,
"metrics": map[string]any{
"error_rate": float64(errs) / float64(total),
"requests_per_window": total, // swapped to 0 after send
"avg_latency_ms": avgLatencyMs,
"goroutines": runtime.NumGoroutine(),
"memory_mb": float64(mem.Sys) / 1024 / 1024,
"queue_failures_per_window": queueFailures.Swap(0),
"poison_messages_per_window": poisonMessages.Swap(0),
"queue_breaker_open": breakersOpen.Load() > 0,
},
}
body, _ := json.Marshal(payload)
http.Post(monsterURL+"/api/heartbeat", "application/json", bytes.NewReader(body))
// Full reference implementation: taxstreem-numen/internal/monitoring/reporter.goPython (FastAPI/async services — run as a background task):
import asyncio, httpx
async def monster_heartbeat_loop(service_key: str, monster_url: str):
while True:
total, errors, lat_sum = WINDOW.snapshot_and_reset() # your counters
payload = {
"service_key": service_key,
"status": "healthy" if (total == 0 or errors / total <= 0.05) else "degraded",
"environment": "staging",
"metrics": {
"error_rate": (errors / total) if total else 0,
"requests_per_window": total,
"avg_latency_ms": (lat_sum / total) if total else 0,
},
}
try:
async with httpx.AsyncClient(timeout=10) as client:
await client.post(f"{monster_url}/api/heartbeat", json=payload)
except Exception:
pass # never let monitoring take down the service
await asyncio.sleep(30 * 60)TypeScript / Node:
setInterval(async () => {
const { total, errors, latencySumMs } = window.snapshotAndReset();
await fetch(`${MONSTER_URL}/api/heartbeat`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
service_key: 'taxstreem-backend-staging',
status: total && errors / total > 0.05 ? 'degraded' : 'healthy',
metrics: {
error_rate: total ? errors / total : 0,
requests_per_window: total,
avg_latency_ms: total ? latencySumMs / total : 0,
},
}),
}).catch(() => {}); // monitoring must never crash the service
}, 30 * 60 * 1000);4 · Route telemetry — per-endpoint health
POST /api/otel/routes answers “which route is hurting?”. Push one record per route per reporting window. MONSTER stores each push, evaluates thresholds (error_rate > 10% or any status ≥ 500 → endpoint_error incident + alert fan-out) and renders per-route sparklines on the service page.
curl -X POST https://monster-sre.taxstreem.com/api/otel/routes \
-H "Content-Type: application/json" \
-d '{
"service_key": "taxstreem-backend-staging",
"routes": [
{
"path": "/api/v1/businesses/:id/transactions/import",
"method": "POST",
"error_rate": 0.02,
"last_status_code": 200,
"avg_latency_ms": 340,
"requests_per_window": 51
}
]
}'OTEL semantic-convention aliases are accepted, so you can forward straight from an OTel processor: http.route → path, http.method → method, http.status_code → last_status_code.
If you already run an OpenTelemetry SDK, you can alternatively point its OTLP/HTTP metrics exporter at POST /api/otlp/v1/metrics — MONSTER accepts ExportMetricsServiceRequest and uses resource.attributes["service.name"] as the service_key.
/api/heartbeat. Heartbeat is liveness; routes are health. They have separate thresholds, separate incidents, and separate graphs.5 · Endpoint monitoring — active probing
POST /api/monitor is the one pull-model tool: MONSTER (or the dashboard, or your CI) actively pings a list of URLs and returns live status + response time. Stateless — nothing is stored; the dashboard keeps rolling history in memory. Use it for public URLs, third-party dependencies, and smoke checks after a deploy.
curl -X POST https://monster-sre.taxstreem.com/api/monitor \
-H "Content-Type: application/json" \
-d '{
"services": [
{ "host": "https://staging.api.taxstreem.com/health", "name": "backend", "serviceType": "backend" },
{ "host": "https://staging.app.taxstreem.com", "name": "webapp", "serviceType": "webapp" }
]
}'
# → { "results": [{ "name": "backend", "status": "UP", "responseTime": 118, ... }], "checkedAt": "..." }You can also push externally-observed status into the generic webhook (POST /api/webhook with { serverId, status, responseTime, path }) if a third-party pinger does the probing and MONSTER should just display it.
6 · Mission-critical points — instrumenting the choke points
Heartbeats tell you a service is up. Route telemetry tells you an endpoint is erroring. Neither catches the third class of failure: a background workflow silently stalling while every HTTP check stays green. Find the points in your pipeline where a failure stops money or data from flowing, and instrument those directly.
records: [] (every chunk had failed upstream). Numen built its Redis MGET keys from that empty array → Redis protocol error → nack → Pub/Sub redelivered the same immutable message every ~10 s, forever. Transactions stopped returning to users, yet both services were UP, all HTTP endpoints were green, and no alert fired. The failure lived between services, in a queue handler.The pattern that now guards that point (and should guard yours) has three parts:
6.1 — Classify failures at the choke point. Split errors into non-retryable (bad payload — retrying can never fix it) and retryable (dependency down — retrying will eventually work). Numen's implementation: messaging.ErrNonRetryable → the message is acked, its payload logged, and MONSTER notified; retryable errors nack and count toward a circuit breaker that pauses the consumer (messages stay queued) instead of burning redelivery attempts.
6.2 — Push an immediate critical event the moment the choke point fails. Don't wait for the next heartbeat window — POST to the generic webhook:
POST /api/webhook
Content-Type: application/json
{
"domain": "numen-staging:queue:numen-results-cg.tag.batch",
"level": "error",
"data": {
"reason": "poison message on numen-results-cg.tag.batch (event_id=fd85..., event_type=tag.batch.result): tag batch result for batch \"019f...\" has zero records (status=500, failures=3)",
"service_key": "numen-staging",
"environment": "staging",
"timestamp": "2026-07-09T09:52:22Z"
}
}Convention for domain: <service_key>:<component>:<resource> — e.g. numen-staging:queue:numen-results-cg.tag.batch. Level error makes it stand out in the webhook feed.
6.3 — Carry a windowed gauge in the heartbeat so the failure also drives the incident machinery (grouping + AI summary + dashboard), not just a log line:
"metrics": {
"queue_failures_per_window": 2, // retryable failures since last beat
"poison_messages_per_window": 1, // > 0 → MONSTER opens an incident
"queue_breaker_open": false // true → MONSTER opens an incident
}Applying this to your own choke points. Ask: “if this exact line fails silently for an hour, does a customer notice before we do?” Typical TaxStreem examples: queue result handlers (this case), filing dispatch to NRS/Rev360, webhook deliveries from AI services, ledger posting, bank-sync ingestion. For each:
| Field | Type | Meaning |
|---|---|---|
| 1. Classify | code | Non-retryable (ack + report) vs retryable (retry/breaker). Never let a poison payload loop. |
| 2. Event | push | POST /api/webhook with domain "<service>:<component>:<resource>", level "error" — immediate visibility |
| 3. Gauge | beat | Add a per-window counter/boolean to your heartbeat metrics — durable incident trail |
| 4. Threshold | server | If it warrants an incident, add the rule to heartbeat detectPayloadIncident() (see queue_breaker_open as the template) |
Reference implementations to copy from: taxstreem-numen/internal/infrastructure/messaging/subscriber.go (classification + breaker) and taxstreem-numen/internal/monitoring/reporter.go (ReportCritical + windowed counters).
7 · Alerts & channels
Incidents fan out to the channels configured in Settings — /api/integrations/slack, teams, gchat, clickup — and email via POST /api/alert ({ serverName, endpoint, timestamp } → “CRITICAL: X is DOWN” to the configured recipients).
NEXT_PUBLIC_ENVIRONMENT is set in MONSTER's environment, /api/alert returns { skipped: true } and sends nothing. This is the #1 “why didn't I get the email?” question — incidents still appear on the dashboard; only email is muted.8 · Endpoint reference
| Field | Type | Meaning |
|---|---|---|
| POST /api/heartbeat | push · 30 min | Liveness + windowed metrics. Requires registered service_key |
| GET /api/heartbeat?serviceKey= | read | Recent heartbeats for a service |
| POST /api/otel/routes | push · per window | Per-route telemetry; OTEL aliases accepted; opens endpoint_error incidents |
| POST /api/otlp/v1/metrics | push · OTLP | Standard OTel ExportMetricsServiceRequest; service.name = service_key |
| POST /api/webhook | push · on event | Generic event log ({domain,data,level}) or status push ({serverId,status,...}). Used for mission-critical events |
| POST /api/alert | push · on incident | Email alert. Delivered when NEXT_PUBLIC_ENVIRONMENT is production/unset; suppressed for staging/dev/other |
| POST /api/monitor | pull · on demand | Actively ping a list of URLs; stateless |
| GET/POST/DELETE /api/services | admin | List (with computed status/trends), register, remove services |
| GET /api/reports | read | Historical reporting data |
| GET/PUT /api/settings | admin | Alert recipients, integrations, thresholds |
| POST /api/analyze | AI | Ad-hoc AI analysis of current state |
| GET/POST /api/threat-hunt | AI | Security-focused sweep |
| GET/POST /api/topology | admin | Service dependency graph shown on the dashboard |
| POST /api/mcp/mcp | MCP · agent | MCP server (Streamable HTTP). 8 read-only SRE tools. Auth: Bearer MCP_AUTH_TOKEN |
Incident-opening thresholds (heartbeat payload):
| Field | Type | Meaning |
|---|---|---|
| error_rate > 0.10 | incident | degraded_payload |
| error_rate > 0.05 | status | DEGRADED (no incident yet) |
| memory_mb > 1800 | incident | degraded_payload |
| memory_pct > 90 | incident | degraded_payload |
| db_pool_saturation > 0.90 | incident | degraded_payload |
| queue_breaker_open = true | incident | degraded_payload — queue consumer paused |
| poison_messages_per_window > 0 | incident | degraded_payload — non-retryable payloads rejected |
| no heartbeat for 38 min | incident | missed_heartbeat |
| route error_rate > 0.10 or status ≥ 500 | incident | endpoint_error (via /api/otel/routes) |
9 · Pitfalls
- Beating every 30 min exactly is too slow in practice. Network jitter puts you over the 32-minute DEGRADED line. Beat every 25–28 minutes.
- 404 on first heartbeat — the service isn't registered yet. Register in Settings (or
POST /api/services) first; the key is derived from the name. - No email on staging is intentional — see section 7.
- Windowed metrics must reset after each beat (swap-to-zero), or your error_rate becomes a lifetime average and never crosses a threshold.
- Monitoring must never take the service down — every push should be fire-and-forget with a timeout and a swallowed error (see the snippets above).
- Green ≠ healthy. If your workflow crosses a queue, instrument the consumer (section 6) — the ai-core → numen incident produced zero red endpoints while transactions were stuck for hours.