MonsterDocs

← Back to dashboard

Connecting your service to MONSTER

MONSTER is push-model: your service reports in, MONSTER never polls you (except the optional active pinger). There are four integration layers — adopt them in order. Base URL: https://monster-sre.taxstreem.com (use your staging deployment's URL in staging).

1 · Core concepts

Service & service_key. Everything hangs off a registered service. Register one in Settings (or POST /api/services) and MONSTER generates a service_key. Every payload you push carries that key. Pushing with an unregistered key returns 404 — register first.

The status ladder is computed from heartbeat age, then demoted by payload health:

FieldTypeMeaning
PENDINGRegistered but no heartbeat ever received
UPLast heartbeat < 32 minutes ago
DEGRADEDHeartbeat 32–38 min ago (late), OR on-time but metrics breached a soft threshold (error_rate > 5%, memory > 1800 MB), OR service self-reported "degraded"
DOWNNo heartbeat in the last 38 minutes → missed_heartbeat incident opens

Incidents. Missed heartbeats, breached payload thresholds, degraded routes, and queue failures all open incidents. Incidents opened by different services within a 3-minute window are correlated into one incident group, and each gets an AI-generated ops summary (Gemini) for the on-call engineer.

2 · Quick start (5 minutes)

Step 1 — register the service (dashboard Settings, or API):

curl -X POST https://monster-sre.taxstreem.com/api/services \
  -H "Content-Type: application/json" \
  -d '{"name": "my-service", "description": "What it does"}'
# → { "service": { "serviceKey": "my-service", ... } }

Step 2 — send your first heartbeat:

curl -X POST https://monster-sre.taxstreem.com/api/heartbeat \
  -H "Content-Type: application/json" \
  -d '{
    "service_key": "my-service",
    "status": "healthy",
    "version": "1.4.2",
    "environment": "staging",
    "response_time_ms": 42
  }'

Step 3 — verify: the service flips from PENDING to UP on the dashboard.

Step 4 — schedule it every ≤ 30 minutes. A cron, a background goroutine, a Cloud Scheduler job — anything. At 32 minutes without a beat you are DEGRADED; at 38, DOWN, and an incident opens.

Step 5 — add the structured metrics block (section 3) so MONSTER can detect problems in a service that is alive but unhealthy.

3 · Heartbeat — liveness + windowed health

POST /api/heartbeat every ≤ 30 minutes. Heartbeat answers “is the process alive and broadly healthy?” — never put per-route telemetry here (that is section 4).

FieldTypeMeaning
service_keystring · requiredKey from MONSTER Settings
statusstring"healthy" | "degraded" — self-reported
response_time_msnumberOverall latency sample for the latency graph
versionstringSemver tag
environmentstring"staging" | "production"
metricsobjectStructured telemetry block — see below
metaobject{ version, commit_sha, region } — shown in incident AI summaries

The metrics block — send what you have; every field is optional:

FieldTypeMeaning
error_ratenumber 0–1Fraction of requests ≥ 400 in the window. > 0.05 → DEGRADED · > 0.10 → incident
requests_per_windownumberRequests since the last heartbeat
avg_latency_msnumberMean request latency
latency_p99_msnumber99th percentile latency
memory_mbnumberTotal memory (OS). > 1800 → incident
memory_pctnumber 0–100Memory utilisation. > 90 → incident
heap_mb / goroutines / gc_cycles / uptime_secondsnumberRuntime snapshot (Go-flavoured, optional)
db_pool_saturationnumber 0–1DB pool usage. > 0.90 → incident
active_requestsnumberIn-flight requests at beat time
queue_failures_per_windownumberRetryable queue-handler failures (nacked) in the window
poison_messages_per_windownumberNon-retryable messages acked+logged. > 0 → incident
queue_breaker_openbooleanA queue consumer is paused by its circuit breaker. true → incident

Go (this is how numen does it — windowed atomics, swap-to-zero on send):

// Count every request into windowed atomic counters, then every 30 min:
payload := map[string]any{
    "service_key":      "numen-staging",
    "status":           status, // "healthy" | "degraded"
    "version":          version,
    "environment":      env,
    "response_time_ms": avgLatencyMs,
    "metrics": map[string]any{
        "error_rate":                  float64(errs) / float64(total),
        "requests_per_window":         total,          // swapped to 0 after send
        "avg_latency_ms":              avgLatencyMs,
        "goroutines":                  runtime.NumGoroutine(),
        "memory_mb":                   float64(mem.Sys) / 1024 / 1024,
        "queue_failures_per_window":   queueFailures.Swap(0),
        "poison_messages_per_window":  poisonMessages.Swap(0),
        "queue_breaker_open":          breakersOpen.Load() > 0,
    },
}
body, _ := json.Marshal(payload)
http.Post(monsterURL+"/api/heartbeat", "application/json", bytes.NewReader(body))
// Full reference implementation: taxstreem-numen/internal/monitoring/reporter.go

Python (FastAPI/async services — run as a background task):

import asyncio, httpx

async def monster_heartbeat_loop(service_key: str, monster_url: str):
    while True:
        total, errors, lat_sum = WINDOW.snapshot_and_reset()  # your counters
        payload = {
            "service_key": service_key,
            "status": "healthy" if (total == 0 or errors / total <= 0.05) else "degraded",
            "environment": "staging",
            "metrics": {
                "error_rate": (errors / total) if total else 0,
                "requests_per_window": total,
                "avg_latency_ms": (lat_sum / total) if total else 0,
            },
        }
        try:
            async with httpx.AsyncClient(timeout=10) as client:
                await client.post(f"{monster_url}/api/heartbeat", json=payload)
        except Exception:
            pass  # never let monitoring take down the service
        await asyncio.sleep(30 * 60)

TypeScript / Node:

setInterval(async () => {
  const { total, errors, latencySumMs } = window.snapshotAndReset();
  await fetch(`${MONSTER_URL}/api/heartbeat`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      service_key: 'taxstreem-backend-staging',
      status: total && errors / total > 0.05 ? 'degraded' : 'healthy',
      metrics: {
        error_rate: total ? errors / total : 0,
        requests_per_window: total,
        avg_latency_ms: total ? latencySumMs / total : 0,
      },
    }),
  }).catch(() => {}); // monitoring must never crash the service
}, 30 * 60 * 1000);

4 · Route telemetry — per-endpoint health

POST /api/otel/routes answers “which route is hurting?”. Push one record per route per reporting window. MONSTER stores each push, evaluates thresholds (error_rate > 10% or any status ≥ 500endpoint_error incident + alert fan-out) and renders per-route sparklines on the service page.

curl -X POST https://monster-sre.taxstreem.com/api/otel/routes \
  -H "Content-Type: application/json" \
  -d '{
    "service_key": "taxstreem-backend-staging",
    "routes": [
      {
        "path": "/api/v1/businesses/:id/transactions/import",
        "method": "POST",
        "error_rate": 0.02,
        "last_status_code": 200,
        "avg_latency_ms": 340,
        "requests_per_window": 51
      }
    ]
  }'

OTEL semantic-convention aliases are accepted, so you can forward straight from an OTel processor: http.route → path, http.method → method, http.status_code → last_status_code.

If you already run an OpenTelemetry SDK, you can alternatively point its OTLP/HTTP metrics exporter at POST /api/otlp/v1/metrics — MONSTER accepts ExportMetricsServiceRequest and uses resource.attributes["service.name"] as the service_key.

Never mix route telemetry into /api/heartbeat. Heartbeat is liveness; routes are health. They have separate thresholds, separate incidents, and separate graphs.

5 · Endpoint monitoring — active probing

POST /api/monitor is the one pull-model tool: MONSTER (or the dashboard, or your CI) actively pings a list of URLs and returns live status + response time. Stateless — nothing is stored; the dashboard keeps rolling history in memory. Use it for public URLs, third-party dependencies, and smoke checks after a deploy.

curl -X POST https://monster-sre.taxstreem.com/api/monitor \
  -H "Content-Type: application/json" \
  -d '{
    "services": [
      { "host": "https://staging.api.taxstreem.com/health", "name": "backend", "serviceType": "backend" },
      { "host": "https://staging.app.taxstreem.com",         "name": "webapp",  "serviceType": "webapp"  }
    ]
  }'
# → { "results": [{ "name": "backend", "status": "UP", "responseTime": 118, ... }], "checkedAt": "..." }

You can also push externally-observed status into the generic webhook (POST /api/webhook with { serverId, status, responseTime, path }) if a third-party pinger does the probing and MONSTER should just display it.

6 · Mission-critical points — instrumenting the choke points

Heartbeats tell you a service is up. Route telemetry tells you an endpoint is erroring. Neither catches the third class of failure: a background workflow silently stalling while every HTTP check stays green. Find the points in your pipeline where a failure stops money or data from flowing, and instrument those directly.

Case study — the 2026-07-09 ai-core → numen Redis incident. AI Core published a batch tag result with records: [] (every chunk had failed upstream). Numen built its Redis MGET keys from that empty array → Redis protocol error → nack → Pub/Sub redelivered the same immutable message every ~10 s, forever. Transactions stopped returning to users, yet both services were UP, all HTTP endpoints were green, and no alert fired. The failure lived between services, in a queue handler.

The pattern that now guards that point (and should guard yours) has three parts:

6.1 — Classify failures at the choke point. Split errors into non-retryable (bad payload — retrying can never fix it) and retryable (dependency down — retrying will eventually work). Numen's implementation: messaging.ErrNonRetryable → the message is acked, its payload logged, and MONSTER notified; retryable errors nack and count toward a circuit breaker that pauses the consumer (messages stay queued) instead of burning redelivery attempts.

6.2 — Push an immediate critical event the moment the choke point fails. Don't wait for the next heartbeat window — POST to the generic webhook:

POST /api/webhook
Content-Type: application/json

{
  "domain": "numen-staging:queue:numen-results-cg.tag.batch",
  "level": "error",
  "data": {
    "reason": "poison message on numen-results-cg.tag.batch (event_id=fd85..., event_type=tag.batch.result): tag batch result for batch \"019f...\" has zero records (status=500, failures=3)",
    "service_key": "numen-staging",
    "environment": "staging",
    "timestamp": "2026-07-09T09:52:22Z"
  }
}

Convention for domain: <service_key>:<component>:<resource> — e.g. numen-staging:queue:numen-results-cg.tag.batch. Level error makes it stand out in the webhook feed.

6.3 — Carry a windowed gauge in the heartbeat so the failure also drives the incident machinery (grouping + AI summary + dashboard), not just a log line:

"metrics": {
  "queue_failures_per_window":  2,      // retryable failures since last beat
  "poison_messages_per_window": 1,      // > 0 → MONSTER opens an incident
  "queue_breaker_open":         false   // true → MONSTER opens an incident
}

Applying this to your own choke points. Ask: “if this exact line fails silently for an hour, does a customer notice before we do?” Typical TaxStreem examples: queue result handlers (this case), filing dispatch to NRS/Rev360, webhook deliveries from AI services, ledger posting, bank-sync ingestion. For each:

FieldTypeMeaning
1. ClassifycodeNon-retryable (ack + report) vs retryable (retry/breaker). Never let a poison payload loop.
2. EventpushPOST /api/webhook with domain "<service>:<component>:<resource>", level "error" — immediate visibility
3. GaugebeatAdd a per-window counter/boolean to your heartbeat metrics — durable incident trail
4. ThresholdserverIf it warrants an incident, add the rule to heartbeat detectPayloadIncident() (see queue_breaker_open as the template)

Reference implementations to copy from: taxstreem-numen/internal/infrastructure/messaging/subscriber.go (classification + breaker) and taxstreem-numen/internal/monitoring/reporter.go (ReportCritical + windowed counters).

7 · Alerts & channels

Incidents fan out to the channels configured in Settings — /api/integrations/slack, teams, gchat, clickup — and email via POST /api/alert ({ serverName, endpoint, timestamp } → “CRITICAL: X is DOWN” to the configured recipients).

Email alerts are suppressed on staging. If NEXT_PUBLIC_ENVIRONMENT is set in MONSTER's environment, /api/alert returns { skipped: true } and sends nothing. This is the #1 “why didn't I get the email?” question — incidents still appear on the dashboard; only email is muted.

8 · Endpoint reference

FieldTypeMeaning
POST /api/heartbeatpush · 30 minLiveness + windowed metrics. Requires registered service_key
GET /api/heartbeat?serviceKey=readRecent heartbeats for a service
POST /api/otel/routespush · per windowPer-route telemetry; OTEL aliases accepted; opens endpoint_error incidents
POST /api/otlp/v1/metricspush · OTLPStandard OTel ExportMetricsServiceRequest; service.name = service_key
POST /api/webhookpush · on eventGeneric event log ({domain,data,level}) or status push ({serverId,status,...}). Used for mission-critical events
POST /api/alertpush · on incidentEmail alert. Delivered when NEXT_PUBLIC_ENVIRONMENT is production/unset; suppressed for staging/dev/other
POST /api/monitorpull · on demandActively ping a list of URLs; stateless
GET/POST/DELETE /api/servicesadminList (with computed status/trends), register, remove services
GET /api/reportsreadHistorical reporting data
GET/PUT /api/settingsadminAlert recipients, integrations, thresholds
POST /api/analyzeAIAd-hoc AI analysis of current state
GET/POST /api/threat-huntAISecurity-focused sweep
GET/POST /api/topologyadminService dependency graph shown on the dashboard
POST /api/mcp/mcpMCP · agentMCP server (Streamable HTTP). 8 read-only SRE tools. Auth: Bearer MCP_AUTH_TOKEN

Incident-opening thresholds (heartbeat payload):

FieldTypeMeaning
error_rate > 0.10incidentdegraded_payload
error_rate > 0.05statusDEGRADED (no incident yet)
memory_mb > 1800incidentdegraded_payload
memory_pct > 90incidentdegraded_payload
db_pool_saturation > 0.90incidentdegraded_payload
queue_breaker_open = trueincidentdegraded_payload — queue consumer paused
poison_messages_per_window > 0incidentdegraded_payload — non-retryable payloads rejected
no heartbeat for 38 minincidentmissed_heartbeat
route error_rate > 0.10 or status ≥ 500incidentendpoint_error (via /api/otel/routes)

9 · Pitfalls

  • Beating every 30 min exactly is too slow in practice. Network jitter puts you over the 32-minute DEGRADED line. Beat every 25–28 minutes.
  • 404 on first heartbeat — the service isn't registered yet. Register in Settings (or POST /api/services) first; the key is derived from the name.
  • No email on staging is intentional — see section 7.
  • Windowed metrics must reset after each beat (swap-to-zero), or your error_rate becomes a lifetime average and never crosses a threshold.
  • Monitoring must never take the service down — every push should be fire-and-forget with a timeout and a swallowed error (see the snippets above).
  • Green ≠ healthy. If your workflow crosses a queue, instrument the consumer (section 6) — the ai-core → numen incident produced zero red endpoints while transactions were stuck for hours.