Target operating model

Whole-system observability and SLO design

A measurable service model from source ingestion through Redis, recognition, warehouse, Accounting Central, posting, and reporting. The design starts with customer outcomes and preserves enough context to explain failures without creating unbounded metric cost.

Status: proposed internal standard · objectives require Product, Support, Accounting, and Platform ratification

Proposed engineering plan

This separate page covers APIs, jobs, failures, workflow, streams, processors, data stores, Accounting Central, reporting, infrastructure, correlation, dashboards, alerting, and SLOs. It is a redacted proposal—not a ratified customer SLA.

Principles

Observe outcomes, integrity, and causes separately

A CPU alarm cannot tell whether a customer sync is correct. A job status cannot explain which dependency or task failed. RevRec needs all three layers, joined by correlation context.

Layer 1

Customer outcomes

Availability, job terminality, end-to-end completion time, data freshness, reconciliation correctness, and report availability. These drive SLOs and error budgets.

Layer 2

Pipeline progress and integrity

Stage transitions, queue age, stream accounting, retry/DLQ, message counters, row counts, and invariant deltas. These show where work stopped or disappeared.

Layer 3

Runtime and dependencies

ECS health, process memory/GC, API latency, Redis, SingleStore, Camunda, SQS, Lambda, EFS, and network signals. These explain why an outcome or pipeline SLI moved.

SLA versus SLO

An SLA is a customer contract with remedies and exclusions. An SLO is the internal reliability target used to operate the system. Start by measuring SLIs and running internal SLOs for at least two weeks. Publish contractual targets only after the data, ownership, maintenance exclusions, and support process are stable.

System boundary

The customer journey being measured

Recognition executes inside orders processing rather than as a separate phase. Warehouse export runs the visitor again. AC adds a 17-step branch before the pipeline returns to aggregation.

Source / HotGlue
Export or UI upload
API + sync entry
Request accepted and job created
Camunda loading
Convert, load, processData
Redis orders stream
Publish, claim, process
Recognition
SalesOrder + RevenueArrangement
Warehouse
Export and derived facts
AC / aggregation
17 steps, posting, reclass
Reports
MSTR / reporting availability

Current state

Useful signals exist, but they do not form one operating model

The problem is not “zero observability.” It is fragmented observability: service resource alarms, text logs, several custom metrics, and no reliable path from a page to the affected job and customer outcome.

Capability Today Value Material gap
ECS infrastructure Standard Container Insights, AWS/ECS CPU/memory, service alarms Cluster/service trends No enhanced task/container drill-down; symptom alarms lack job impact
Application logging Component prefix with clientId, jobId, level, userId, timestamp Manual job/task correlation Mostly text; no universal event schema or trace propagation
Custom metrics ProcessorError/Crash, job governor, reporting, export, cache, DB/Redis health Several local dashboards and alarms are possible No unified lifecycle, stream integrity, process heap, or SLO metrics
Heap diagnostics Health heapUsed, manual /heap, periodic snapshot service Can capture V8 state Periodic capture disabled in prod; heapUsed alone misses RSS/external/GC
Job state Redis + Camunda, warehouse JobStatus2 query surface Phase and terminal status No continuously evaluated stuck/terminality/freshness SLI
Tracing None found — No single trace from ingress through jobs, streams, processors, and warehouse
Resource tagging ECS managed cluster/service tags; service Name tag Basic ownership by service name No propagated owner, criticality, runbook, cost, or data-classification tags

SLIs and SLOs

Proposed internal objectives

Targets below are a starting point. Timeliness should be tiered by input size, job type, and product contract; a single latency target across all tenants would hide workload reality.

Customer journey SLI definition Initial internal objective Exclusions that must be explicit
API availability Valid requests receiving a non-platform-error response ÷ valid requests ≥99.9% rolling 30 days Client 4xx, documented maintenance, upstream customer auth failure
Sync admission Accepted jobs entering processing within target ÷ accepted jobs ≥99.5%; p95 admission latency by priority/weight Customer-requested schedule time, explicit org lock policy
Sync terminality Jobs reaching explicit SUCCESS/FAILED/TIMEOUT ÷ jobs started ≥99.9%; zero silent RUNNING jobs beyond budget Explicit customer cancellation remains terminal and visible
Sync success Successful jobs ÷ terminal non-customer-cancelled jobs Baseline first; target by job type and driver Invalid source data should be separately classified, not hidden
Sync timeliness Successful jobs completing within size-tier budget ÷ successful jobs ≥99% after two-week p95/p99 baseline Upstream export time separated from RevRec processing time
Data freshness Elapsed source-complete → report-queryable ≥99% within customer-tier target Customer source unavailability
Stream integrity (completed + pending + durable DLQ) ÷ published 100%; unknown deletion = 0 None—this is a safety invariant
Financial correctness Completed jobs without unresolved material recon/posting divergence ÷ completed jobs Set by Accounting policy; material silent divergence = 0 Documented accepted source-data exceptions only
Report availability Successful report requests ÷ valid report requests ≥99.9%; latency target by report class User cancellation and invalid filters
Error-budget policy

Page on fast burn rather than single threshold crossings. A useful initial pattern for a 30-day SLO is a fast page when both the one-hour burn is at least 14.4× and the six-hour burn is at least 6×; open a ticket for slower multi-day burn. Recalculate windows when the SLO period or objective changes.

Metric catalog

What every major component should emit

Use distributions/histograms for latency and size. Counters should be monotonic per process/export interval. Gauges need a collection time and freshness indicator.

Component Core metrics Why they matter Primary alert
API Gateway / API service requests, 4xx/5xx by route class, latency p50/p95/p99, throttles, Lambda duration/error/concurrency/cold start Ingress availability and customer latency SLO burn, sustained 5xx, throttle exhaustion
HotGlue / source integration webhooks received, accepted/rejected, source-complete lag, export status, files/bytes/rows, callback retries Separates upstream delay from RevRec delay Completed upstream export with no admitted job
Camunda active processes, incidents, external tasks by topic, fetch/lock/complete/fail, lock age, retries, process duration Shows orchestration stalls and worker availability Incident count, oldest locked task, no completion progress
Job governor/admission waiting/running jobs and weight, admission latency, oldest wait, duplicate prevented, lock contention, capacity budget Explains why jobs have not started Oldest high-priority job exceeds budget
Loading workers phase attempts/outcomes/duration, file bytes/rows, conversion/load throughput, retry reason, processData failure Measures ingestion performance and data quality Phase failure burst or no throughput with active work
Redis streams published, read, claimed, redelivered, completed, DLQed, pending, oldest pending age, consumer count, owner changes Provides the processing-integrity equation Unknown delta, retry exhaustion, pending age
Order processors batch size, in-flight, payload bytes, arrangement IDs/order, order duration/outcome, heap/RSS/external, GC pause, event-loop lag Connects workload shape to resource failure OOM, low heap headroom, long GC, no progress
Recognition engine orders and arrangements processed, model path, rule duration, schedule count, validation/error codes, output size Explains expensive or failing accounting work Error-code burst or p99 duration regression
Warehouse phase / SingleStore query count/error/latency by fingerprint, pool active/idle/wait, rows/bytes, load duration, lock wait, leaf/aggregator health, disk/replication Measures data-plane latency and saturation Connection starvation, failed load, unhealthy leaf, disk risk
Accounting Central step 1–17 duration/outcome, input/output rows, skipped transactions, mapping failures, recon errors, unbalanced amounts Localizes AC failures and correctness risks Material recon or step failure; missing step progress
Aggregation / posting / reclass phase duration/outcome, records, journals, posting retries, external response class, reclass count, unbilled delta Tracks terminal financial processing Posting divergence or repeated retry
Reporting / MSTR request rate/error/latency, report/export duration, cache hit/miss/bytes, report freshness, active exports, DB health Measures the customer read path Availability burn or stale report data
ECS / EC2 desired/running/pending, task starts/stops/restarts, stop reason, container health, per-task CPU/memory/network/storage, ASG capacity Runtime availability and capacity running < desired, crash loop, no placement capacity
EFS / S3 / SQS EFS throughput/credits/IO, S3 request/error/latency, SQS visible/in-flight/oldest age/DLQ Supporting storage and async delivery health Oldest message, DLQ, storage errors, EFS saturation

Correlation contract

Structured events, traces, and controlled dimensions

Keep high-cardinality identity searchable without turning every job into a paid custom metric time series.

Metric dimensions: bounded values only

  • environment and profile
  • service, component, operation
  • jobType, stage, driver
  • messageType, dependency
  • outcome, controlled errorClass
  • taskDefinitionFamily, not task ID

Log/span properties: high-cardinality identity

  • orgId, jobId, jobGroupId
  • orderId, messageId, consumerId
  • taskArn, containerId
  • traceId, spanId, source request ID
  • queryFingerprint, not raw SQL as a dimension
  • Payload metadata and sizes, never secrets or full customer documents

Canonical event envelope

{
  "timestamp": "2026-09-09T06:16:21.568Z",
  "event": "stream.message.claimed",
  "severity": "INFO",
  "environment": "prod",
  "profile": "b",
  "service": "processor",
  "component": "orders-consumer",
  "stage": "orders",
  "outcome": "success",
  "durationMs": 12,
  "batchSize": 20,
  "orgId": "Org A",
  "jobId": "Job A2",
  "messageId": "message-alias",
  "consumerId": "consumer-alias",
  "taskArn": "...",
  "traceId": "..."
}
Existing high-cardinality review

Current reporting/export metrics use OrgId, UserId, and TemplateId dimensions; those represented 395 of 561 observed identities in the revlock namespace. Keep a metric dimension only when it directly powers a bounded operational decision. Otherwise, store the ID as an EMF property or span/log attribute and query it on demand.

Dashboards

Four views, four audiences

A single mega-dashboard becomes a wall of symptoms. Each view should answer one operational question and link to the next level.

Product + leadership

Service SLO and error budget

  • API availability and latency SLO
  • Sync terminality, success, and timeliness
  • Data freshness and report availability
  • Error-budget remaining and burn by journey
  • Affected organizations and incident count
On-call

Pipeline operations

  • Jobs by current phase, type, driver, and outcome
  • Oldest waiting/running stage and no-progress duration
  • Redis published/completed/pending/DLQ accounting
  • Camunda incidents and external-task age
  • Dependency health and deployment annotations
Revenue Engine

Processor and heap

  • Per-task heap, RSS, external, and headroom
  • GC pause and event-loop delay
  • Batch size, in-flight, payload bytes, arrangements/order
  • Redelivery, ownership changes, retry exhaustion
  • OOM/task-stop timeline with org/job properties
Data platform

Redis and SingleStore

  • Redis shard memory, CPU, evictions, latency, replication
  • Stream pending age and consumer health
  • SingleStore leaves/aggregators, disk, connections, locks
  • Query fingerprint latency/error and pool wait
  • Warehouse row/byte throughput and phase freshness

Alert policy

Page only when a human can act now

Resource warnings are useful for diagnosis and capacity planning, but customer impact, progress loss, and integrity violations should dominate paging.

Severity Examples Route Required context
Page Fast SLO burn, OOM/crash loop, unknown stream delta, active-message deletion, material recon divergence, running < desired 24×7 on-call Runbook, owner, service, profile, first/last seen, task, affected-job query link
Urgent ticket Slow SLO burn, heap baseline growth, p99 regression, increasing redelivery, queue age trend, worker scaling suspended unexpectedly Owning team backlog with due date Trend, baseline comparison, suspected deployment/config change
Dashboard only Momentary CPU peak, expected deployment stop, transient dependency retry, low-volume single error No notification Searchable event and deployment annotation

Processor-specific alert corrections

Collection architecture

Use native AWS coverage, then add application semantics

The fastest path is not a platform replacement. Upgrade ECS visibility, standardize structured events/metrics, and layer tracing onto high-value journeys.

Infrastructure

Container Insights enhanced

Upgrade the cluster setting from enabled to enhanced. Retain the performance log group long enough to cover the incident-review window. Use curated task and container dashboards.

Metrics + logs

EMF with a schema guard

Emit Embedded Metric Format events for job, stream, process, and dependency signals. Enforce an allowlist for metric dimensions; preserve high-cardinality context as properties.

Tracing

ADOT / Application Signals

Add OpenTelemetry to API and ECS with a CloudWatch agent sidecar/daemon. Propagate trace context explicitly through Camunda variables, job params, Redis stream fields, and SQS.

Instrumentation order

Instrument state boundaries first: request accepted, job created, stage entered/exited, message published/claimed/completed/DLQed, and job terminal. Auto-instrumentation is valuable for HTTP/Redis/SQL latency, but it cannot infer RevRec’s phase counters, correctness invariants, or customer outcome.

Retention and sampling defaults

Signal Suggested retention Sampling / volume control
SLO metrics 15 months or business reporting requirement Never sample aggregated counters
Operational metrics 3–15 months by cost/value Use bounded dimensions and rollups
Structured application logs 30–90 days hot; archive per compliance policy Sample repetitive success details; never sample errors/integrity events
Traces 7–30 days hot Head sample normal traffic; retain errors, high latency, and selected jobs
Heap profiles/snapshots Incident-specific, encrypted, short retention Canary only; snapshots may contain customer data

AWS references: ECS enhanced observability, Application Signals on ECS, and Embedded Metric Format.

Roadmap

A staged rollout with measurable exit criteria

Observability is complete only when an owner uses it in incident response and reliability review.

Phase Deliverables Exit criteria
0 · 48 hours OOM and retry/DLQ alarms, task-state stop events, enhanced Container Insights, affected-job runbook Synthetic processor death pages in <2 minutes with task/job query link
1 · Week 1 Canonical event envelope; job lifecycle, stream integrity, processor memory/GC metrics; three operational dashboards On-call can identify org/job/stage and integrity state without broad log scans
2 · Weeks 2–3 API and dependency tracing, Camunda propagation, warehouse fingerprints, internal SLO dashboard One trace links ingress to job/stage/message/dependency for sampled work
3 · Month 1 Error-budget policy, ownership/runbooks, cardinality/cost review, report/AC correctness SLIs Weekly reliability review acts on burn, toil, and top failure modes
4 · Ongoing SLO calibration, alert quality review, retention tuning, game days, customer SLA decision Pages are actionable; silent work loss is impossible by invariant

Ownership required

For the immediate production defect, continue to the heap and remediation playbook.