Monitoring & metrics

Audience: developers & AI agents · Scope: CloudWatch metrics, the deferred-put queue, log types and the liveness probes · Last reviewed: 2026-07-20

TL;DR — Most of this site's work happens unattended, so "did it work?" is answered by CloudWatch metrics and KLog log types, not by anyone watching a screen. Metrics are usually queued to the database and drained by a cron, not sent inline. Coverage is uneven — some subsystems report richly, others fail completely silently — so know what is watched before relying on it.

The two ways a metric is sent

Everything goes through AwsMetrics (data/customization/system_overrides/AwsMetrics.php).

  scheduleMetricDataPut(namespace, metric, value, unit, dimensions)      ← the normal path
        └─ INSERT into #__configbox_external_scheduled_metric_puts (time_sent = NULL)
                 │
                 │  cron: cli/cb_aws_put_scheduled_metrics.php
                 ▼
           executeScheduledMetricDataPuts()
                 └─ putMetricData(...) → CloudWatch
                      success → stamp time_sent
                      failure → store exception_message   ⚠ never retried

  putMetricData(namespace, metric, value, unit, dimensions)              ← immediate, used sparingly

Prefer scheduleMetricDataPut(). A web request or CLI job should not wait on the CloudWatch API, and a metric failing must never break the work it was measuring. putMetricData() is for the few places that need the call to happen there and then.

cb_aws_put_scheduled_metrics.php is therefore load-bearing: until it runs, no metric has actually been sent. Its schedule is not recorded in this repo — see the crontab gap.

cb_aws_prune_metric_log_entries.php deletes sent rows older than 30 days, 500 at a time.

Credentials & dimensions

• Credentials come from an INI file in the private data store — private/custom_media/aws/bc-website-metric-putter@betacalco.ini — loaded via the AWS SDK's CredentialProvider::ini(...) and memoised. It is a dedicated, narrowly-scoped IAM user; it is not the site's general AWS identity.
• Every metric carries an Environment dimension (live / staging / dev), merged into whatever dimensions the caller passes. Always filter dashboards by it — otherwise dev and staging noise lands on the same graph as live.
• Log type: custom_aws_metrics.

Namespaces

Namespace	Covers
Infor-Rest-API	IDO REST calls — see infor/rest-api.md
Infor-DB-Checks	the Infor DB liveness probes
Infor-Order-Notifications	order-line status notifications
BOM-Cache	BOM cache refresh — counts, sizes, per-item timings
Margin-Analysis-Tool	costing sheet generation, including its OOM failures
Specsheet-Cache-Refresh	cached spec-sheet regeneration
Acoustic-Calculator	the acoustic calculator
Chat-Bot	the Vertex-AI-backed chatbot
BigQuery-Usage	monthly bytes-billed and job counts per service account
Test-BOM-Calculation	production BOM/CPQ metrics — see the naming warning below
Test1-Pipedrive-Sync	production Pipedrive sync, webhook queue and API exception metrics
Test1-Cold-Quotes	production cold-quote prompt, landing-page and feedback metrics

⚠️ Three production namespaces are named as if they were experiments. Test-BOM-Calculation, Test1-Pipedrive-Sync and Test1-Cold-Quotes are used throughout live code paths — the webhook queue, the CPQ fail observer, the cold-quote landing page. They are not test data. Renaming them would orphan any existing dashboards and alarms, so treat it as a deliberate migration rather than a tidy-up. Recorded in known issues (docs/_known-issues.md).

Probes

Three scripts exist purely to answer "is Infor reachable?", from two angles:

Script	Angle	Output
cb_check_infor_db.php	PDO SELECT 1	exit 0/1, --output-mode=json|prometheus
cb_crafty_infor_db_probe.php	same, plus a connection reset and an infor_db_up field	JSON
cb_crafty_infor_ping_probe.php	ICMP ping of the Infor host	infor_vpn_up, infor_vpn_probetime_ms

cb_aws_send_metric_infordb.php and cb_aws_send_metric_apptrix_vpn.php push the same signals into CloudWatch (connect success/failure/time, ping success/failure/time), dimensioned by IP.

Splitting database reachability from VPN reachability is deliberate: they fail independently, and knowing which one is down tells you whether to look at the network or at SQL Server.

Logs

KLog writes per-area log types; the full table lives in scheduled-jobs.md. The ones that answer "why did this integration fail?":

Question	Log type
Can we reach the Infor database?	custom_infor_pdo
Can we authenticate to the Infor REST API?	custom_infor_api_connect_fails
Why is a sheet picker empty?	custom_google_sheet_cache
Did a SQL export run, and what did it write?	custom_sql_exports (plus the #__configbox_external_sql_export_log table)
Did metrics actually get sent?	custom_aws_metrics

What is not watched

Worth knowing before you assume an alert would have fired:

• The Infor MSSQL path has no CloudWatch metrics at all — only the custom_infor_pdo log. The REST path is instrumented; the database path is not. (Margin-Analysis-Tool is the exception, and only for the costing sheet.)
• cb_order_line_status_notify.php exits 0 on failure. It catches Throwable and bumps a counter, so cron-level alerting can never fire for it.
• A failed metric put is never retried. The drain query excludes any row that already has an exception_message, so a metric that fails once is silently dropped — and because the failure is the signal, an outage that breaks metric delivery also hides itself.
• Nothing watches whether the crons ran at all. There is no heartbeat metric per job; a job that stops being scheduled looks identical to a job with nothing to do.
• Most full-replace Google Sheet exports report nothing on success, so a stale tab is the only symptom — and for the SQL exports a stale tab legitimately means "the query returned no rows".

Gotchas & caveats

• Metrics lag by however often the drain runs. A dashboard that looks flat may just mean cb_aws_put_scheduled_metrics.php has not run.
• The queue table grows if the drain is broken — pruning only removes rows that were successfully sent. A large #__configbox_external_scheduled_metric_puts with time_sent IS NULL is itself a symptom.
• Metric name casing is inconsistent (BOM-Cache-* alongside Bom-Cache-Size-*), which matters because CloudWatch metric names are case-sensitive.
• Dev and staging emit real metrics. They are separated only by the Environment dimension, not by namespace or account.

Possible follow-ups

• Retry failed metric puts (with a bounded attempt count) instead of dropping them.
• Add a heartbeat metric per scheduled job so a job that stops running is visible.
• Give the Infor MSSQL path the same CloudWatch coverage as the REST path.
• Migrate the three Test* namespaces to real names, coordinating with whatever dashboards exist.

Related docs

• Scheduled jobs & CLI — what runs, and its log types
• Infor / ERP · Google Sheets
• Known issues (docs/_known-issues.md)