How to Set Up a Monitoring Server with Automated Alerts for SIMRS & ERP

At 02:15 WIB, the SIMRS service at a type C hospital stopped responding. There was no notification at all. The IT team only found out at 06:40 when registration staff complained they could not create a SEP, and for 4.5 hours patients were served with paper records that then had to be re-entered. Incidents like this are almost always not about complicated technology, but about the absence of a system that gives advance notice. Server monitoring is not just putting CPU graphs on a dashboard that is never opened. What determines it is a combination of three things: the right metrics, reasonable thresholds, and automated alerts that reach people who can act — quickly, with enough context to start an investigation. This article discusses the concrete steps, from choosing metrics to alerts arriving in the ops team's Telegram and WhatsApp.

Start with the Right Metrics, Not All Metrics

The foundation of healthy monitoring still refers to the four golden signals: latency, traffic, error, and saturation. In healthcare systems, these signals need to be translated into real business flows — patient registration, pharmacy, cashier, and BPJS bridging. A good alert answers the question "are users affected?", not "is this number going up?". Metrics that cannot be acted on should not be made into alerts at all because they only add noise.

For the infrastructure layer, the following thresholds have been tested and do not trigger excessive false alarms: CPU above 85% for 5 consecutive minutes, RAM usage above 90% for 10 minutes, and disk above 80% (warning) and 90% (critical). Disk thresholds should be stricter on database servers because MySQL binlogs and patient scan files pile up quickly. Load average above 2x the number of vCPUs for 10 minutes, as well as swap usage above 20%, are also worth making early warnings.

At the application layer, measure what users experience. Real examples: p95 latency of the registration endpoint above 1.5 seconds for 5 minutes, HTTP 5xx error rate above 1% of total requests, database connection pool usage above 90% for 3 minutes, and bridging job queue backlog of more than 50 jobs. For BPJS/SatuSehat integration, also monitor response times above 3 seconds and spikes in authentication failures, because problems on the partner side often appear as timeouts that are not visible in server metrics.

Don't forget the "boring" but fatal things: SSL certificate validity of less than 14 days, failed daily backups, and ETL cron processes that do not finish within their time window. These three things rarely make it onto dashboards, even though they are often what triggers major incidents later.

Building a Monitoring Stack: Prometheus 2.51, Node Exporter, and Alertmanager

The minimal stack sufficient for 20–50 targets is Prometheus 2.51.1 as the metrics collector, node_exporter 1.8.2 for host metrics, blackbox_exporter 0.25.0 for HTTP/TCP probes, mysqld_exporter 0.15.1 for databases, Alertmanager 0.27.0 for notification routing, and Grafana 11.1 for visualization. All of this can be run via Docker Compose on a single VM with 2 vCPU/4 GB. Set scrape_interval to 15 seconds, evaluation_interval to 15 seconds, and retention to 30 days — for 30 targets, disk requirements usually range from 15–25 GB.

Install node_exporter on every server with a systemd unit so it automatically starts after reboot, then add a blackbox job to monitor critical endpoints such as the SIMRS login page and bridging endpoints. For probes, use the http_2xx module with a 5-second timeout so network problems are detected quickly before users report them. After that, write alert rules in the rules/simrs.yml file as in the following example.

groups:
- name: simrs-prod
  rules:
  - alert: DiskHampirPenuh
    expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
    for: 10m
    labels: { severity: critical, team: ops }
    annotations:
      summary: "Disk {{ $labels.instance }} tersisa {{ printf \"%.1f\" $value }}%"
  - alert: ErrorRate5xxTinggi
    expr: sum(rate(http_requests_total{job="simrs",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="simrs"}[5m])) > 0.01
    for: 5m
    labels: { severity: critical, team: ops }
    annotations:
      summary: "Error 5xx SIMRS di atas 1% selama 5 menit"

Always validate before reloading: run promtool check rules /etc/prometheus/rules/simrs.yml, then curl -X POST http://localhost:9090/-/reload if Prometheus is run with the --web.enable-lifecycle flag. This habit prevents Prometheus from failing to start just because of one YAML indentation error — a mistake that often takes down all monitoring without anyone realizing.

For environments with small teams, add a lightweight cron-based exporter if business metrics are not available over HTTP. Simply create a script that writes metrics to a text file and then exposes them through node_exporter --collector.textfile.directory. This approach is used to send the number of failed SEPs, the number of waiting queues, or daily backup status to Prometheus without building a special application.

Routing Alerts to Channels the Team Reads and Avoiding Alert Fatigue

Alertmanager determines whether warnings actually arrive. Practical routing rules: critical severity is sent to the Telegram ops group plus a WhatsApp gateway (for example, WhatsApp Business API or a local gateway provider) and phone escalation after 10 minutes without acknowledgment; warning severity only needs Telegram. Use group_wait: 30s, group_interval: 5m, and repeat_interval: 4h for critical and 24 hours for warning. Without this, one server going down can produce 40 messages in a minute and the team stops reading all of them.

Use inhibition rules so that derived alerts are not sent when their root cause has already been reported. Example: if NodeDown is active, suppress all CPU, RAM, and disk alerts from the same instance. Also complete every annotation with a runbook link, for example runbook_url: https://wiki.internal/runbook/disk-penuh, so the on-call staff do not have to guess the first step at three in the morning.

Finally, set an explicit weekly on-call schedule and conduct a monthly alert review. Count how many alerts actually require action; if below 70%, the thresholds need improvement. Every incident should be closed with a brief note: what failed to be detected, and what metrics need to be added. This is the part that distinguishes living monitoring from a dashboard that is merely on display.

Best Practices

1. Alerts must be actionable. Every alert must have an owner, a clear user impact, and initial steps in the runbook.
2. Use for of at least 5 minutes. Prevents notifications from momentary spikes while backups or deployments are running.
3. Monitor from outside, not only from inside. External HTTP probes detect DNS, firewall, and certificate issues that are invisible to host metrics.
4. Test the alert path every quarter. Shut down one service in staging and ensure the message actually arrives in Telegram/WhatsApp within 60 seconds.
5. Retain metrics for at least 30 days. Needed for capacity trend analysis when deciding on adding servers or upgrading databases.

FAQ

Is Prometheus suitable for monitoring a SIMRS running on-premise?

Very suitable. Prometheus is pull-based and lightweight, so it is safe to run on a separate VM on the hospital network. For multi-site architectures such as Yogyakarta and Magelang, run a local Prometheus at each site and then aggregate with Prometheus federation or Thanos so that dashboards remain centralized.

How much cost and time does implementation take for one hospital?

For 20–30 servers, basic implementation usually takes 3–5 working days: installing the stack, drafting 15–25 alert rules, integrating Telegram/WhatsApp, and creating 3 main dashboards. The main costs are VMs and engineer time, not licenses, because all components are open source.

How do you avoid false alerts during the nightly backup process?

Use a longer for on I/O metrics, or add an inhibition rule that suppresses disk and CPU alerts while the backup job is running. Another alternative is to temporarily raise thresholds via the Alertmanager API, but this method should be used sparingly.

Can monitoring also monitor POS and ERP applications?

Yes. Expose metrics from the POS or ERP application via a /metrics endpoint using a Prometheus client library (Go, PHP, or Node.js), then add a new scrape job. Business metrics such as failed transactions and stock synchronization queues are actually more valuable than server metrics alone.

Good monitoring turns incidents from "found by users" into "found by the system, handled before patients feel the impact". Start with the five most critical alerts this week, measure their accuracy, then expand gradually. If you manage a SIMRS, SIM Klinik, ERP, POS, or website and need assistance building monitoring and automated alerts that match operational workflows, our IT consulting team provides application development and IT consulting for Yogyakarta, DIY, and Magelang, Central Java — including infrastructure audits, runbook preparation, and internal team training so that the systems built are truly used.

This article is part of the knowledge base of Nugroho Setiawan, an IT consultant in Jogja (Yogyakarta). Related solutions: PilarPOS SIMRS (Hospital Information System) BPJS & SatuSehat Integration. Need a free consultation?