Referral Alerts: Measure the Action Taken After the Notification
Audit whether alerts reach the responsible team, receive a useful response and create redundant work; tune notification rules.
A sent or opened event establishes only that event. It does not establish that the intended administrative action occurred. An alert can be delivered to a shared inbox nobody owns, opened by someone who already handled the case through another channel, or dismissed because the referral it refers to was closed yesterday.
Why delivery metrics are not effectiveness metrics
The Agency for Healthcare Research and Quality frames health IT success in terms of its impact on both clinical and administrative workflow, and its workflow assessment toolkit is aimed at people planning, implementing, and using health IT in ambulatory care (AHRQ). Use that general perspective to inspect the work around a notification. The ledger below is an original method for recording action and uncertainty, not an AHRQ alert-effectiveness instrument.
This is a different question from the throughput measures covered in referral operations dashboard metrics or the worklist states described in referral tracking in healthcare. Those articles define broader metrics and actionable worklist states. This one follows the notification event and the evidence available afterward.
Step 1: Define what action each alert is intended to trigger
Start with an inventory, not a report. For every alert rule currently firing, record:
- Trigger: the event or condition that fires it.
- Intended owner: the role, not the individual, expected to respond.
- Intended action: the specific next step, such as call the patient, request records, reassign, or close.
- Expected window: how long after delivery the action should normally occur.
- Suppression conditions: cases where the alert should not fire, such as a referral already in a terminal status.
If the intended owner or action is unknown, assign a rule-review owner and resolve the dependency. Do not retire or suppress it solely because the inventory is incomplete. Keep this inventory as the first tab of the ledger.
Step 2: Join notification delivery to acknowledgment and action
The core artifact is a flat ledger with one row per alert instance. Suggested columns:
| Column | What to record |
|---|---|
| alert_id | Unique identifier for the instance |
| rule_name | From the inventory in Step 1 |
| referral_id | The referral the alert concerns |
| fired_at | Timestamp the rule fired |
| delivered_to | Role or queue that received it |
| acknowledged_at | Timestamp someone marked it seen, if your tooling captures this |
| action_taken | Intended action observed, other action observed or no action established yet |
| action_at | Timestamp from the authorized source that records the action, with source identified |
| prior_action_at | Most recent action on the referral before the alert fired |
| review_class | Action observed, duplicate, stale, blocked, pending or unknown (Step 3) |
| observation_cutoff | When this record was assessed and whether its response window had elapsed |
Choose an authoritative source for each event and preserve that source in the ledger. Action evidence may live in the referral system or another authorized workflow log. Compare event times with ingestion times before deciding the order. A prior action can explain a redundant notification; a later action alone cannot prove that the alert prompted it.
If your systems cannot produce these joins automatically, sample instead. A bounded manual review can identify cases for investigation. Report its selection and coverage; it does not necessarily reveal the distribution across all alerts.
Step 3: Classify duplicate, stale, and unactionable alerts
Use a written review rule so the same alert is not counted in several headline categories. Preserve additional reasons separately when they overlap.
| Primary review class | Evidence required |
|---|---|
| Action observed | The intended action is recorded within the defined observation window |
| Duplicate | Evidence shows the same request already had a notification or action that made this instance redundant |
| Stale | The requested action no longer applied at the relevant delivery event |
| Blocked | The assigned owner could not complete the action, with the documented dependency recorded |
| Pending | The observation ended before the response window or assigned follow-up was complete |
| Unknown | Evidence is insufficient to establish the action or reason for its absence |
Inspect possible explanations instead of assigning causes from the category. Overlapping rules can create a duplicate; delayed updates, configuration or another dependency can produce a stale event. A blocked action might require missing information or a different owner. Check the actual trace before changing a rule.
A synthetic review
In an invented cohort of 200 alerts from one administrative follow-up rule, 90 have a subsequent action recorded, 40 are confirmed duplicates, 25 are stale, 20 are blocked, 15 remain pending and 10 are unknown. The categories sum to 200 under the hypothetical review rule. The 90 observed actions are 45% of the entire reviewed cohort, but that is not a causal success rate. Pending and unknown cases remain visible.
Before considering suppression of a duplicate, inspect the related action and its current state. A prior alert firing does not establish that anyone completed the work. Before routing blocked items elsewhere, obtain acceptance from the receiving role. These are investigation questions, not recommended universal timing or suppression rules.
Step 4: Test a rule change on a bounded queue
Start with synthetic or historical replay in an approved environment, then use a bounded rollout only after the responsible owners accept the evidence and recovery plan. A focused change can make interpretation easier. It does not remove confounding from staffing, volume, case mix or concurrent changes; record those conditions.
Suggested experiment record:
- Rule changed and the exact before and after logic.
- Queue or location in scope.
- Start and end dates.
- Compared measures: raw counts and shares by review class, observation coverage, completed response windows and time to recorded action.
- Missed-action check: a separately defined cohort of referrals that met the rule condition, including those for which no alert was sent. Check expected actions against the approved process.
The alert ledger alone cannot reveal every item that received no alert. Keep the rule-eligible cohort and its action review separate from the notification denominator. Investigate a missing action before attributing it to suppression.
If you are evaluating referral coordination software alongside this work, the referral coordination software pillar and a demo provide a place to ask which alert routing, source events and ownership records the proposed configuration can demonstrate. None of those specific behaviors is assumed here.
Step 5: Review workload and missed actions before expanding
After the bounded test, review two things with the people who received the alerts.
Workload: did the change reduce the time spent dismissing or re-reading alerts? Ask the staff directly and compare the ledger. Capacity freed this way is capacity, not cash; keep it separate from any financial assumptions, consistent with the practice of separating activity metrics from outcomes and financial assumptions in Linear Health's evaluation kit (Operational AI Evaluation Kit).
Missed actions: inspect rule-eligible referrals with no expected action, including those that received no alert. Establish the sequence and dependencies before assigning a cause. If the change contributed to a missed action, follow the agreed stop and recovery plan.
An absence of observed failures in a small test does not guarantee safe expansion. The responsible owners should review coverage, unresolved cases, recovery evidence and the intended rollout scope before deciding the next step.
Applying this to vendor evaluation
Linear Health's Operational AI Evaluation Kit distinguishes claimed, demonstrated and production-verified evidence and asks buyers to define completion in observable terms. Apply that to the task being purchased. Delivery may be a valid outcome for a notification-only service; it does not establish that a staff action or referral journey is complete.
Ask each vendor to trace an alert, a duplicate and an unresolved action. Inspect the source event, recipient, response evidence and work still assigned to staff. Request an export that distinguishes missing data from a recorded no-action result. Do not infer those fields from a screen that displays a notification.
Linear Health lists athenahealth, eClinicalWorks, NextGen and Greenway Health as established integrations. Field-level actions, event coverage and configuration require demonstration. The operational AI evaluation kit can guide the evidence record and open questions; export or keep the record in your authorized system. For a focused demo, bring your rule inventory and synthetic traces, including a rule-eligible referral that received no alert.
Frequently asked questions
What is the simplest first version of the ledger?
How large a sample is needed?
Should alerts be turned off while testing?
Does an action after an alert prove the alert worked?
Does this article cover clinical urgency rules?
Sources
Sources accessed November 4, 2026.
Linear Health publishes educational articles about operational AI for healthcare.