A calm customer incident report from your own notes

Input (invented):

## Who the report is for and the service involved
Our client Harbour Dental's online booking site, which we host and maintain (the practice manager is the reader)

## Your incident notes
Fri 3 Oct. 09:12 UTC monitoring alert: booking page returning 502. 09:20 on-call engineer acknowledged. 09:31 found the app server out of disk space, the log folder had grown after the 08:45 deploy turned on debug logging by mistake. 09:48 cleared old logs, restarted app, booking page back. 09:55 confirmed test booking works. Checked afterwards: 37 booking attempts failed during the outage per the app log, no data lost. Debug logging setting reverted at 10:05. Nobody noticed disk alert because the disk alert threshold was set at 95% and the disk went from 90% to full within minutes. We haven't decided yet whether to add log rotation. Mark was the one who did the deploy.

## Anything else the report may say (optional)
Support contact is [email protected]. We have not offered any credit.

Result:

## Read this first
The notes clearly cover the timeline, the cause the team found (the server ran out of disk space after a deploy switched on extra logging by mistake), the fix, and the impact count (37 failed booking attempts, no data lost). The year is not stated, and no follow-up action has an owner or date. The biggest gap before sending is **What we are doing next**: log rotation is still undecided, and the notes give no other follow-up. You need to decide what, if anything, you will commit to.

## Customer incident report
**Summary**
On Friday 3 Oct [year?], the Harbour Dental online booking page was unavailable. The cause was that the server running the booking site ran out of disk space. Service was restored at 09:48 UTC, a test booking was confirmed working at 09:55 UTC, and the incident is resolved.

**Impact**
People trying to use the online booking page received an error and could not book. The problem was first detected at 09:12 UTC, and the page was back at 09:48 UTC. According to the application log, 37 booking attempts failed during the outage. No data was lost. We are sorry for the disruption.

**Timeline** (all times UTC, Friday 3 Oct [year?])
- 08:45 – A planned update to the booking site was deployed. It switched on detailed ("debug") logging by mistake.
- 09:12 – Our monitoring alerted that the booking page was returning a "502" error, meaning the page could not load.
- 09:20 – The on-call engineer acknowledged the alert.
- 09:31 – The engineer found that the server had run out of disk space. The folder holding log files had grown after the 08:45 deploy.
- 09:48 – Old log files were cleared and the application was restarted. The booking page was back.
- 09:55 – A test booking was confirmed working.
- 10:05 – The debug logging setting was switched back off.

**Cause** (confirmed by our investigation)
The 08:45 deploy switched on debug logging by mistake. The extra log files filled the server's disk, and the booking application stopped responding. Our disk-space alert is set to trigger at 95% full. Disk usage went from 90% to full within minutes, so that alert did not give warning in time.

**Resolution**
We cleared old log files to free disk space and restarted the application, which restored the booking page at 09:48 UTC. We confirmed a test booking worked at 09:55 UTC. We then turned the debug logging setting back off at 10:05 UTC.

**What we are doing next**
[follow-up actions, owner and date — e.g. decision on adding log rotation (automatic clearing of old log files), not yet decided]

**Contact**
If you have questions about this incident, please contact us at [email protected].

## Status page post
**Resolved** – Online booking was unavailable on Fri 3 Oct. The issue was first detected at 09:12 UTC. The booking server ran out of disk space after a configuration mistake. We freed space and restarted the service, and booking was restored at 09:48 UTC. A test booking was confirmed working at 09:55 UTC. This incident is closed.

## Cover email
**Subject:** Incident report: online booking outage, Friday 3 Oct

Dear [practice manager name],

On Friday 3 Oct, your online booking page was unavailable. We first detected the problem at 09:12 UTC and restored the page at 09:48 UTC. The application log shows 37 booking attempts failed during this time. No data was lost.

The cause was a configuration mistake in an update we deployed at 08:45 UTC. It produced extra log files that filled the server's disk. That setting has now been switched off.

The attached report gives the full timeline and details. We are sorry for the disruption.

If you have any questions, please reply or contact [email protected].

Kind regards,
[sender name]

## Blanks to fill
- **[year?]** (report Summary and Timeline): check the incident date in your monitoring alert or deploy history.
- **[follow-up actions, owner and date]** (report, What we are doing next): your team's decision on log rotation and any change to the disk alert threshold. Add an owner and date only once these are agreed.
- **[practice manager name]** (cover email): your client records for Harbour Dental.
- **[sender name]** (cover email): whoever at your agency is sending the report.

## Claims to double-check
- The Cause is labelled "confirmed" because the notes state the finding directly. Make sure the team agrees the debug logging fully explains the disk filling up before sending it on that basis.
- "First detected at 09:12 UTC" is the monitoring alert time. The page may have started failing earlier, between 08:45 and 09:12, so check your logs before treating 09:12 as the start.
- "37 booking attempts failed" counts attempts in the app log, not unique people. Some people may have tried more than once, and others may have given up without the log recording an attempt.
- "No data was lost" is a strong statement. Confirm it covers all booking data, not only the failed attempts that were checked.
- The report explains why the disk alert did not warn in time. Decide whether you want the client to see this detail, because they may ask what you will change. The notes give no committed change.

## Before you send
- Check every time and the 37-attempt count against your monitoring, deploy history and application logs.
- Have someone who was not on call read the report for clarity and accuracy.
- Confirm with your own contract or lawyer before sending anything that could commit you to credits, or that could be read as accepting fault.
- Remove anything you would not want the practice manager to forward. The report does not name the engineer who ran the deploy, so keep it that way.