[{"data":1,"prerenderedAt":846},["ShallowReactive",2],{"\u002Fblog\u002Fwebsite-outage-response-runbook":3},{"id":4,"title":5,"author":6,"body":8,"category":835,"date":836,"description":837,"extension":838,"faq":839,"howTo":839,"image":839,"lastUpdated":836,"meta":840,"navigation":87,"path":841,"readingTime":842,"seo":843,"stem":844,"__hash__":845},"blog\u002Fblog\u002Fwebsite-outage-response-runbook.md","Website Outage Response Runbook: What to Do in the First 60 Minutes",{"name":7},"Theo Cummings",{"type":9,"value":10,"toc":818},"minimark",[11,15,18,21,26,29,52,55,58,62,67,72,75,116,125,127,131,134,190,193,195,199,222,232,235,237,241,244,268,271,273,277,280,283,288,316,321,336,341,362,376,381,402,407,428,490,495,516,518,522,525,528,554,557,572,575,577,581,584,605,608,610,614,617,632,663,687,689,693,696,704,707,715,723,725,729,732,738,741,743,747,780,784,814],[12,13,14],"p",{},"When your website goes down, the first 10 minutes are chaotic. People flood Slack. Someone starts investigating. Someone else starts a different investigation. Nobody has told customers anything. Nobody has posted to the status page. The CEO is DMing the on-call engineer.",[12,16,17],{},"A runbook stops the chaos before it starts. It replaces \"what do we do?\" with a sequence your team follows every time, regardless of who is on call.",[12,19,20],{},"This is a copy-ready runbook template for the first 60 minutes of a production outage.",[22,23,25],"h2",{"id":24},"before-you-need-this-prerequisites","Before You Need This: Prerequisites",[12,27,28],{},"This runbook assumes three things are in place:",[30,31,32,40,46],"ol",{},[33,34,35,39],"li",{},[36,37,38],"strong",{},"Uptime monitoring with alerting"," - you know about the outage from your monitoring tool, not from a customer tweet",[33,41,42,45],{},[36,43,44],{},"A status page"," - a public URL where customers check service status",[33,47,48,51],{},[36,49,50],{},"Defined severity levels"," - at minimum, a distinction between \"all users affected\" and \"some users affected\"",[12,53,54],{},"If any of these are missing, set them up before the next incident. Monitoring takes under 5 minutes to set up in Vantaj. Not having it is the single most expensive preparation gap.",[56,57],"hr",{},[22,59,61],{"id":60},"the-runbook","The Runbook",[63,64,66],"h3",{"id":65},"t0-alert-fires","T+0: Alert Fires",[12,68,69],{},[36,70,71],{},"The alert lands in Slack (or via SMS, email, or phone call).",[12,73,74],{},"Do these three things in the first 2 minutes:",[76,77,80,90,110],"ul",{"className":78},[79],"contains-task-list",[33,81,84,89],{"className":82},[83],"task-list-item",[85,86],"input",{"disabled":87,"type":88},true,"checkbox"," Open the monitoring dashboard. Note: which service, which regions, what error",[33,91,93,95,96,100,101,109],{"className":92},[83],[85,94],{"disabled":87,"type":88}," Claim the incident: post in ",[97,98,99],"code",{},"#incidents",": ",[36,102,103,104,108],{},"\"I'm on this. SEV-",[105,106,107],"span",{},"1\u002F2\u002F3",".\""," This single message prevents the situation where three people each assume someone else is handling it",[33,111,113,115],{"className":112},[83],[85,114],{"disabled":87,"type":88}," Check if it is real: go directly to the affected URL. Confirm the error from your browser",[12,117,118,119,124],{},"If you cannot reproduce the error manually, check whether your monitoring uses multi-region consensus. If it does and it still fired, the outage is real. If it does not, you may have a ",[120,121,123],"a",{"href":122},"\u002Fblog\u002Freduce-false-positive-alerts","false positive",".",[56,126],{},[63,128,130],{"id":129},"t2-severity-classification","T+2: Severity Classification",[12,132,133],{},"Classify the incident. This determines the next steps.",[135,136,137,153],"table",{},[138,139,140],"thead",{},[141,142,143,147,150],"tr",{},[144,145,146],"th",{},"Severity",[144,148,149],{},"Criteria",[144,151,152],{},"Response",[154,155,156,168,179],"tbody",{},[141,157,158,162,165],{},[159,160,161],"td",{},"SEV-1",[159,163,164],{},"Full outage, all users affected, or data loss risk",[159,166,167],{},"Wake everyone. Status page immediately.",[141,169,170,173,176],{},[159,171,172],{},"SEV-2",[159,174,175],{},"Partial outage or degraded service, significant user impact",[159,177,178],{},"On-call investigates. Status page within 5 min.",[141,180,181,184,187],{},[159,182,183],{},"SEV-3",[159,185,186],{},"Minor issue, small subset of users affected",[159,188,189],{},"On-call investigates. Internal tracking only.",[12,191,192],{},"When in doubt, classify up. A SEV-1 that turns out to be a SEV-2 is fine. A SEV-2 that was actually a SEV-1 costs you 20 minutes of delayed communication with customers.",[56,194],{},[63,196,198],{"id":197},"t3-open-the-incident-channel","T+3: Open the Incident Channel",[76,200,202,216],{"className":201},[79],[33,203,205,207,208,211,212,215],{"className":204},[83],[85,206],{"disabled":87,"type":88}," Create a Slack channel: ",[97,209,210],{},"#inc-YYYYMMDD-[short-description]"," (e.g., ",[97,213,214],{},"#inc-20260628-api-down",")",[33,217,219,221],{"className":218},[83],[85,220],{"disabled":87,"type":88}," Post the incident brief in the channel:",[223,224,229],"pre",{"className":225,"code":227,"language":228},[226],"language-text","🔴 INCIDENT OPEN\n\nService: [which service or endpoint]\nImpact: [what users experience]\nSeverity: SEV-[1\u002F2\u002F3]\nIncident Commander: @you\nStarted: [time] UTC\nMonitoring link: [direct link to the failing monitor]\nStatus page: [link]\n\nAll incident discussion here only.\n","text",[97,230,227],{"__ignoreMap":231},"",[12,233,234],{},"This channel becomes the incident timeline. Everything that happens, every hypothesis tested, every change made - post it here as it happens. You will need this log for the postmortem.",[56,236],{},[63,238,240],{"id":239},"t5-update-the-status-page","T+5: Update the Status Page",[12,242,243],{},"For SEV-1 and SEV-2, update the status page before you know the cause. Post this:",[245,246,247,255,262],"blockquote",{},[12,248,249],{},[36,250,251,252],{},"Investigating - ",[105,253,254],{},"Service Name",[12,256,257,258,261],{},"We are investigating reports of ",[105,259,260],{},"service"," being unavailable. Engineers are actively working on this.",[12,263,264,265],{},"Next update: ",[105,266,267],{},"T+20 from now",[12,269,270],{},"Customers who see this update stop filing support tickets. Support ticket volume during an acknowledged incident drops by 60-80% compared to a silent outage. You get your investigation time back.",[56,272],{},[63,274,276],{"id":275},"t5-to-t30-diagnosis","T+5 to T+30: Diagnosis",[12,278,279],{},"Your goal in this window: identify the category of the problem. You do not need the root cause yet. You need enough to either restore service or escalate.",[12,281,282],{},"Work through this checklist in order. Skip steps you can verify quickly.",[12,284,285],{},[36,286,287],{},"Check 1: Recent changes",[76,289,291,300,306],{"className":290},[79],[33,292,294,296,297],{"className":293},[83],[85,295],{"disabled":87,"type":88}," Was there a deployment in the last 30 minutes? ",[97,298,299],{},"git log --oneline -10",[33,301,303,305],{"className":302},[83],[85,304],{"disabled":87,"type":88}," Was there a config change, environment variable update, or infrastructure change?",[33,307,309,311,312,315],{"className":308},[83],[85,310],{"disabled":87,"type":88}," If yes to either: ",[36,313,314],{},"rollback first, investigate second",". Restoring service takes priority over understanding the cause.",[12,317,318],{},[36,319,320],{},"Check 2: External dependencies",[76,322,324,330],{"className":323},[79],[33,325,327,329],{"className":326},[83],[85,328],{"disabled":87,"type":88}," Check the status pages of your key dependencies: Stripe, Auth0, AWS, Cloudflare, Vercel, etc.",[33,331,333,335],{"className":332},[83],[85,334],{"disabled":87,"type":88}," If a dependency is down and you use it in the affected flow: that is your cause. Update the status page with the dependency name.",[12,337,338],{},[36,339,340],{},"Check 3: Server health",[76,342,344,350,356],{"className":343},[79],[33,345,347,349],{"className":346},[83],[85,348],{"disabled":87,"type":88}," CPU: is it pegged?",[33,351,353,355],{"className":352},[83],[85,354],{"disabled":87,"type":88}," Memory: is it near capacity?",[33,357,359,361],{"className":358},[83],[85,360],{"disabled":87,"type":88}," Disk: is it full?",[12,363,364,365,368,369,368,372,375],{},"For each: ",[97,366,367],{},"top",", ",[97,370,371],{},"free -h",[97,373,374],{},"df -h",". On cloud providers, check your dashboard for resource graphs with the spike visible.",[12,377,378],{},[36,379,380],{},"Check 4: Application errors",[76,382,384,390,396],{"className":383},[79],[33,385,387,389],{"className":386},[83],[85,388],{"disabled":87,"type":88}," Check application logs for the time the outage started",[33,391,393,395],{"className":392},[83],[85,394],{"disabled":87,"type":88}," Look for: stack traces, connection errors, timeout messages, OOM kills",[33,397,399,401],{"className":398},[83],[85,400],{"disabled":87,"type":88}," Check your error tracker (Sentry, Datadog, etc.) for new error types that appeared at the outage start time",[12,403,404],{},[36,405,406],{},"Check 5: Database",[76,408,410,416,422],{"className":409},[79],[33,411,413,415],{"className":412},[83],[85,414],{"disabled":87,"type":88}," Can the application connect to the database?",[33,417,419,421],{"className":418},[83],[85,420],{"disabled":87,"type":88}," Are there long-running queries blocking normal operations?",[33,423,425,427],{"className":424},[83],[85,426],{"disabled":87,"type":88}," Is the connection pool exhausted?",[223,429,433],{"className":430,"code":431,"language":432,"meta":231,"style":231},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# PostgreSQL connection count\npsql -c \"SELECT count(*) FROM pg_stat_activity;\"\n\n# Long-running queries\npsql -c \"SELECT pid, now() - pg_stat_activity.query_start AS duration, query FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC LIMIT 10;\"\n","bash",[97,434,435,443,464,470,476],{"__ignoreMap":231},[105,436,439],{"class":437,"line":438},"line",1,[105,440,442],{"class":441},"sHwdD","# PostgreSQL connection count\n",[105,444,446,450,454,458,461],{"class":437,"line":445},2,[105,447,449],{"class":448},"sBMFI","psql",[105,451,453],{"class":452},"sfazB"," -c",[105,455,457],{"class":456},"sMK4o"," \"",[105,459,460],{"class":452},"SELECT count(*) FROM pg_stat_activity;",[105,462,463],{"class":456},"\"\n",[105,465,467],{"class":437,"line":466},3,[105,468,469],{"emptyLinePlaceholder":87},"\n",[105,471,473],{"class":437,"line":472},4,[105,474,475],{"class":441},"# Long-running queries\n",[105,477,479,481,483,485,488],{"class":437,"line":478},5,[105,480,449],{"class":448},[105,482,453],{"class":452},[105,484,457],{"class":456},[105,486,487],{"class":452},"SELECT pid, now() - pg_stat_activity.query_start AS duration, query FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC LIMIT 10;",[105,489,463],{"class":456},[12,491,492],{},[36,493,494],{},"Check 6: DNS and SSL",[76,496,498,507],{"className":497},[79],[33,499,501,503,504],{"className":500},[83],[85,502],{"disabled":87,"type":88}," Does the domain resolve? ",[97,505,506],{},"dig yourdomain.com",[33,508,510,512,513],{"className":509},[83],[85,511],{"disabled":87,"type":88}," Is the SSL certificate valid? ",[97,514,515],{},"echo | openssl s_client -connect yourdomain.com:443 2>\u002Fdev\u002Fnull | openssl x509 -noout -dates",[56,517],{},[63,519,521],{"id":520},"t20-first-status-update-if-unresolved","T+20: First Status Update (if unresolved)",[12,523,524],{},"Post an update to the status page even if you have not resolved the issue. Keep your stated update commitment.",[12,526,527],{},"If you have found the cause:",[245,529,530,537,543,550],{},[12,531,532],{},[36,533,534,535],{},"Issue Identified - ",[105,536,254],{},[12,538,539,540,124],{},"We have identified the cause: ",[105,541,542],{},"one plain-language sentence about what went wrong",[12,544,545,546,549],{},"We are working on a fix. ",[105,547,548],{},"Specific features or flows"," remain affected.",[12,551,264,552],{},[105,553,267],{},[12,555,556],{},"If you have not found the cause:",[245,558,559,565,568],{},[12,560,561],{},[36,562,251,563],{},[105,564,254],{},[12,566,567],{},"We continue to investigate. Engineers are actively working to identify and resolve the issue.",[12,569,264,570],{},[105,571,267],{},[12,573,574],{},"Post the same update to the incident channel.",[56,576],{},[63,578,580],{"id":579},"t30-escalation-decision-point","T+30: Escalation Decision Point",[12,582,583],{},"If the incident is not resolved or clearly on the path to resolution within 30 minutes, escalate.",[76,585,587,593,599],{"className":586},[79],[33,588,590,592],{"className":589},[83],[85,591],{"disabled":87,"type":88}," Is a second engineer needed? Page them.",[33,594,596,598],{"className":595},[83],[85,597],{"disabled":87,"type":88}," Does the CEO or a customer-facing team member need to know? Brief them in 2 sentences in a separate channel. Do not add them to the incident channel unless they can help resolve the issue.",[33,600,602,604],{"className":601},[83],[85,603],{"disabled":87,"type":88}," Does a vendor need to be contacted? Open a support ticket with them now, not after the incident.",[12,606,607],{},"Escalation is not failure. Sitting on an unresolved SEV-1 for 45 minutes without escalating because you feel like you should be able to solve it alone is failure.",[56,609],{},[63,611,613],{"id":612},"resolution-service-restored","Resolution: Service Restored",[12,615,616],{},"When monitoring confirms recovery:",[76,618,620,626],{"className":619},[79],[33,621,623,625],{"className":622},[83],[85,624],{"disabled":87,"type":88}," Wait 5 minutes after the monitoring reports green before declaring resolved. Premature resolution declarations followed by a second failure are worse than staying in \"monitoring\" state.",[33,627,629,631],{"className":628},[83],[85,630],{"disabled":87,"type":88}," Update the status page to Resolved:",[245,633,634,641,652],{},[12,635,636],{},[36,637,638,639],{},"Resolved - ",[105,640,254],{},[12,642,643,644,647,648,651],{},"This incident is resolved. ",[105,645,646],{},"Service"," is fully operational as of ",[105,649,650],{},"time"," UTC.",[12,653,654,655,658,659,662],{},"Duration: ",[105,656,657],{},"X hours Y minutes",". Cause: ",[105,660,661],{},"one honest sentence",". We will publish a post-incident review within 48 hours.",[76,664,666,675,681],{"className":665},[79],[33,667,669,671,672,674],{"className":668},[83],[85,670],{"disabled":87,"type":88}," Post to ",[97,673,99],{}," with resolution time and who worked it",[33,676,678,680],{"className":677},[83],[85,679],{"disabled":87,"type":88}," Post to the incident channel with the resolution summary and a note that the channel will be archived",[33,682,684,686],{"className":683},[83],[85,685],{"disabled":87,"type":88}," Schedule the postmortem within 48 hours",[56,688],{},[63,690,692],{"id":691},"t2-hours-customer-email-decision","T+2 Hours: Customer Email Decision",[12,694,695],{},"Send a customer email for:",[76,697,698,701],{},[33,699,700],{},"Any SEV-1 incident",[33,702,703],{},"Any SEV-2 incident lasting over 30 minutes",[12,705,706],{},"Do not send a customer email for:",[76,708,709,712],{},[33,710,711],{},"Outages under 10 minutes with no confirmed user impact",[33,713,714],{},"SEV-3 incidents affecting a small subset of users",[12,716,717,718,722],{},"See ",[120,719,721],{"href":720},"\u002Fblog\u002Fincident-communication-templates","Incident Communication Templates"," for email copy.",[56,724],{},[22,726,728],{"id":727},"incident-timeline-log-template","Incident Timeline Log Template",[12,730,731],{},"Copy this into the incident channel at the start of every significant incident:",[223,733,736],{"className":734,"code":735,"language":228},[226],"INCIDENT TIMELINE\n-----------------\n[time] UTC: Alert fired. [Monitor name] confirmed down from [regions].\n[time] UTC: Incident opened. IC: @name. Severity: SEV-[level].\n[time] UTC: Status page updated: Investigating.\n[time] UTC: [Action taken \u002F hypothesis tested \u002F finding]\n[time] UTC: [Action taken \u002F hypothesis tested \u002F finding]\n[time] UTC: Root cause identified: [description]\n[time] UTC: Fix deployed: [description]\n[time] UTC: Monitoring: watching for recovery.\n[time] UTC: Resolved. Recovery confirmed.\n",[97,737,735],{"__ignoreMap":231},[12,739,740],{},"Filling this in as the incident progresses takes 10 seconds per entry. Having it saves 2-3 hours of postmortem reconstruction.",[56,742],{},[22,744,746],{"id":745},"after-the-incident-48-hour-checklist","After the Incident: 48-Hour Checklist",[76,748,750,756,762,768,774],{"className":749},[79],[33,751,753,755],{"className":752},[83],[85,754],{"disabled":87,"type":88}," Postmortem written while details are fresh",[33,757,759,761],{"className":758},[83],[85,760],{"disabled":87,"type":88}," Action items assigned with owners and due dates",[33,763,765,767],{"className":764},[83],[85,766],{"disabled":87,"type":88}," Customer email sent (if applicable)",[33,769,771,773],{"className":770},[83],[85,772],{"disabled":87,"type":88}," Monitoring alert thresholds reviewed - could detection have been faster?",[33,775,777,779],{"className":776},[83],[85,778],{"disabled":87,"type":88}," Runbook updated - did anything in this runbook not fit the actual incident?",[22,781,783],{"id":782},"related-guides","Related Guides",[76,785,786,792,798,802,808],{},[33,787,788],{},[120,789,791],{"href":790},"\u002Fblog\u002Fincident-response-checklist-startups","Incident Response Checklist for Startups",[33,793,794],{},[120,795,797],{"href":796},"\u002Fblog\u002Fhow-to-communicate-during-service-outage","How to Communicate During a Service Outage",[33,799,800],{},[120,801,721],{"href":720},[33,803,804],{},[120,805,807],{"href":806},"\u002Fblog\u002Fincident-postmortem-template","How to Write an Incident Postmortem",[33,809,810],{},[120,811,813],{"href":812},"\u002Fblog\u002Finstant-website-downtime-alerts","How to Get Instant Alerts When Your Website Goes Down",[815,816,817],"style",{},"html pre.shiki code .sHwdD, html code.shiki .sHwdD{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic}html pre.shiki code .sBMFI, html code.shiki .sBMFI{--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B}html pre.shiki code .sfazB, html code.shiki .sfazB{--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D}html pre.shiki code .sMK4o, html code.shiki .sMK4o{--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":231,"searchDepth":445,"depth":445,"links":819},[820,821,832,833,834],{"id":24,"depth":445,"text":25},{"id":60,"depth":445,"text":61,"children":822},[823,824,825,826,827,828,829,830,831],{"id":65,"depth":466,"text":66},{"id":129,"depth":466,"text":130},{"id":197,"depth":466,"text":198},{"id":239,"depth":466,"text":240},{"id":275,"depth":466,"text":276},{"id":520,"depth":466,"text":521},{"id":579,"depth":466,"text":580},{"id":612,"depth":466,"text":613},{"id":691,"depth":466,"text":692},{"id":727,"depth":445,"text":728},{"id":745,"depth":445,"text":746},{"id":782,"depth":445,"text":783},"tutorials","2026-06-17","A copy-ready incident response runbook for when your website goes down. Covers the first 60 minutes minute by minute: acknowledgment, diagnosis, communication, recovery, and handoff.","md",null,{},"\u002Fblog\u002Fwebsite-outage-response-runbook",11,{"title":5,"description":837},"blog\u002Fwebsite-outage-response-runbook","SWY9_8CHLhzb7Xvrh4nC-YO4jTSgblzjruLuAJbMszM",1783446134538]