Black Friday System Readiness: A Technical Guide

Short answer: Preparation must be finished before November begins. Run load tests above the expected peak; measure p95 and p99 latency, error rate, database connection count and cloud quotas; test the restore, not the backup. Restrict changes throughout the month, apply a full change freeze around 11.11 and Black Friday–Cyber Monday, and plan the on-call rotation to cover four weeks. Base the go decision not on a single "we're ready" statement but on seven evidence gates: demand, capacity, journey, resilience, protection, operations, change. The smallest verified limit is the campaign's real limit.

Don't Deploy on Friday. Verify on Friday.

The carousel above and the downloadable PDF are in Turkish; this page is the full English version of the guide.

Why is campaign traffic different?

In 1961, the Philadelphia police called the Friday after Thanksgiving "Black Friday" because the city ground to a halt. Today the same crowd brings checkout pages to a halt. Back then the crowd fit into a single Friday; today e-commerce discounts run throughout November, and 11.11, Black Friday and Cyber Monday are the busiest days of the period. From a technical standpoint, this is not a single peak; it is a baseline that runs high for four weeks, with several peaks stacked on top of it.

A high-traffic period is not a bigger version of a normal day. Traffic rises all at once, user behavior changes, third-party limits become visible and small delays turn into waves of retries. The shape of the load is the same at every peak: requests pile up on the same few flows (product page, cart, checkout), and bot traffic adds to the pile-up. The application layer scales horizontally; the database, session store, inventory lock and payment integration do not scale at the same rate. Look for the bottleneck in these layers first.

That is why preparation doesn't end with allocating more CPU or servers. The team has to prove how critical customer journeys behave at the expected load and under anticipated failures.

A campaign that spans the whole month changes preparation in three ways:

• A change freeze doesn't fit into one week. You need restricted change throughout the month and a full freeze on peak days.
• Capacity stays high for weeks. Temporary quota increases and autoscaling minimums add cost for the whole month.
• The team stays on-call for four weeks. A one-night plan isn't enough for the second peak.

In 2026, these days fall on Wednesday, November 11 (11.11), Friday, November 27 (Black Friday) and Monday, November 30 (Cyber Monday); Thanksgiving is the fourth Thursday of November, which is November 26. The calendar section is built on these dates; shift it to fit your own campaign calendar.

The definition of ready

A system doesn't count as ready just because it carries the forecast peak traffic under normal conditions. A ready system keeps critical journeys within the target SLO, degrades in a controlled way in the most important failure scenarios, protects itself during overload and recovers within a time frame the team knows. Each of these claims is verified with a test result, dashboard, runbook or drill record.

Peak reliability = verified capacity × fault tolerance × operational capability

When any one factor approaches zero, overall reliability drops with it; the smallest verified limit is the campaign's real limit. That is why this guide combines capacity planning, performance testing, resilience, graceful degradation, change management and incident command practices into a single system of work. It draws on the scale test and Resiliency Matrix approach from Shopify's BFCM preparation program, on Google SRE's principles for overload, load shedding, retries and incident response, and on the AWS Operational Readiness Review and Azure mission-critical testing practices; the sources are listed in the last section.

Core principles

How to use this guide

1. The tech lead names the event owner and the critical customer journeys in scope.
2. Each service owner fills in their own capacity limit, dependencies, failure modes and evidence.
3. A central readiness review assesses the risks against the seven evidence gates.
4. Red gates must be closed before a go decision is made. Amber gates require a written risk acceptance, trigger thresholds and a fallback plan.
5. After the campaign, the forecast is compared with actuals; the checklist is updated with incident findings.

Each checklist item has four parts. What to do: what should be done and with which tool. How you notice: where the problem first shows up when the check is skipped. Why it matters: the technical rationale behind the check, and its source. Command: commands given with example values; change the thresholds to fit your own SLO.

Preparation calendar · November 2026

Groundwork first, then two freezes, with a rehearsal in between.

Countdown to peak

The countdown below is relative to the first peak (11.11). For Black Friday, the same cycle runs again in shortened form with the November 13–17 mid-campaign review; the first peak's findings are the second peak's test scenario.

The seven evidence gates and go / no-go

The go or no-go decision rests not on a general impression but on seven independent areas of evidence. Each gate has a single owner, a date for its latest evidence and its open risks. A green gate does not mean the document exists; it means the content is current and tested.

Status rules

An average readiness score hides a critical red. Completing 95 of 100 checks looks impressive; if the five missing checks concern payment database capacity, WAF fail-open behavior or rollback authority, the system is not ready. Gates don't compensate for one another: the executive summary can be color-coded, while the technical decision is made gate by gate.

Go / no-go review

The purpose of the review meeting is not to listen to presentations but to surface missing evidence and unaccepted risk. In three minutes, each gate owner presents the target, the result, the open risk and the decision they are asking for; the discussion stays on the evidence. Before going into the meeting, the following must be ready:

• Demand model and marketing campaign calendar are up to date (evidence: forecast version · owner: event owner)
• Full-scale and failover under load results are accessible (test run ID · platform)
• Critical journey SLO dashboards are live (dashboard URL · journey owner)
• Open P0/P1 technical risks are listed with owner and date (risk register · tech lead)
• Vendor quota and escalation confirmations are complete (vendor record · service owner)
• On-call roster, break-glass accounts and war room have been rehearsed (drill log · ops lead)
• Freeze and emergency change authority have been announced (change plan · change owner)

The decision record contains these fields: decision (go / conditional go / no-go), decision time, Event Commander, red gates, accepted amber risks, conditions and deadline, next review.

No-go criteria

• A critical journey has no peak test result, or its SLO is not met.
• Verified capacity remaining after the loss of a single failure domain is below the forecast peak.
• Idempotency and reconciliation are unclear in payment, order or similar write flows.
• Overload behavior produces uncontrolled retries, unbounded queues or process crashes.
• A critical dependency has no owner, alert or working runbook.
• There are unverified gaps in incident response roles or production access.
• An irreversible high-risk migration is scheduled during the event window.

Demand model and capacity envelope

A capacity forecast is not a single RPS number. The same total traffic produces a completely different infrastructure load when the cache hit ratio, share of authenticated users, product mix, payload size or checkout conversion rate changes. That is why demand is modeled as a multidimensional workload vector for each critical journey. The safe limit you can use during the campaign is not the highest lab result; it is the verified capacity that remains after subtracting the planned failure scenario and the safety margin.

Demand vector

Core calculations

P = baseline × growth × campaign effect × uncertainty factor

In the peak forecast P, the factors must not be derived from the same data; otherwise the same growth is counted twice. Best-case, expected and stress scenarios are kept separate.

H = verified capacity / forecast peak

Headroom H is calculated not only for normal conditions but also after removing the largest failure domain the plan accounts for losing.

L = λ × W

According to Little's law, the average number of concurrent jobs in the system, L, is the product of the arrival rate λ and the time spent in the system, W. If latency doubles, concurrency at the same RPS roughly doubles too; that is why connection pools and thread pools fill up as a function of latency.

ΔQ = arrival rate − processing rate

Even if a positive difference in queue growth ΔQ looks short-lived, it quickly drives up queue age. Dashboards should show oldest message age, not just queue depth.

Capacity envelope worksheet

Two profiles are drawn up for each component: normal limit and limit under failure. Headroom is calculated separately against each.

Red flags for the capacity plan

• The quota that autoscaling needs has not been allocated, or scale-out time is longer than the peak ramp time.
• The combined database connection pool and streaming connection total has not been verified for an increased number of application instances.
• The failover region looks idle, but it can't take traffic in time because of image pull, warm-up, cache fill or data replication.
• The test is weighted toward anonymous browse; authenticated checkout, webhook or admin flows are not represented in their real proportions.
• The cloud or CDN provider has not been notified in advance of large quota and capacity increases.
• The only result is average latency; p99, error rate, saturation and recovery time are not reported.

Critical customer journeys and SLOs

Services are divided according to how the infrastructure is organized; the customer experience doesn't follow those boundaries. That is why the basic unit of preparation is the critical journey, not the service: view product → add to cart → payment → order confirmation. Each journey is tracked end to end, from the edge to the data layer and external services.

Journey reliability sheet

Fill in one row for each critical journey; any field left blank is an open risk.

Choosing SLIs

Synthetic transaction design

• A shallow probe only checks that the endpoint is reachable; a deep probe verifies the actual business outcome. The two are tied to separate alerts.
• Set aside test users, products and payment data so they won't run out during peak or collide with real customers.
• Synthetic transactions run from different regions and ISP vantage points. A service that looks healthy from the inside can be unreachable for an external user.
• Canary results don't get lost in the overall error rate; they have their own metric and alert.
• Every critical journey shows up on the dashboard as a single row with success, latency, volume and saturation.

Checklist · 01–03 · Load testing, limits and change policy

01 · Generate peak load yourself before the campaign

Run a scenario that follows the real user path (home page, search, product, cart, checkout) and push it past the expected peak request rate. k6, JMeter, Locust or Gatling will do the job. Instead of averages, look at p95 and p99 latency, error rate, database connection count and queue depth. Also run the test through the CDN and WAF, because that is where traffic will come from on campaign day. A successful test does more than reach the target RPS; it reveals the system's limit, the signals that show the limit is approaching, the behavior the customer sees once the limit is exceeded and the recovery time after load drops. Every test starts with a hypothesis and ends with a reproducible result.

Realistic traffic generation. Use an open workload model: new requests don't wait for the previous response to finish, so load doesn't fall away as latency rises; that is also how real internet traffic behaves. Weight user flows by their production ratios; cover rare but expensive flows separately in a stress scenario. Run warm-cache and cold-cache cases separately; use cache busting only as a deliberate scenario. Test data should cover hot keys, campaign products, high fan-out and real payload sizes. Verify that load generator capacity is above the target; don't mistake generator saturation for a system limit. Don't make calls that would cause real damage to third-party systems; use mocks, sandboxes or pre-approved limits.

// k6 · example values: adjust the target rate and thresholds to your own peak traffic and SLO.
import http from 'k6/http';

export const options = {
  scenarios: {
    campaign: {
      executor: 'ramping-arrival-rate', // open workload model
      startRate: 50,
      timeUnit: '1s',
      preAllocatedVUs: 200,
      maxVUs: 2000,
      stages: [
        { target: 600, duration: '10m' }, // ramp to peak rate
        { target: 600, duration: '20m' }, // hold at peak
      ],
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<800'], // milliseconds
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  // In a real scenario, the home page → search → product → cart → checkout steps are called in order.
  http.get('https://staging.example.com/product/123');
}

How you notice: While the dashboards look normal, customer service starts getting "the checkout page keeps spinning" complaints.

Why it matters: The average hides the latency experienced by requests waiting in the queue; p95 and p99 show it. The thresholds in the example say this: at most one in every 20 requests may exceed 0.8 seconds, and at most one in every 100 requests may fail. A test that passes with a closed model on a warm cache may not represent the real peak.

Test acceptance criteria. A test counts as passed only if the following six pieces of evidence exist:

• Critical journey SLOs were met throughout the test (run report and dashboard snapshot · journey owner)
• Normal and failure scenarios were reported separately (two capacity profiles · platform)
• The first saturation point and the leading indicator were identified (graph and threshold · service owner)
• Retry and timeout behavior was measured together with downstream load (attempt histogram · app owner)
• After the test, the backlog drained within the target time (queue age trend · data owner)
• Findings were fixed and the same scenario was run again (comparison run ID · event owner)

The test report includes the following graphs on the same time axis: offered load, accepted load and completed business transactions; journey success rate and p50/p95/p99 latency; CPU plus concurrency, connections, queue age, database wait, cache hits and third-party errors; autoscaler desired / ready / serving instance counts and warm-up time; timestamps for failure injection, operator action, recovery start and full recovery.

02 · Gather quotas and limits in one table

List the vCPU, IP address, load balancer and NAT gateway quotas in your cloud account side by side. Next to them, add the database's max_connections value, the application's connection pool size and the firewall's limits for concurrent sessions and new connections per second (CPS). Verify that the autoscaling upper bound fits within these limits. The "normal limit" column in the capacity envelope table is filled in from this list.

# AWS: EC2 quotas
aws service-quotas list-service-quotas --service-code ec2 --output table

# Azure: vCPU usage and limits in the region
az vm list-usage --location westeurope --output table

-- PostgreSQL: current connection count and upper limit
SELECT count(*) FROM pg_stat_activity;
SHOW max_connections;

How you notice: Autoscaling tries to launch new servers but can't; a one-line quota error sits in the console. Or the servers come up, but the database starts returning "too many connections" errors.

Why it matters: Autoscaling can only grow as far as your quota allows, and every new application instance brings its own connection pool. Think of it like tables in a restaurant: every instance arrives with its own reservation, but the restaurant's number of tables (max_connections) doesn't change; once the tables are full, the database turns away new connections. Multiply the connection pool size by the instance count and tie the total to the database's connection budget. A quota increase may require provider approval; open the request before November and update your budget alerts to match the increased quotas.

03 · Restrict changes all month, freeze them during peaks

Throughout November, only tested, reversible and low-risk changes should go to production, and only outside busy hours. Apply a full change freeze around the peak days (the two freeze periods on the calendar). Put the scope in writing: application releases, Infrastructure as Code (Terraform, etc.) changes, database schema changes, WAF and CDN rules. A freeze isn't about making the system untouchable; the goal is to reduce the number of changes, make exceptions visible and preserve rollback capacity.

Emergency change record. For every exception during a freeze, fill in these fields: trigger (which metric and threshold call for the change), hypothesis (by what mechanism it will reduce customer impact), blast radius (which region, tenant, endpoint or percentage of traffic), validation (which metrics will be watched in the first 5, 10 and 20 minutes), rollback (single step, owner, duration and revert threshold), approver (IC and technical owner; security owner if there is a security impact), record (timestamp, commit or config and outcome).

Canary and rollback rules. Judge a canary not only by error rate but also by critical journey success, tail latency and saturation; if the canary doesn't represent the real traffic mix, don't generalize the result. Rollback gets tested as thoroughly as a new deploy; if a schema or data migration can't be reversed, have a forward-fix plan and a protection mode. Config changes are also versioned, peer reviewed and audited like code. During peak, don't test more than one hypothesis at a time; otherwise you can't tell which change affected the outcome.

How you notice: The outage starts after a "small" configuration change made the day before the campaign, and the first question in the war room is "What changed last?"

Why it matters: According to Google's SRE book, roughly 70% of outages are due to changes in a live system. Freezing the whole month, however, creates a different risk: changes pile up and ship in December all at once, as one bigger release. Combining restricted change with peak freezes balances these two risks. That is also the rule of our Reliability Friday series: Don't Deploy on Friday. Verify on Friday.

Checklist · 04–06 · Restore, synthetic monitoring and on-call

04 · Measure the restore, not the backup

Restore critical databases from the most recent backup into a test environment. Measure two durations separately: the time it takes to restore the data, and the time until the application can take orders again. If you use point-in-time recovery (PITR), also try recovering to a specific time, and compare the results with your RPO and RTO targets. The evidence is not that a backup exists; it is the restore time in the latest test record and verified data integrity.

How you notice: The backup dashboard has been green for months. The restore is first attempted during a real failure and finishes hours later than planned.

Why it matters: Missing WAL or binlog files, inaccessible encryption keys and version mismatches only surface during a restore. Every minute lost on a campaign day costs more orders than it would on an ordinary day.

05 · Monitor the customer's path with synthetic checks

Set up synthetic checks that exercise the home page, search, add-to-cart and checkout steps externally, from more than one region, once a minute. Tie alerts to the SLO, not to CPU and memory thresholds: for example, error rate and p95 latency on the checkout step over a five-minute window. The five rules from the previous section apply to synthetic transaction design.

# A simple endpoint check: HTTP status code and total time
curl -s -o /dev/null -w '%{http_code} %{time_total}s\n' https://www.example.com/cart

On the observability side, close out these six checks before the campaign:

• The dashboard shows RED and USE metrics on the same time axis as the critical journeys.
• The telemetry pipeline works at peak load; log volume and the cardinality limit do not leave you blind during an incident.
• Alerts are tied to an SLO threat and to an action; noisy alerts have been cleaned up ahead of the campaign.
• Third-party dependencies, quotas, certificates, replication and queue age have leading indicator alerts.
• Deploys, config changes, failovers and marketing pushes show up on the graphs as event markers.
• For the war room, a single-screen executive view is kept separate from the deep technical dashboards.

How you notice: Orders drop while the infrastructure dashboards are green. The first to notice the problem is not you but a customer complaining on social media.

Why it matters: Infrastructure metrics show the cause; synthetic checks and SLO alerts show the outcome the customer experiences.

06 · Plan on-call for four weeks and runbooks around campaign hours

For the campaign month, prepare the on-call rotation, the escalation matrix and short runbooks for critical scenarios. If the payment provider slows down, database connections run out or the CDN returns errors, what are the first three steps? Write down who owns the decisions to roll back, stop the campaign and notify customers. When the same person tries to fix the system, coordinate and keep stakeholders informed all at once, response quality drops; roles are assigned before an incident starts and practiced during the dress rehearsal.

This division of roles is an example; in your own organization, roles may be combined or named differently.

Incident declaration triggers

War room operating rules. There is one decision channel, one incident record and a fixed conference bridge. Every action is written down with an owner and a timestamp; no hypothesis is forgotten before its result comes in. Technical teams work in threads or separate workstreams, and decisions are summarized in the main channel. The update cadence is set in advance; even when there is no new information, the status and the time of the next check are posted. The IC does not dive into fixing the system; they keep control of coordination, priorities and communication. A recovery declaration does not rest on a single metric: the journey SLO, backlog, saturation and error trend must return to normal together.

Event dashboard layout

How you notice: An alert fires at 2:00 a.m. Whoever answers the phone doesn't know who to contact at the payment provider or who has the authority to stop the campaign.

Why it matters: Every piece of information you have to look up during an incident adds to response time. The rotation and runbooks let even a tired team take the same steps.

Checklist · 07–10 · External services, bots, dates and kill switches

07 · Put timeouts, retry budgets and circuit breakers on external services

List your payment, 3D Secure, SMS verification (OTP), shipping, inventory and email integrations. For every call, define an explicit timeout and limited retries with exponential backoff and jitter. Then add a circuit breaker and a fallback behavior: queueing, an alternative provider or a clear message to the user. Use idempotency keys for payment calls. During overload, the goal is not to accept every request but to complete the most valuable customer transactions while keeping the system from crashing; these controls must work consistently from upstream to downstream, and autoscaling, load balancing and load shedding must not produce conflicting signals.

Timeout budget. A customer request's total deadline cannot be protected by giving every call in the chain its own, longer timeout. Each hop must know the remaining budget; work that can no longer benefit the user must not start, or must be canceled. Work that keeps running on the server side after the client has given up adds to the invisible load. The breakdown below is an example; the values are derived from your journey SLO.

Retry budget. Retries do not create capacity; they only rely on the chance that transient, local errors will succeed on another attempt. When several layers retry independently, a single user operation multiplies: if each of three layers makes four attempts, including the first, that can add up to 4³, or 64, attempts by the time it reaches the database. That is why retries happen in a single controlled layer, with exponential backoff and jitter, within an explicit budget.

How you notice: The call center starts hearing the same sentence: "My card was charged, but my order isn't showing up."

Why it matters: A call with no timeout, or with a timeout left too long, ties up threads and connections waiting on a slow external service; the queue grows and healthy requests slow down too. Unlimited retries multiply the load on a service that is already slowing down and turn the slowdown into an outage. Retrying a payment call without an idempotency key creates a risk of double charging. A circuit breaker works like a fuse in your home: the fault stays in one room, and the whole house doesn't go dark.

08 · Separate bot traffic from the checkout flow

Configure bot management and rate limiting rules in your WAF per endpoint: separate limits for login, cart, inventory lookups and payment. To catch credential stuffing and card testing attacks, monitor the failed-attempt rate on login and payment endpoints. Verify that your DDoS protection also covers the application layer (L7). Rate limits are applied at the global, tenant and endpoint levels according to business priority; the rules that tell bot behavior apart from normal customer bursts are tested for false positive risk before the campaign. For every security control, the fail-open or fail-closed decision must be written down and approved; the latency and fail mode of fraud checks must be tested together with the checkout journey. Privileged access, break-glass accounts, MFA and audit logs, along with an emergency rollback and a communication owner for security rule changes, are also part of this item.

How you notice: The campaign product shows as "sold out" before real customers can reach the cart.

Why it matters: According to Imperva's 2026 Bad Bot Report, more than half of web traffic (53%) comes from bots. Every bot request that is not filtered out uses the same connection pool, the same inventory lookups and the same payment limit as real customers.

09 · Check certificate, domain, DNS and disk dates

Before the campaign starts, renew the TLS certificates and domains that expire during November; automate certificate renewal with ACME. Also verify that secret, license and token expiry dates fall outside the event window. If you plan to reroute traffic during the campaign, lower your DNS TTL values in advance. Because log and metric volume will stay high for weeks, review disk usage alerts, log rotation and retention periods.

# Certificate expiry date
echo | openssl s_client -servername www.example.com -connect www.example.com:443 2>/dev/null | openssl x509 -noout -enddate

# TTL value of the DNS record (second column)
dig +noall +answer www.example.com

How you notice: One morning, the mobile app and API clients hit connection errors at the same time, and the browser shows a red security warning. Or a disk fills up with logs, the service can no longer write, and it stops.

Why it matters: Under a CA/Browser Forum decision, browser-trusted TLS certificates issued since March 15, 2026, are valid for at most 200 days; every certificate must be renewed at least twice a year. On March 15, 2027, the maximum will drop to 100 days, and on March 15, 2029, to 47 days, which will mean renewing roughly eight times a year.

10 · Prepare kill switches and a virtual waiting room

Put the components that can be turned off under load (recommendation engine, live inventory counters, personalization, heavy reports) behind feature flags so they can be switched off in a single step. Set a priority order that keeps the checkout flow at the top. Define a virtual waiting room threshold that kicks in as you approach the capacity limit. Feature flags and kill switches must not depend on a single owner; they must have audit and rollback steps.

Load shedding priority

Graceful degradation catalog

How you notice: When load rises, the checkout page slows down too, because it shares the same resources as the heavy components and there is no switch to turn them off.

Why it matters: Graceful degradation is the difference between a full outage and partial service. If the recommendation engine shuts down, the customer may not notice; if checkout shuts down, they will. A virtual waiting room is like taking a number at the bank: nobody is turned away, but not everyone gets in at once.

Dependency and failure model

A dependency map falls short if it only shows which services call each other. For every connection, write down the failure semantics: what happens on a timeout, who owns the retry, whether stale data is acceptable, whether the service starts when a startup dependency is missing and how traffic recovers when the dependency comes back.

Resiliency Matrix

The Resiliency Matrix, which Shopify uses for high-traffic events, shows in a single table which journeys a customer can complete, and at what quality, in each failure scenario. Filling in the table is a tabletop exercise in its own right; scanning the rows and columns shows where the system is most fragile. The cells below are examples; fill them in according to your own dependencies.

Failure mode record

Behind every cell in the matrix sits a record that answers eight questions.

Failure classes to test

The card for each Game Day scenario carries these fields: hypothesis, starting load, injected failure, expected customer behavior, expected alerts and automation, abort criteria, observed behavior, recovery time, findings and owner.

Technical layer checks

The checks below are the minimum that service owners must verify in their own areas. Each check is recorded with an evidence link, an owner and the date it was last verified. Observability and security checks are covered in items 05 and 08, so they are not repeated here.

DNS, CDN, WAF and edge

• DNS TTL, secondary DNS and registrar access have been verified; emergency change authority has been tested.
• CDN cache key, TTL, stale-if-error, origin shield and request coalescing behavior has been measured.
• Origin capacity has been tested with a cold cache and after a cache purge.
• WAF rule evaluation, bot management, the DDoS profile and rate limit capacity have been tuned for peak.
• TLS handshake, certificate expiry, OCSP and connection reuse metrics are on the dashboard.

Network and load balancer

• New connection rate, active connections, SYN backlog, ephemeral port and NAT table capacity have been measured.
• The health check measures whether the application can actually take traffic; a node is not counted as healthy until warm-up is complete.
• Connection draining and keepalive timeouts are consistent with deploy, scale and failover behavior.
• The effect of a zone or region loss on routing convergence and session stickiness has been tested.
• The load balancer does not treat load shedding errors as successes; the feedback loop has been checked.
• In addition to bandwidth, packet rate (PPS), drops, retransmits and load balancer data plane limits are monitored.

Compute, Kubernetes and runtime

• Request and limit values match the actual profile; CPU throttling and memory OOM behavior were measured in the peak test.
• The HPA signal uses a leading indicator; where queue age or concurrency is needed ahead of CPU, it has been put in place.
• The cluster autoscaler has been tested together with quota, IP, subnet, image pull and scheduling constraints.
• Startup, readiness and liveness probes do not cause restart loops under overload.
• JVM GC, heap, thread pool; .NET thread starvation; runtime connection pool metrics are visible.
• Pod disruption budget and topology spread protect the remaining capacity when a failure domain is lost.

Application and API

• Deadline, bounded retry, idempotency and cancellation behavior is defined for every downstream call.
• N+1 query, expensive serialization, large payload and synchronous fan-out patterns have been eliminated or limited.
• Circuit breaker thresholds and half-open behavior have been tested; no thundering herd forms during recovery.
• Concurrency limits and queue bounds are explicit; unbounded in-flight requests are not accepted.
• Business errors do not get lost in the technical error rate; checkout rejections are kept separate from system failures.

Database and data platform

• Peak TPS, connection, lock wait, buffer cache, IOPS, log flush and replication lag limits have been measured.
• Skew has been tested for hot partitions, hot keys and campaign products.
• Stale read and read-your-writes decisions under read replica lag are documented per journey.
• Write fencing, transaction retry and duplicate write behavior during failover have been tested.

Cache, queue and streaming

• In addition to cache hit ratio, origin amplification, eviction, memory fragmentation and hot keys are monitored.
• Jittered TTL, request coalescing or single flight has been applied against cache stampedes.
• In addition to queue depth, oldest message age, arrival rate, completion rate and dead letters are measured.
• Consumer scaling is aligned with partition count, ordering and downstream capacity.
• Idempotent consumer behavior has been tested for duplicate delivery and out-of-order messages.
• Backlog drain time has been calculated to stay within the post-campaign recovery SLO.

Third party and vendor

• Quota, burst, concurrent connection and daily limits have been confirmed in writing.
• The vendor has been informed of the campaign dates, expected volume and escalation contact.
• Timeout, 429, partial response and schema deviation scenarios have been simulated.
• A data consistency and reconciliation plan exists for the fallback provider or manual process.
• The vendor status page is not the only signal; independent synthetic measurement is in place.
• The support contract, severity definitions and escalation times have been written into the event command plan.

Peak day runbook

This section offers a generic skeleton; add the commands, dashboards, owners and escalation links for each environment based on your organization's own systems. Run the same skeleton separately for 11.11, Black Friday and Cyber Monday.

Live decision thresholds

The thresholds are examples; fill in the blank values from your own SLO.

Safety conditions for scale-down

• Offered load has been trending down for the defined period.
• Queue and streaming backlogs are draining within the target recovery time.
• Cache hit ratio and database load have returned to normal.
• Late-arriving webhook, batch and reconciliation traffic has been accounted for.
• There is enough time before the next geographic peak or campaign wave begins.
• The scale-down preserves the minimum failover capacity per failure domain.

Mid-campaign review and after the campaign

Use the first peak as a rehearsal for the second. In the days after 11.11, run a short mid-campaign review and close out the findings before Black Friday. Schedule the scale-down of the resources you grew (autoscaling minimums, temporary servers, upsized database classes) for when November ends; otherwise they will show up on the December bill.

Hold a review even when the campaign goes smoothly. It answers which assumptions in the preparation program proved correct, which capacity went unused, which alerts were unnecessary and which manual interventions could be automated in the future. These notes become the load test scenarios for the next campaign.

Remove, standardize or productize

Every temporary mechanism built during campaign preparation is assigned to one of three groups: remove, standardize or productize. Capabilities such as the Resiliency Matrix, synthetic journeys, the capacity model and incident drills are not kept for Black Friday alone; they are added to the standard for normal operations. That way, preparation for the next peak does not start from scratch.

Action quality criteria

• The action reduces a specific failure mechanism; it is not a generic statement of improvement.
• It has an owner and a date; its success metric is measurable.
• Where possible, it is turned into a guardrail or an automated check.
• It includes a horizontal sweep that looks for the same class of incident in other services.
• Once complete, it is re-verified with the same test or Game Day scenario.

This guide is a living standard. After any high-impact incident, failed test or campaign review, a new item can be added to the checklist; the new item must rest on a specific failure mechanism, clear evidence and a workable owner model. Ownership, tool links and risk thresholds are reviewed at least once a year.

Anti-pattern catalog

The statements below come up often in readiness meetings, and each one masks a gap in the evidence.

Checklist table · print and fill in

Ten items and two post-campaign tasks; an owner and a date on every row. Fill in the table before November begins. During freeze periods, only the "Status" column is updated; no new work is added.

Event readiness master record

Fill in this record on a single page at the start of the campaign: campaign and dates, Event Commander, tech lead, critical journeys, forecast peak and burst, freeze start, go / no-go date, war room, vendor escalation. A summary of the seven gates sits on the same page: owner, evidence, status (R / A / G) and open risk for each gate.

Runbook verification record

• Another team member was able to execute the runbook (drill record)
• All commands and links are up to date (date checked)
• The required production permissions have been verified (access test)
• The rollback step has been tried in a production-like environment (test run)
• The recovery condition is clear and measurable (dashboard)
• Escalation contacts and times are up to date (roster)

Eclit note: Find your system's limit before your customers do.

We also run some of these checks as an ongoing service: 24/7 monitoring, performance and capacity analytics, backup and recovery and application security (WAF). In the previous episode of the series, we looked at Terraform drift and what "No changes" does not prove.

Every Friday. One Production Check.

Sources

The checklists and decision gates are not verbatim copies of the sources; they are common reliability principles applied to the campaign period. All of the pages below were opened and checked on October 30, 2026.

• Shopify Engineering, How we prepare Shopify for BFCM (2025), Kyle Petroski and Matthew Frail, November 20, 2025: scale test program, Game Day, preparation focused on critical journeys
• Shopify Engineering, Performance Testing at Scale: for BFCM and Beyond, Jordan Neufeld, January 27, 2023: full-scale test, testing behind the CDN and with cache busting
• Shopify Engineering, Resiliency Planning for High-Traffic Events, Ryan McIlmoyl, December 10, 2020: Resiliency Matrix
• Google SRE Book, Introduction: roughly 70% of outages stem from a change made to a live system (item 03)
• Google SRE Book, Handling Overload: retry budget, client-side throttling
• Google SRE Book, Addressing Cascading Failures: load shedding, graceful degradation, retry amplification
• Google SRE Workbook, Managing Load: overload control
• Google SRE Workbook, Incident Response: incident command roles, war room setup
• AWS Well-Architected, Operational Readiness Reviews: the readiness review and evidence gate approach
• Microsoft Azure Well-Architected, Deployment and testing for mission-critical workloads: failover under load, chaos and freeze practices
• Imperva, 2026 Bad Bot Report announcement, May 5, 2026: more than 53% of web traffic comes from bots (item 08)
• DigiCert, CA/Browser Forum SC-081v3 certificate validity schedule: 200 days from March 15, 2026; 100 days from March 15, 2027; 47 days from March 15, 2029 (item 09)