Load Shedding Aware Scraping: Schedule Apify Runs Around SA Outages
Your overnight Apify cron still assumes 24/7 uptime. During Stage 4–6, scrapers fire when target sites or local ops are dark, burn compute credits on empty runs, and leave pricing and competitor datasets with Monday holes.
We design SA outage-aware scheduling so resilient data collection survives load shedding without wasted Apify credits.
Sound Familiar?
These are the exact issues SA ops and data leads brought to us before load-shedding-aware Apify schedule design:
- Overnight Apify cron jobs fire into Stage 4–6 windows when target sites or local ops are dark, then finish empty or timed out
- Pricing and competitor datasets open on Monday with holes nobody noticed until the dashboard looked wrong
- Apify still bills compute units for doomed runs, retries, and timeouts even when no usable rows land
- Schedules assume 24/7 uptime and never check Eskom stage or ESP area windows before starting
- Ops spends the morning re-running missed windows by hand with no stage-based pause, shift, or catch-up playbook
Recovery is not the same as immunity. CSIR reported load shedding energy down about 76% in 2024 versus 2023, yet early 2025 still saw Stage 2–6 returns. Residual schedules that still assume 24/7 uptime are what burn Apify credits the next time stages spike.
What Load Shedding Aware Scraping Actually Does
Stage check → schedule gate → Apify run or shift → catch-up if the window was dark.
Outage Feed Checked
National stage and ESP area windows are read before the scheduled Apify start
Schedule Gate
Run proceeds, pauses, or shifts to the next powered window based on your rules
Actor Collects
Scrapes only fire when targets and verification paths are expected to be live
Catch-Up If Needed
Missed overnight slots re-queue so Monday pricing and competitor packs stay complete
Everything You Need for SA Outage-Aware Schedule Design
Stage-Aware Schedule Gates
Apify schedules pause or shift when national stage or ESP area windows collide with your scrape slot, so Actors do not fire into known outages.
EskomSePush / ESP Integration
Live stage and suburb schedules from public load shedding APIs (including EskomSePush) feed the gate before each run, not a static calendar someone forgets to update.
Window Shift & Soft Windows
When Stage 4–6 hits overnight, collection slides to the next powered window instead of burning credits on a doomed start.
Catch-Up Runs
Missed overnight windows re-queue automatically once power and target availability return, so Monday pricing packs stay complete.
Credit Guardrails
Timeouts, max-items caps, and outage pre-checks stop empty retries from quietly eating Apify compute units when scrapers cannot reach live sites.
Ops Visibility
Stage events, skipped runs, and catch-ups stay visible so data leads trust the calendar without living in Apify Console or ESP.
What We Wire Into Your Apify Calendar
From Monday Data Gaps to Complete Overnight Packs
How a Johannesburg retail pricing team stopped burning Apify credits on Stage collisions and closed holes in competitor price datasets.
Blind Overnight Cron
- Three Apify Actors ran every night at 01:00 SAST with no stage check
- During Stage 4–6 weeks, roughly one in three runs timed out or returned empty after retries
- Apify still billed compute for doomed starts and retry loops
- Monday pricing packs routinely missed whole competitor slices
- Ops spent 4–6 hours a week re-running windows and explaining gaps to category leads
Load-Shedding-Aware Calendar
- EskomSePush stage and ESP area gates sit in front of every overnight schedule
- Colliding windows shift or pause; catch-up fires once power returns
- Doomed overnight scrapes dropped by about 80% within the first month
- Monday competitor packs arrived complete without manual re-runs
- Ops only gets pulled when a real Actor or site break needs a human
Before vs After Outage-Aware Scheduling
How It Works
From first conversation to live outage-aware schedules in 2–4 weeks.
Tell Us Your Setup
Which Actors run overnight, which SA areas matter for targets or ops, and where Monday data gaps currently appear.
Free Scoping Call
30-minute call to map stage risk, ESP areas, shift rules, and which windows must never be skipped without a catch-up.
Build & Test
We wire load-shedding-aware Apify schedule design against live stage feeds, then simulate Stage 4–6 collisions on your real Actors.
Go Live & Monitor
Switch off blind 24/7 cron. Stage gates, shifts, and catch-ups keep resilient data collection without burning credits on dark windows.
Frequently Asked Questions
What is load shedding aware scraping on Apify?
It means your Apify schedules check South African outage intelligence (national stage and ESP area windows) before an Actor starts. When Stage 4–6 would put target sites or your local verification path offline, the run pauses, shifts, or queues a catch-up instead of wasting compute on empty or timed-out scrapes.
How do you use EskomSePush or other load shedding APIs?
We pull national status and suburb schedules from public load shedding APIs such as EskomSePush (ESP), then gate Apify Schedules against those windows. Ops gets stage-aware pause/resume without someone refreshing ESP by hand every evening.
Why redesign schedules if load shedding has eased since 2023?
Eskom recorded load shedding on 280 days in 2023, and CSIR measured roughly a 76% drop in load shedding energy in 2024 as the fleet recovered. Residual Stage risk remains: early 2025 still saw Stage 2–6 events. Blind overnight cron that assumes 24/7 uptime is what leaves Monday holes when stages return.
Does Apify still charge when a scrape fails during an outage?
Yes. Apify bills compute time for runs that fail, time out, or return empty after retries. On Scale and Business plans that is roughly R2.10–R2.60 per compute unit at current Rand rates (about $0.13–$0.16/CU). Outage-aware gates stop those doomed starts before they spend credits.
Will this replace our existing Apify monitoring and alerts?
No. Monitoring catches silent empty datasets after a run. Load-shedding-aware scheduling prevents the run from starting into a known dark window in the first place. Most SA teams want both: stage gates up front, then status and field alerts if something still goes wrong.
How much does load-shedding-aware Apify schedule design cost?
Straightforward stage-gate and shift setups for a small Actor set start from around R15,000. Multi-Actor calendars with ESP area mapping, catch-up queues, and credit guardrails typically range from R25,000 to R60,000. Teams burning overnight credits on Stage collisions and Monday backfills usually recover the project cost within 2–3 months from compute waste and ops time alone.
Stop Running Scrapers Into the Dark
If overnight Apify jobs still ignore Stage risk, you are paying for failed runs and Monday data gaps that load-shedding-aware scheduling already solves.
Tell us which Actors run overnight, which SA areas matter for your targets, and where datasets currently hole out. We will show you exactly how stage gates, shift windows, and catch-up runs would work for your calendar.