Energy and Utilities · Enterprise · $540K · owned by Yusuf Abdel-Rahman
Steve has not written one for this account. Everything it would read is below.
Every record Steve is allowed to cite
Director of Grid Data Systems
Principal Data Engineering Lead
Energy and Utilities · Enterprise · $540K · owned by Yusuf Abdel-Rahman
Steve has not written one for this account. Everything it would read is below.
Every record Steve is allowed to cite
Director of Grid Data Systems
Principal Data Engineering Lead
VP Operations Technology
Next step: Joint load test to 1.2M events per second on 2026-07-29
No next step set
Sustained SCADA ingest above roughly 640k events per second triggered consumer lag and backpressure into their collectors, with 90 minutes of delayed telemetry before mitigation. Rebalanced partitions from 96 to 240 and the cluster now holds a sustained 680k events per second. First response missed the P1 target by four hours.
The 90 day substation rollup returns at p95 1.4 seconds and p99 4.6 seconds against Granite Peak's internal 2 second target. Reproduced on their cluster. Candidate fixes under review are a pre-aggregated daily rollup and a change to the partition pruning strategy. Still open.
Rob asked for the replay harness and a documented sizing model so his team can run the load test on their own infrastructure. Shipped the harness, the cluster sizing worksheet and the recommended broker and writer counts for the 1.2M events per second target.
Agreed a joint load test on 2026-07-29 replaying a 90 minute winter-peak window of captured SCADA traffic at two times speed, with pass criteria of 1.2M events per second sustained for 30 minutes at under 5 seconds end to end lag. Craig confirmed the Tier 4 purchase decision follows the load test result and that budget is already provisioned in the operations technology plan. Rob asked that the open rollup latency work stay on a separate track so it does not gate the test.
Rob presented the measurements from the June 15 incident showing a sustained ceiling around 680k events per second on the current cluster shape before consumer lag builds. Granite Peak's forecast for the winter peak is 1.1M events per second across 34 substations, with a short duration burst target of 1.2M. Nadia said the expansion case rests on demonstrating that headroom rather than on price. She referred to an internal capacity study her team completed in May and said its conclusions are what the operations technology committee will weigh, but that she cannot circulate it outside Granite Peak. Agreed we would produce a sizing model for the 1.2M target.
VP Operations Technology
Next step: Joint load test to 1.2M events per second on 2026-07-29
No next step set
Sustained SCADA ingest above roughly 640k events per second triggered consumer lag and backpressure into their collectors, with 90 minutes of delayed telemetry before mitigation. Rebalanced partitions from 96 to 240 and the cluster now holds a sustained 680k events per second. First response missed the P1 target by four hours.
The 90 day substation rollup returns at p95 1.4 seconds and p99 4.6 seconds against Granite Peak's internal 2 second target. Reproduced on their cluster. Candidate fixes under review are a pre-aggregated daily rollup and a change to the partition pruning strategy. Still open.
Rob asked for the replay harness and a documented sizing model so his team can run the load test on their own infrastructure. Shipped the harness, the cluster sizing worksheet and the recommended broker and writer counts for the 1.2M events per second target.
Agreed a joint load test on 2026-07-29 replaying a 90 minute winter-peak window of captured SCADA traffic at two times speed, with pass criteria of 1.2M events per second sustained for 30 minutes at under 5 seconds end to end lag. Craig confirmed the Tier 4 purchase decision follows the load test result and that budget is already provisioned in the operations technology plan. Rob asked that the open rollup latency work stay on a separate track so it does not gate the test.
Rob presented the measurements from the June 15 incident showing a sustained ceiling around 680k events per second on the current cluster shape before consumer lag builds. Granite Peak's forecast for the winter peak is 1.1M events per second across 34 substations, with a short duration burst target of 1.2M. Nadia said the expansion case rests on demonstrating that headroom rather than on price. She referred to an internal capacity study her team completed in May and said its conclusions are what the operations technology committee will weigh, but that she cannot circulate it outside Granite Peak. Agreed we would produce a sizing model for the 1.2M target.