Wholesale Distribution · Enterprise · $480K · owned by Yusuf Abdel-Rahman

Wholesale Distribution · Enterprise · $480K · owned by Yusuf Abdel-Rahman

What happened

1. IssueLIN-DEM-1072 hour ingest soak test aborted at 42 hours, sustained backpressure

   updated 2026-07-21

   The streaming tier soak test was aborted at hour 42 when consumer lag passed 900 seconds and did not recover. The ingest API call rate flattened near 9.4M per week while event throughput held near 40k per second. Marcus Rho's team asked whether the ceiling is a per tenant shard limit or their own broker configuration. Still open, no root cause issued.

   TodoHighOpen in Linear
2. IssueLIN-DEM-11Pen test finding: customer-managed keys not enforced on staged objects

   updated 2026-07-21

   Northwind's third party pen test found that objects in the ingestion staging bucket are encrypted with platform-managed keys rather than the customer's KMS key. Priya Nandakumar's team logged it as a medium finding requiring remediation or a documented compensating control before production cutover. Key management design doc supplied on 2026-07-09, customer verification pending.

   TodoMediumOpen in Linear
3. IssueLIN-DEM-12PrivateLink endpoint required, public ingest endpoint not permitted in prod

   updated 2026-07-21

   Northwind's network standard forbids egress to public service endpoints from the production VPC. They need a PrivateLink endpoint in eu-central-1 plus a documented DNS pattern for their split-horizon resolver. Currently the pilot runs over an allow-listed NAT gateway that their platform team has confirmed will not be extended into production.

   TodoMediumOpen in Linear
4. IssueLIN-DEM-8CDC connector drops rows during Postgres primary failover

   updated 2026-07-21

   During Northwind's own failover drill the logical replication slot was not preserved on promotion of the standby, and the connector resumed from a stale LSN. Ines Duarte reconciled 41,900 missing rows across three fact tables. Phase 2 exit criterion 3 requires zero row loss across a controlled failover, so this run is recorded as a fail. Engineering has a fix targeted but no build date has been given to the customer.

   In ReviewHighOpen in Linear
5. IssueLIN-DEM-9p99 query latency above target on the wide-fact workload

   updated 2026-07-21

   Northwind measured p95 at 410ms and p99 at 1,240ms on their 92 column order-line fact table at 40k events per second. Resolved as configuration rather than defect: partition key and clustering guidance applied, which brought p99 down to 640ms on the customer's re-run. The phase 2 exit criterion is p99 under 500ms at 40k events per second, so the criterion remains unmet.

   DoneMediumOpen in Linear
6. CallGONG-934Platform team check-in

   2026-07-14

   Marcus said his team has started a parallel technical evaluation and that an internal build on their existing Kafka and Flink stack is on the table as a serious option. He was direct that the soak test abort and the unresolved failover behaviour are what put the build option back on the table, and he will not agree a re-run date until there is a root cause for the backpressure. Dana Whitfield did not attend and has not responded to three meeting invitations since 2026-06-26.

7. UsageUSG-2207pilot ingest API volume climbed from 4.6M to 9.4M calls per week and then flattened within one percent for four consecutive weeks

   2026-07-14
8. UsageUSG-2214weekly active seats fell from 221 to 146 against a contracted 250, down about 34 percent

   2026-07-14
9. CallGONG-917Security architecture review

   2026-07-02

   Priya took us through three items: customer-managed key enforcement on staged objects, data residency confirmation that no pilot data leaves eu-central-1, and SOC2 CC6.1 evidence for our own key rotation. She was constructive and said none of the three are unusual for a platform at this stage. She asked for a key management design document and a written residency attestation, and confirmed her team would verify against the pen test finding once received.

10. CallGONG-902Phase 2 pilot exit criteria review

    2026-06-16

    Walked the six phase 2 exit criteria one by one. Four pass: schema evolution, backfill of 14 months of history, RBAC mapping to their Okta groups, and dbt model parity against their existing warehouse. Two fail: zero row loss across a controlled Postgres failover, and p99 under 500ms at 40k events per second on the wide-fact workload. Dana said she will not take the expansion to Hal Brenner until both are green, and asked for a tuned configuration and a soak test plan before the end of the month. She also set the outside date: the streaming tier has to be in production before their November peak trading window, because the current nightly pipeline will not carry that volume and they will not run a cutover during peak.

What happened

1. IssueLIN-DEM-1072 hour ingest soak test aborted at 42 hours, sustained backpressure

   updated 2026-07-21

   The streaming tier soak test was aborted at hour 42 when consumer lag passed 900 seconds and did not recover. The ingest API call rate flattened near 9.4M per week while event throughput held near 40k per second. Marcus Rho's team asked whether the ceiling is a per tenant shard limit or their own broker configuration. Still open, no root cause issued.

   TodoHighOpen in Linear
2. IssueLIN-DEM-11Pen test finding: customer-managed keys not enforced on staged objects

   updated 2026-07-21

   Northwind's third party pen test found that objects in the ingestion staging bucket are encrypted with platform-managed keys rather than the customer's KMS key. Priya Nandakumar's team logged it as a medium finding requiring remediation or a documented compensating control before production cutover. Key management design doc supplied on 2026-07-09, customer verification pending.

   TodoMediumOpen in Linear
3. IssueLIN-DEM-12PrivateLink endpoint required, public ingest endpoint not permitted in prod

   updated 2026-07-21

   Northwind's network standard forbids egress to public service endpoints from the production VPC. They need a PrivateLink endpoint in eu-central-1 plus a documented DNS pattern for their split-horizon resolver. Currently the pilot runs over an allow-listed NAT gateway that their platform team has confirmed will not be extended into production.

   TodoMediumOpen in Linear
4. IssueLIN-DEM-8CDC connector drops rows during Postgres primary failover

   updated 2026-07-21

   During Northwind's own failover drill the logical replication slot was not preserved on promotion of the standby, and the connector resumed from a stale LSN. Ines Duarte reconciled 41,900 missing rows across three fact tables. Phase 2 exit criterion 3 requires zero row loss across a controlled failover, so this run is recorded as a fail. Engineering has a fix targeted but no build date has been given to the customer.

   In ReviewHighOpen in Linear
5. IssueLIN-DEM-9p99 query latency above target on the wide-fact workload

   updated 2026-07-21

   Northwind measured p95 at 410ms and p99 at 1,240ms on their 92 column order-line fact table at 40k events per second. Resolved as configuration rather than defect: partition key and clustering guidance applied, which brought p99 down to 640ms on the customer's re-run. The phase 2 exit criterion is p99 under 500ms at 40k events per second, so the criterion remains unmet.

   DoneMediumOpen in Linear
6. CallGONG-934Platform team check-in

   2026-07-14

   Marcus said his team has started a parallel technical evaluation and that an internal build on their existing Kafka and Flink stack is on the table as a serious option. He was direct that the soak test abort and the unresolved failover behaviour are what put the build option back on the table, and he will not agree a re-run date until there is a root cause for the backpressure. Dana Whitfield did not attend and has not responded to three meeting invitations since 2026-06-26.

7. UsageUSG-2207pilot ingest API volume climbed from 4.6M to 9.4M calls per week and then flattened within one percent for four consecutive weeks

   2026-07-14
8. UsageUSG-2214weekly active seats fell from 221 to 146 against a contracted 250, down about 34 percent

   2026-07-14
9. CallGONG-917Security architecture review

   2026-07-02

   Priya took us through three items: customer-managed key enforcement on staged objects, data residency confirmation that no pilot data leaves eu-central-1, and SOC2 CC6.1 evidence for our own key rotation. She was constructive and said none of the three are unusual for a platform at this stage. She asked for a key management design document and a written residency attestation, and confirmed her team would verify against the pen test finding once received.

10. CallGONG-902Phase 2 pilot exit criteria review

    2026-06-16

    Walked the six phase 2 exit criteria one by one. Four pass: schema evolution, backfill of 14 months of history, RBAC mapping to their Okta groups, and dbt model parity against their existing warehouse. Two fail: zero row loss across a controlled Postgres failover, and p99 under 500ms at 40k events per second on the wide-fact workload. Dana said she will not take the expansion to Hal Brenner until both are green, and asked for a tuned configuration and a soak test plan before the end of the month. She also set the outside date: the streaming tier has to be in production before their November peak trading window, because the current nightly pipeline will not carry that volume and they will not run a cutover during peak.