Risks — Mirakl / PIM / Offers Data Lineage

Things we do know, that are structurally fragile, silent-failure-prone, or otherwise worth proactive attention — as opposed to open-questions.md (things we don't know), data-lineage.md (the confirmed lineage itself), or opportunities.md (dormant data sources not currently used by supplysync.com — NeuCo, Rheem, Hydros). These are risk statements grounded in confirmed code behavior, not speculation.

1. Missing per-BU data silently produces gaps, not failures, across multiple domains

Confirmed in: §1 (attributes), §2 (PID↔ERP matching), §3 (branch inventory).

Every gap-fill/join mechanism traced in this pipeline degrades silently on a missing match:

• §1: a missing per-BU attribute row leaves the base PIM export's (possibly blank) description standing — no error, just a short/blank description on that product.
• §2: merge_bu_erp()'s missing='' leftjoin means an unmatched product gets an empty erp_id, not a dropped row or a raised error.
• §3: a BU's empty/absent/failed availability converter output means that BU's branches simply don't appear downstream — no row to join, no failure signal.

Risk: there is no single dashboard/alert surfacing "BU X's feed had 0 rows today" or "N products got an empty erp_id this run" across these domains. A silent per-BU feed failure could persist for a long time before anyone notices a symptom (missing descriptions, missing sellers, missing stock) — and even then, tracing the symptom back to "BU X's feed failed on date Y" requires knowing to look in three different places.

2. A missing per-BU ID match silently produces a Mirakl offer with no seller to attach to — confirmed, not just suspected

Confirmed in: §2.

mirakl_seller_wso_process() — which used to filter seller records to a diff_list (products in the current PIM run) and pim=True (must exist in the merged all-products feed) — no longer exists post-WWIES-5783. The new mirakl_seller_process() does its own narrower W1-presence check, but doesn't replicate the diff-list filtering, and merge_bu_erp()'s missing='' leftjoin means an unmatched product gets an empty erp_id that still flows into the generated seller CSV.

Per the team, the actual consequence is now confirmed: a missing/empty erp_id means that BU's seller-channel entity doesn't get matched to the product in Mirakl, so there's no seller record for that BU to attach an offer to — this doesn't require any API-level rejection, it's a structural gap. Cross-BU failure mode: if none of a product's BUs match, the product has no seller record anywhere in Mirakl, and PriceFX's offer for that PID has nothing to attach to at all, for any BU.

Risk: this is a real, confirmed failure mode with (as far as this analysis found) no visible failure signal — the offer simply has nowhere to land, silently. Worth a monitoring/alerting gap-check: is anything watching for "PriceFX has an active offer for PID X, but PID X has zero matched seller records in Mirakl"? If not, this can persist undetected.

3. No single BU→converter format table exists elsewhere — this doc's §3 table is the only consolidated reference, and it has a gap

Confirmed in: §3.

The HM (Homans) BU converter (split_homans) has a confirmed function and BU code but an unconfirmed exact call site — meaning even this consolidated reference isn't fully verified end-to-end for every row. (NeuCo/Rheem/Hydros's equivalent gaps are tracked in opportunities.md instead, since none of those three are part of this pipeline.)

Risk: low urgency on its own, but compounds risk #1 — if there's no other place this per-BU mapping is documented, and this doc's own coverage has gaps, diagnosing a BU-specific availability issue may require re-deriving the DAG wiring from scratch rather than looking it up.

4. airflow-pim-dags has ~150 generated DAGs; this analysis read a deliberately narrow slice

Confirmed in: data-lineage.md's "Sources" section.

This entire lineage analysis is scoped to the jobs-inf-w1-* DAGs plus a specific list of normalization/xmlbuilder files — a small fraction of the repo's total DAG count (per-BU/locale variants for Shopware and Algolia were explicitly out of scope, per the existing architecture docs' own "Scope" section).

Risk: this doc can create false confidence if treated as exhaustive. Claims here are accurate for the files read; domains/behaviors implemented only in one of the ~140+ unread DAG files won't be caught by this analysis, and a future reader might not notice the scope boundary if they're not looking for it. Re-emphasizing here since it's easy to skim past a caveat buried in a "Sources" section.

5. Both repos' commit pins in this doc are snapshots, not live-tracked

Confirmed in: data-lineage.md's "Sources" section; demonstrated directly in this session (101 commits landed in airflow-pim-dags between the initial analysis and the first re-verification pass, materially changing §2's mechanism).

Risk: this doc will silently go stale as both repos continue to change. There's no automation re-checking the pinned commits against HEAD and flagging drift — it depends on someone remembering to re-run a verification pass, the way this session did manually. Worth considering a lightweight recurring check (e.g. a scheduled job that diffs HEAD against the pinned commit for the specific files this doc cites, and flags if any changed) rather than relying on manual re-verification.

6. The offers/pricing pipeline processes batches strictly sequentially with no parallelism — a real scaling ceiling regardless of how Open Question #6 resolves

Confirmed in: watsco1-etl-offers/dags/etl_lib/pipeline.py's process_phase() — batches are iterated with a plain for loop, one at a time, each with up to 10 retry attempts and exponential backoff on failure. There is no batching parallelism (no concurrent futures, no Airflow dynamic task mapping, no multi-worker fan-out) anywhere in this code path.

What this means concretely: total phase runtime is a direct linear function of batch count × average per-batch latency (PriceFX poll-until-ready + Mirakl submit-and-poll-until-done, both involve real network round-trips and PriceFX's own processing time). batch_size is capped at 100,000 records and defaults to 10,000 (config.py) — so batch count is total_records / batch_size, and runtime is batch_count × per_batch_latency.

A viable fix exists, but is gated on an open question. Per the team: PriceFX batches are already seller-scoped ("PriceFX creates batches per seller automatically", "Submit one store per import" — confirmed in Open Question #6), so parallelizing this loop by seller wouldn't require any data-model redesign, only a code change to run independent per-seller batch sequences concurrently. Whether this is actually safe to do depends entirely on Open Question #7 (whether Mirakl's documented rate limit — "max once per minute" — is enforced per-store or account-wide). If per-store, parallelizing is close to a free win. If account-wide, parallelizing wouldn't help and could increase 429 volume.

Risk: even if the pricing-record growth question (Open Question #6) resolves favorably (linear with SKU count, insensitive to customer count), this pipeline has no architectural headroom beyond "run for longer" until Open Question #7 is answered — there's no way to safely add throughput by adding workers/parallelism without first confirming the rate limit's scope. Current runtime (~25 min for the longest observed phase, against roughly a ~50-minute full-DAG budget out of the ~1 hour ceiling the user cited) leaves real but finite headroom. If the growth question resolves unfavorably (multiplicative with customer-org count), this ceiling will be hit by a runtime problem before it's hit by any Mirakl/PriceFX rate limit, because there's currently no parallel path in code to fall back on. Worth flagging to whoever owns capacity planning: get Open Question #7 answered proactively, since it's the fact that determines whether "add per-seller parallelism" is this risk's mitigation or a false lead.