Open Questions — Mirakl / PIM / Offers Data Lineage

Things we genuinely don't know and need someone on the team to answer. Companion to data-lineage.md (what the data lineage actually is), risks.md (things we do know that are fragile), and opportunities.md (dormant data sources not currently used by supplysync.com — NeuCo, Rheem, Hydros). If you can answer one of these, update data-lineage.md's relevant section and remove it from this list — don't just delete the question without updating the source doc.

1. Does BI's historical-sales feed for CE originate from CE's own ERP, or is BI itself the point of origin?

Where this came from: §6. Confirmed: BI delivers historical sales via a nightly S3 feed that filterSales.process_ce reads — that ingestion mechanism is settled. What's open is one layer further back: is BI aggregating/relaying a feed that itself comes from CE's ERP, or is BI the actual system of record for this data?

Why it matters: matters for the same reason as any proxy-vs-source question in this doc — if BI is a pass-through, staleness/errors in BI's own upstream ingestion could silently affect what this pipeline treats as historical sales, and the true root cause would be at CE's ERP rather than BI.

2. Is "PriceFX pushes a priced-items list into PIM via S3" implemented anywhere?

Where this came from: §4 / originally README.md's "Key relationships" section. This claim predates the lineage work in this session — no pricefx_pim_priced_items, priced_items, or equivalent string/path was found anywhere in airflow-pim-dags or watsco1-etl-offers when searched.

What we need to know: is this implemented in a system outside both repos (e.g. directly in Informatica Product360, or a script not in either codebase), or is this stale/aspirational documentation that never got built?

Why it matters: if this hand-off doesn't actually exist, the "Key relationships" narrative in README.md/index.md needs a correction, not just a caveat.

3. What's the actual code path connecting PriceFX's product-to-sell selection to which BU seller channels get a Mirakl record?

Where this came from: §5. Confirmed: PriceFX's product-to-sell selection is broken out into per-BU seller channels in Mirakl. Not traced: the specific mechanism connecting "PriceFX selected this product" to "these specific BU channels get a seller record for it."

Why it matters: without this, we can't fully explain what happens end-to-end when a product should be pulled from one BU's channel but not another's — useful for debugging seller-channel discrepancies.

4. Can a BU's raw ERP export and that same BU's PIM output disagree, and if so, how (if at all) is that reconciled?

Where this came from: the "BU PIM vs. BU ERP" callout in the coarse table section. These are confirmed to be different systems (BU PIM = each BU's own {bu}_product.json output, where ID matching happens; BU ERP = the raw SAP/Mincron/etc. feed that stock/availability reads). Nothing in this session traced whether the two are ever cross-validated.

Why it matters: if they can drift silently, a product's attributes/ID (from BU PIM) and its stock/availability (from BU ERP) could be internally inconsistent with no code-level safeguard.

5. Exact call site for the HM (Homans) availability converter was not directly observed

Where this came from: §3's BU→converter table. Confirmed split_homans/HM exists as a function, but the exact DAG file/line calling it wasn't found in the files read this session — flagged as "unconfirmed exact call site." (HY/RM's equivalent open call-site question moved to opportunities.md along with the rest of their content, since neither feeds the W1 pipeline.)

What we need to know: low-priority, but worth a follow-up pass through the Homans-specific DAG files not read in this session, if this table needs to be relied on for anything precise (e.g. an on-call runbook).

6. Does adding customer organizations cause linear or exponential growth in pricing record volume?

This is a scalability question for watsco1-etl-offers, not the PIM pipeline — the user has confirmed product count is expected to stay flat, so the product/PIM pipeline is out of scope for this question. The concern is specifically the pricing phase of pfx_offer_sync.py.

What's confirmed from code + docs:

• The pricing CSV format includes both customer_group_ids and customer_organization_ids as distinct fields (dags/etl_lib/config.py's DEFAULT_PRICE_FIELD_MAPPING; confirmed in aidlc-docs/api-docs/mirakl/API Reference.md: "For Customer Pricing: customer-organization-ids and customer-group-ids columns are required. Multiple price lines per offer are supported (up to 50).") — i.e. PriceFX can scope a price to an individual customer org, not only a group.
• Real QA test data (aidlc-docs/audit.md, 2026-04-07) shows 2732 pricing records against 1232 offer records in the same run — roughly a 2.2:1 ratio, consistent with multiple price lines per offer-SKU, not a flat 1:1.
• Batches are processed strictly sequentially, one at a time, with retries (etl_lib/pipeline.py's process_phase() — no parallelism across batches). Runtime scales with batch count (total records / batch_size), and batch_size is capped at 100,000, defaults to 10,000 (config.py).
• Current observed max phase runtime is ~25 minutes (process_pricing, 2026-08-07 run) against a full DAG budget that appears to have significant headroom (well under the ~1 hour total the user cited).
• Confirmed (per the team): PriceFX batches are already seller-scoped — aidlc-docs/inception/requirements/requirements.md states "PriceFX creates batches per seller automatically", and each Mirakl OF01/PRI01 import is required to be single-store ("Submit one store per import"). This means batch-level parallelism, if pursued (see Open Question #7), wouldn't require any redesign of how batches are grouped — the seller partitioning already exists.

What's NOT confirmed — this is the actual open question: whether PriceFX's MiraklPricesUpload formula generates one pricing record per (offer-SKU × customer group) or per (offer-SKU × customer organization). This determines the growth curve:

• If keyed to customer groups (a much smaller, slower-growing set — new customers typically join existing pricing groups/tiers rather than each getting a unique group), pricing record volume grows close to linearly with SKU count and is largely insensitive to customer count.
• If keyed to individual customer organizations with org-specific pricing (common in B2B distribution for negotiated/contract pricing), pricing record volume can grow closer to (SKU count × customer-org count) — a multiplicative relationship, not additive. Adding customers would then multiply, not add to, total pricing volume.

This is a PriceFX-side business/configuration question (the MiraklPricesUpload formula/report definition), not something visible in either repo's code — same category as the "which products to sell" and BI-sourcing questions elsewhere in this doc set.

Why it matters: if the growth curve turns out to be multiplicative, current headroom (25 min against whatever the real ceiling is — MWAA doesn't have a fixed 1-hour DAG timeout by default, but very long-running tasks have real operational costs: cost, alerting noise, delayed downstream freshness) could erode faster than expected as customer count grows, even with product count held flat. Ask PriceFX/pricing team directly: for a given SKU, is the number of distinct price lines generated bounded by customer-group count (stable) or does it scale with individual customer-org-specific pricing (variable, potentially growing with customer count)?

7. Is Mirakl's rate limit ("max once per minute") per-store, or account-wide across all stores?

Where this came from: follow-up to Open Question #6. The user pointed out that PriceFX batches are already seller-scoped (confirmed above), making per-seller parallelization of the batch loop in process_phase() (etl_lib/pipeline.py) architecturally straightforward — no batch-regrouping needed, since the data is already partitioned by store.

What's confirmed: the requirements documented rate limit guidance — "Submit batches respecting rate limits: recommended every 5 minutes, max once per minute" — and mirakl.py implements 429 detection with Retry-After header handling (RATE_LIMIT_WAIT = 10, exponential backoff up to 60s, max 10 retries).

What's NOT confirmed, and is the actual blocker to answer before parallelizing: whether that rate limit is enforced per Mirakl store/seller or account-wide across the whole Mirakl operator connection. This determines whether parallelizing by seller helps or backfires:

• If per-store, parallelizing submissions across different sellers is safe and should proportionally reduce total runtime — each store has its own rate budget.
• If account-wide (a single limit shared across all sellers under this Mirakl operator account), firing concurrent submissions across sellers would just compete for the same budget and likely trigger more 429s, not fewer — parallelization wouldn't reduce runtime and could make it worse (more retries/backoff).

Why it matters: this is the single fact that determines whether the user's parallelization idea (per-seller concurrent batch processing) is a viable fix for Risk #7's sequential-processing ceiling, or a dead end. Ask Mirakl / check Mirakl's operator API documentation directly — this isn't something either repo's code can answer, since the client just reacts to whatever 429s Mirakl actually sends back, and it hasn't hit them under current sequential (effectively well-under-once-per-minute) load.