Mirakl / PIM / Offers Dataflow

Documentation of how products, offers, and pricing flow from backend systems of record through to the customer-facing storefronts on supplysync.com — covering airflow-pim-dags and watsco1-etl-offers.

Diagrams

Start with the context diagram, then drill into whichever pipeline you need detail on. Diagrams are SVG with hover tooltips on every node and edge, and support pan/zoom (drag to pan, scroll to zoom, buttons top-right to reset or go fullscreen).

Context

The whole ecosystem, tiered by distance from the customer: Backend Systems of Record → ETL Pipelines → Midtier Integration Services → Customer-Facing. Includes a team ownership color overlay.

PIM DAG detail

Drill-down into ecom_w1_full, ecom_w1_full_pre, ecom_w1_delta, ecom_w1_delta_pre — sources, normalization, and the Mirakl/Shopware/Algolia fan-out.

Offers DAG detail

Drill-down into pricefx_mirakl_load_{stg,prd} — the PriceFX 3-step batch workflow.

Scope

Pattern-level detail, not full enumeration. The wider airflow-pim-dags repo has ~150 generated DAGs (per-BU/locale variants for Shopware and Algolia — jobs-shw-*, jobs-alg-*, etc.); those are out of scope here. This documentation covers only the four jobs-inf-w1-* DAGs plus watsco1-etl-offers, since those are the ones supplysync.com/W1 depends on and that touch Mirakl.

The dataflow, tier by tier

Backend systems of record

• Business Unit ERPs — one per BU (BK, CE*, EC, GM, HM, PP, TE, ...). Feed the PIM DAG in batch, and are queried live by Shopware via Conduit for real-time inventory.
• PriceFX — pricing engine. Bidirectional: pulls batches for the offers pipeline, then writes per-batch status and aggregate run stats back.
• BI (Snowflake) — tells PriceFX which products to sell, and separately signals current stock to the PIM feeds bucket. The stock signal is inventory-by-branch, used as an ERP proxy for Algolia sorting (not a live ERP read — actual real-time stock is served to Shopware via Conduit). BI also supplies historical item sales data, used for ranking/relevance alongside the inventory signal.
• PIM (Informatica Product360) — the product master. Picks up newly-priced items from a dedicated PriceFX→PIM S3 bucket, flags them active in SupplySync, and exports nightly to the PIM feeds bucket.
• PriceFX→PIM bucket — s3://watsco-pricefx/Production/SupplySync/pricefx_pim_priced_items_*.csv. The hand-off point between PriceFX's product selection and PIM's product mastering.
• PIM feeds bucket — nightly Product360 export, read by the PIM DAG.

ETL pipelines

• PIM DAG (ecom_w1_full, ecom_w1_full_pre, ecom_w1_delta, ecom_w1_delta_pre) — normalizes PIM/sales/TE feeds, then fans out to Mirakl (products, taxonomy, sellers, offers-read), Shopware (product export to S3), and Algolia (supply_sync index). The per-BU PIM feed supplies long description (authoritative, not a gap-fill — no separate ALL/W1 base description field exists) and matches Mirakl PIDs to each BU's ERP item IDs, via the same merge_bu_erp mechanism — a product not present in a BU's feed gets no seller-channel entity for that BU in Mirakl.
• Offers DAG (pricefx_mirakl_load_{stg,prd}) — pulls PriceFX batches, validates, transforms to CSV, submits to Mirakl (OF01-03 / PRI01-03), and writes status back to PriceFX.

Cross-repo dependency: the Offers DAG will fail with "product does not exist" Mirakl errors if the PIM DAG hasn't already pushed the referenced SKU as a Mirakl product(operator and mapped sellers). There's no enforced ordering between the two Airflow environments (MWAA for offers vs. the legacy PIM Airflow) — this is an implicit operational dependency, not a coded one[RISK].

Midtier integration services

• Mirakl Marketplace Operator API — receives products/taxonomy/sellers/offers from the PIM DAG and offers/pricing from the Offers DAG; is read by Shopware for offer display.
• Algolia (supply_sync index) — receives the merged Mirakl-offers + Shopware-fields feed from the PIM DAG; queried by storefronts for search.
• Conduit — internal bespoke ESB, fronting the per-BU ERPs with per-BU auth. Used by Shopware for real-time inventory lookups, separate from the batch ERP feed into the PIM DAG.

Customer-facing

• Shopware — the ecommerce engine for supplysync.com. Pulls the product export from S3 (doesn't get pushed to directly), calls Mirakl's offer read APIs to surface marketplace offers, and calls Conduit for live inventory.
• W1 storefronts (supplysync.com) — the only thing customers actually touch. Reached via Shopware and via Algolia search — never directly via Mirakl.

Team ownership

A color overlay in the context diagram maps each node to an owning team. This is a working assumption provided by the team, not sourced from any org chart or code metadata — DAG owner fields in both repos are generic placeholders ('watsco', 'data-engineering'), not real team assignments.

Team	Owns
Ecom	Everything not otherwise assigned: PIM DAG, Offers DAG, Algolia, Shopware, W1 storefronts
Ops	Mirakl, Conduit
PIM	PIM (Informatica Product360), PIM feeds bucket
Pricing/Product	PriceFX, PriceFX→PIM bucket, BI (Snowflake)
Business Units	Business Unit ERPs

Contributing

This documentation is diagram-as-code, built with the Python diagrams library and Graphviz, structured as a C4-style model (one context diagram, two component-level detail diagrams) rather than a single monolith. See fullstack/README.md for the full directory layout, the Makefile targets, why the C4 split exists (a single monolithic diagram broke Graphviz's layout engine), and icon sourcing/licensing notes.

cd docs/fullstack
make diagrams     # bootstraps the venv (first run) and renders all 3 SVGs

Source: Bitbucket