Opportunities — Dormant Data Sources Not Used by supplysync.com

While researching data-lineage.md, several data sources and integrations turned up in airflow-pim-dags that are real, working code — but none of it feeds the W1/Mirakl/Algolia/Shopware pipeline that powers supplysync.com. They're documented here, separately from the lineage doc, so they don't get mixed in with what's actually live. If any of these ever gets wired into W1, move the relevant content back into data-lineage.md as a real source.

NeuCo

What it is: a distinct product/pricing data source with two independent, unrelated integrations in airflow-pim-dags:

1. Standalone NeuCo Algolia index — jobs-alg-neuco.py (dag_id='ecom_algolia_neuco'), jobs-neuco-reformat.py, jobs-neuco-s3.py. These SFTP-pull NeuCo source files, reformat them, and push to a dedicated NeuCo Algolia index (Variable.get('NEUCO_ALGOLIA_INDEX')) via neuco_utils.process(price_path, xref_path, dimensions_path, out_path, env). All three DAGs run on live cron schedules (NEUCO_UTC_START_TIME, NEUCO_UTC_REFORMAT_START_TIME, NEUCO_UTC_S3_START_TIME) — this part is genuinely active, just not part of the W1/supplysync.com pipeline.
2. PIM attribute/availability gap-fill — availabilityNormalizer.convert_neuco_data() (computes NeuCo's own cost/price via NEUCO_MARKUP, and stock with branch hardcoded to '1' — no real per-branch granularity) and all_products.add_neuco_attrs() (appends NEUCO_PRICE/NEUCO_MIN_QTY/NEUCO_N2G/NEUCO_TRUCK_SHIPMENT pseudo-attributes). Gated behind use_neuco/ALL_USE_NEUCO/XREF_USE_NEUCO flags (default 0/off) in jobs-alg-all.py and jobs-alg-xref.py. Confirmed not called anywhere in any of the four W1 DAGs (jobs-inf-w1-{full,full-pre,delta,delta-pre}.py — zero neuco/NEUCO hits).

Opportunity, if this ever needs to reach supplysync.com: the gap-fill logic and pricing computation already exist and are tested (they run live for the standalone NeuCo Algolia index). Wiring NeuCo into W1 would mean either (a) calling merge_bu_erp-equivalent logic against NeuCo's own {bu}_product.json-style feed the way other BUs already work, or (b) extending all_products.py's W1 fan-in to optionally include NeuCo the way jobs-alg-all.py does. Either way, this isn't a from-scratch build — the components exist, just aren't connected to W1.

NeuCo's BU code is NC (confirmed via addfield('bu', 'NC') in convert_neuco_data and the bu_availability_files.pop('NC', None) toggle).

Rheem

What it is: a per-BU pricing/availability integration, structurally identical in pattern to NeuCo. availabilityNormalizer.convert_rheem_data() computes its own cost/price via RHEEM_COST_ADJUST/RHEEM_PRICE_ADJUST multipliers, reads a Rheem pipe-delimited feed plus an endless-aisle PID/MFG cross-reference, and is gated behind a use_rheem flag (default off) in the same jobs-alg-all.py/jobs-alg-xref.py DAGs. Confirmed not called anywhere in any of the four W1 DAGs (zero rheem/RHEEM hits).

Rheem's BU code is RM (confirmed via addfield('bu', 'RM') and the use_rheem toggle in availabilityNormalizer.merge()/merge_pid()).

Unlike NeuCo, Rheem doesn't appear to have a dedicated standalone Algolia index/DAG family of its own in the files read this session — its footprint in the repo is smaller, scoped to the same gap-fill/pricing pattern inside jobs-alg-all.py/jobs-alg-xref.py.

Opportunity: same shape as NeuCo — the per-BU pricing/availability computation already exists, just needs a W1 integration point if this BU ever needs to sell through supplysync.com.

Hydros (HY)

What it is: availabilityNormalizer.convert_hy_data() — a Hydros-specific availability feed (env.hydros_items_colmap-defined, branch hardcoded to '500', a single-branch BU), plus jobs-alg-hydros.py (HYDROS_USE_NEUCO flag lives here too). Not called in any W1 DAG.

Lower-confidence entry than NeuCo/Rheem — the exact call site for convert_hy_data() wasn't directly observed in the files read this session (see the equivalent open question in open-questions.md), so this is included for completeness rather than as a fully-traced opportunity.

ecom_products_xref (the "xref" DAG)

What it is: jobs-alg-xref.py (dag_id='ecom_products_xref') — a standalone DAG with schedule_interval=None (manually/externally triggered, not on a cron schedule), writing to its own separate S3 archive (XREF_S3_ARCHIVE). It's not a stage in the W1 pipeline and isn't referenced by any W1 DAG file. It's also where XREF_USE_NEUCO and the older two-hop xrefNormalizer.merge_erp()/erp_pim_data_{bu}.csv xref-file mechanism (superseded in W1 by merge_bu_erp() per data-lineage.md §2) still live.

Not really an "opportunity" in the same sense as NeuCo/Rheem/Hydros — it's a separate, independently-purposed DAG rather than a dormant BU integration. Included here mainly so a reader who noticed XREF_USE_NEUCO/XREF_S3_ARCHIVE while reading the code understands why it doesn't show up in data-lineage.md: it's genuinely a different pipeline, not part of W1's scope by any measure (schedule, trigger, output destination, or code reference).

Why these were removed from data-lineage.md

data-lineage.md is scoped to the pipeline actually powering supplysync.com (the jobs-inf-w1-* DAGs plus watsco1-etl-offers). Documenting NeuCo/Rheem/Hydros/xref there — even with "confirmed not used" caveats — mixed the signal: a reader skimming that doc for "what actually feeds the site" would have to read through several paragraphs of "this exists but doesn't apply" per dormant source. Per the team, since none of this data is currently used in supplysync.com, it doesn't belong in the lineage document at all — it belongs here, as a record of dormant capability worth knowing about if priorities change.