Lineage: where data comes from and where it goes

What it is

Lineage answers “where did this data come from?” and “who uses it?” For a governed data product, that means: which systems or pipelines feed into this product (e.g. Shopify orders, Stripe payments, Zendesk tickets), and which apps, reports, or AI flows consume it. Lineage is used for impact analysis (“if we change the orders schema, what breaks?”), compliance (“prove where this field originated”), and debugging (“why is this value wrong?”). In Loxtep, every governed data product has lineage metadata so you and your AI can reason about provenance and risk.

How it’s made

Lineage for a governed data product is captured and maintained as follows:

• Upstream registration. When you connect a source (e.g. Shopify, Stripe) or a pipeline to the asset, we record it as an upstream node: system name, connection id, which entities/fields are ingested, and when the link was created. That gives a clear “this asset is fed by these systems.”
• Downstream registration. When a consumer (app, report, AI agent) is granted access or first calls the consumption API, we can register it as a downstream node: consumer name, use (e.g. “support dashboard,” “refund bot”), and optionally which fields or endpoints they use. So you know “this asset is used by these consumers.”
• Field-level lineage (optional). For stricter governance, lineage can be tracked at field level: e.g. “order.total comes from Shopify order.total_amount.” That’s stored as (asset, field) → (source_system, source_field) and is used for impact analysis and compliance.
• Graph and API. Lineage is stored as a directed graph (nodes = systems/asset/consumers, edges = feeds/consumes). It’s exposed via an API (e.g. “GET /assets/{id}/lineage”) and can be visualized in the Loxtep UI so you see the full picture at a glance.

Example lineage payload for an Orders context asset:

{
  "asset_id": "orders-context",
  "lineage": {
    "upstream": [
      { "system": "Shopify", "connection_id": "conn_shopify_1", "entities": ["Order", "Customer"], "linked_at": "2025-01-15" },
      { "system": "Stripe", "connection_id": "conn_stripe_1", "entities": ["Payment"], "linked_at": "2025-01-15" },
      { "system": "Zendesk", "connection_id": "conn_zendesk_1", "entities": ["SupportTicket"], "linked_at": "2025-02-01" }
    ],
    "downstream": [
      { "consumer": "Support dashboard", "use": "Order/customer context in ticket view", "first_used": "2025-02-10" },
      { "consumer": "Refund approval bot", "use": "Order + Payment + Decision context", "first_used": "2025-03-01" }
    ],
    "field_lineage_sample": [
      { "asset_field": "order.total", "source": "Shopify", "source_field": "order.total_price" }
    ]
  }
}

In Loxtep, lineage is updated when you add or remove connections and when consumption is first used. It’s part of the governed data product’s metadata and is available for impact analysis, compliance reports, and AI context (e.g. “this context is built from Shopify, Stripe, Zendesk”).

Why it matters for governed data products and AI

Without lineage, you can’t answer “why do I see this value?” or “what breaks if we change that source?” For AI, lineage helps the system (and you) understand the provenance of the context it’s using — which builds trust and supports compliance. In Loxtep, every governed data product has lineage so the context we deliver is traceable from source to consumer.