Data Pipelines

Data pipelines ingest data from external sources — Shopify, Facebook Marketing, Google Ads, TikTok, Klaviyo, and others — into your Athena data lake. Pipelines are scoped to a tag and managed through Airbyte, which runs inside your Blotout Cloud deployment and is orchestrated by Airflow.

Create a pipeline

Pipelines are created from a tag, not from the Data Pipelines page directly. Each pipeline is scoped to the tag you select.

Prerequisite — Blotout Cloud setup

If Blotout Cloud is not set up for the tag, complete that first before creating a pipeline. Open Tags from the left sidebar under Platform, find your tag, and click Setup Blotout Cloud in the Action column. See Setting up a tag for details.

1. Go to Blotout Cloud and select your Team from the top bar.
2. Open Tags from the left sidebar under Platform.
3. Find the tag you want to connect a source to. Use the search box to filter by tag name if needed. Confirm the Blotout Cloud column shows a checkmark for that tag.
4. In the Action column, click Create Pipeline for that tag row.

5. On the Add a new data pipeline screen, choose a source connector from the catalogue (for example, Facebook Marketing, Shopify, Google Ads, Recharge, or TikTok Marketing). Use Search Available Sources to filter the list.
6. Complete the remaining steps — Schema and Summary — and enter the credentials required by the connector (API tokens, account IDs, or OAuth depending on the source).
7. Configure the sync schedule and destination schema. Blotout creates an Athena schema following the naming convention <SOURCE_NAME>_<ENV> documented in Data Storage Standards.
8. Save and run the first sync.

Once the pipeline is active, Airflow creates an organization-scoped DAG (for example, acme_shopify) that runs on the configured schedule.

What happens after connection

Step	System	Result
1	Airbyte	Syncs raw source data into S3
2	Airflow DAG	Triggers on schedule
3	dbt models	Transform source data into analytics-ready tables
4	Dashboards	Dashboard DAGs refresh reporting models

Supported sources

Source	Setup guide
Facebook Marketing	Sync Meta Ads campaigns, creatives, and insights via OAuth
Shopify	Sync orders, products, and customers via API Password
Google Ads	Sync campaign and performance metrics via OAuth
Recharge	Sync subscriptions, charges, and customers via Access Token
TikTok Marketing	Sync campaigns, ads, and delivery metrics via OAuth 2.0
Amazon Ads	Sync Sponsored Products, Brands, and Display via Client ID / Secret / Refresh Token
Amazon Seller Partner	Sync Seller/Vendor Central orders and catalog via LWA credentials
Klaviyo	Email / CRM data
Postgres, MySQL	Database connectors

Additional connectors can be configured through the Airbyte UI. Contact your Blotout administrator if you need a source that is not listed.

Related documentation

• Airflow — DAG schedules and monitoring
• Dashboards — analytics built on pipeline data
• Data Storage Standards — where pipeline data is stored
• Tags — pipelines are scoped to a tag