Central orchestrator invokes all transformers in parallel. Consolidated Slack alerts with
per-transformer status. Handles failures gracefully without blocking other pipelines.
Lambda Invoke
Parallel Execution
Slack Alerts
Read Parquet from S3, transform to platform-specific formats. GA4 Measurement Protocol, Meta
Conversions API, Klaviyo Events API. Server-to-server, no client-side tracking.
GA4 S2S
Meta CAPI
Klaviyo
FRED API for macroeconomic indicators (interest rates, GDP, CPI). Census API for demographic and
business data. Enriches marketing data with external context.
FRED API
Census API
Parquet
Processes JSON events from S3 (via website Lambda). Outputs events.parquet, conversions.parquet,
sessions.parquet. Daily runs at 10:00 AM UTC for previous day's data.
S3 JSON
Parquet Output
Daily Schedule
Full Terraform modules for Lambda, S3, EventBridge. Environment-specific configs (dev/prod).
Lifecycle rules for data retention and S3 Glacier transitions.
Terraform
EventBridge
Modular IaC
Sentry for error tracking across all Lambdas. Slack webhooks for pipeline status. CloudWatch logs
with structured JSON. Per-transformer success/fail counts.
Sentry
Slack
CloudWatch
AWS
Lambda (7)
S3
EventBridge
Secrets Manager
CloudWatch
IAM
Stack
Node.js 20
TypeScript (strict)
parquetjs
Terraform
Schedule
Website Analytics: 10:00 UTC
FRED: 10:30 UTC
Census: Quarterly
Orchestrator: 10:45 UTC
Outputs
GA4 Measurement Protocol
Meta Conversions API
Klaviyo Events API
Neural Core Data Lake
S3 Bucket
neural-analytics-data-production-047719662689
Lambdas
pipeline-orchestrator-prod
website-analytics-collector-prod
fred-collector-prod
Transformers
ga4-transformer-prod
meta-capi-transformer-prod
klaviyo-transformer-prod
Repository
github.com/neuralpartners/neural-partners-data (private)