kafka · flink · spark · aws glue · athena · rbac
Streaming ingestion platform on AWS
End-to-end reference architecture: multi-source ingestion through Kafka, Flink, and JDBC connectors, ETL with Spark and Glue, Athena serving dashboards — governed by RBAC and IAM from day one. Watch the data flow.
The architecture
Animated edges show live data paths; dashed lines show where security policy is enforced. Drag any node to explore:
How data moves
Ingestion. Application events and server logs stream directly into Kafka. Operational databases flow in through JDBC connectors on Kafka Connect — change data without touching production query load. SaaS APIs land via Flink jobs that normalize as they ingest.
Processing. Flink handles the streaming path — enrichment, sessionization, and late-data handling on the way into the lake. Spark runs the heavier batch ETL. Both write to a bronze/silver/gold S3 lakehouse, with AWS Glue maintaining the catalog and running scheduled transformation jobs.
Serving. Athena provides interactive SQL over the governed gold layer, feeding BI dashboards without a warehouse to babysit — you pay per query, not per idle cluster.
Security is a layer, not an afterthought
Every component sits on the same governance plane: IAM policies scope service access, RBAC (via Lake Formation) controls who queries which tables and columns, data is encrypted at rest and in transit, and audit logging covers every access path. The dashed edges in the diagram are the point — security attaches to the platform, not to individual pipelines, so new data products inherit it automatically.