I am pretty honest: there are a ton of vibecoded projects around, and I made another one... great huh?
I'm working as a data platform engineer in Germany. My day job is basically spinning up & running lakehouse platforms on US hyperscalers for SMB and some DAX companies. I mean, data engineers like to work on these platforms but since some time I really have ethical concerns about it - like: - vendor lock-in - high costs (because of cloud costs - and in most cases you are not able to connect your own metal if you have one) - non-EU vendors... - other open-source solutions have high license costs for LDAP etc. pp.
That's why is started to build my own opinionated data lakehouse platform in my spare time (tbh I also started learning agentic engineering and what harnesses are..), only curated from open source projects already available.
The building blocks are pretty straightforward imo: - Iceberg tables on whatever S3 storage you already have - Trino for SQL and Spark Connect for jobs - Airflow for schedules, plus notebooks, Kafka streams and dashboards - One web UI and one API over all of it - stuff around like OIDC, secrets management etc.
It's fully pre-alpha. But I want to use it as an internal pitch project at my current employer to push agentic engineering & the idea of creating our own data platform for our non-hyperscaler customers.
Site: https://stardelt.io Code: https://github.com/stardelt
I'm happy to hear what you think about it :)
Cheers, Martin
Comments URL: https://news.ycombinator.com/item?id=50411800
Points: 1
# Comments: 0
