{"id":72937,"title":"PeerDB - Modern ETL in minutes, with SQL","tagline":"Data store-first ETL/ELT, enabling 10x faster syncs and transforms","body":"## **TL;DR**\n\nPeerDB is a Postgres-compatible SQL interface to seamlessly integrate multiple data stores. It enables data engineers to sync, transform and query data across their stores using simple SQL commands. PeerDB takes a data store native approach in engineering — enabling 10x faster ETL/ELT.\n\n## **Problem**\n\nExisting ETL/ELT tools primarily focus on supporting a wide range of connectors at the expense of delivering high-quality ones. This becomes evident when your workloads need scale or have demanding feature requirements. While Sai was working at Microsoft, it was common to see large enterprise customers try out the existing ETL tools and fail – as the tools were not meeting performance and reliability SLAs or were lacking the required features. Customers spent significant time and resources to build and manage in-house data pipelines.\n\n## **Our Solution**\n\n### Data-store nativity at its core, enabling 10x faster ETL/ELT\n\nPeerDB takes a data store-first approach to data movement. It supports highly adopted stores and implements multiple infrastructural and data store native optimizations – providing a blazing-fast and feature-rich ETL experience.\n\nWe are starting our journey by providing the world's best data integration/ETL experience for **PostgreSQL,** [one of most used and popular databases](https://db-engines.com/en/ranking_trend/system/PostgreSQL). For example, on Postgres we parallelize initial load for larger tables using CTID scans. This yields orders of magnitude speedups - Syncing 100s of GB reduces from days to minutes.\n\nOur architecture is designed for real-time syncs, which enables 10x faster Change Data Capture with data freshness of a few seconds even at large throughputs (10k+ tps). For example, in a sync from Postgres to BigQuery or Snowflake, PeerDB is 10 times faster than other tools and provides \u003c30s data-freshness guarantees.\n\n![uploaded image](/media/?type=post\u0026id=72937\u0026key=user_uploads/728174/dc134471-e0d0-48c5-bbda-6d2ec339c421)\n\n### SQL as the ETL/ELT standard\n\nThe Postgres-compatible SQL layer for ETL makes the life of data engineers very easy. They can develop pipelines using a framework they are familiar with, without needing to deal with custom UIs and REST APIs. They can use [Postgres' 1000s of integrations](https://github.com/PeerDB-io/peerdb#postgres-compatible-sql-interface-to-do-etl) to build and manage ETL.\n\nWe extend Postgres' SQL grammar with a few new intuitive SQL commands to enable real-time data streaming across stores. As we evolve, we will make this grammar richer and the standard for ETL/ELT.\n\n## **Our Asks**\n\nIf PostgreSQL is a critical component of your data stack and you are running into challenges moving data in and out of Postgres, reach out to us at [founders@peerdb.io](mailto:founders@peerdb.io) or join our waitlist at \u003chttp://peerdb.io/\u003e We would love to collaborate!\n\nAlso check out PeerDB’s Open Source repo - \u003chttps://github.com/PeerDB-io/peerdb\u003e (don’t hesitate to leave us a star!)\n\n## **Finally, a bit about us**\n\nWe are Sai and Kaushik. We've been best buddies for the past 15 years now. We were roommates back in undergrad and actively collaborated since then. Now we have decided to start a business together! 🙂\n\n![uploaded image](/media/?type=post\u0026id=72937\u0026key=user_uploads/728174/4795901c-d1ac-4e3e-b565-7113c9ef3ca0)\n\nPrior to PeerDB, Sai worked at Microsoft leading Solutions Eng. for all PostgreSQL services on Azure. He was an early engineer at Citus Data (YC S11) and saw it through the Microsoft acquisition. Prior to PeerDB, Kaushik built operating systems and led data teams at Google, Safegraph and Palantir. He also represented India in ACM ICPC World Finals. With our diverse and complementary experiences, we are excited and confident to build the next-generation ETL/ELT!","slug":"IyP-peerdb-modern-etl-in-minutes-with-sql","created_at":"2023-07-11T16:24:18.528Z","updated_at":"2026-05-25T01:49:21.198Z","total_vote_count":60,"url":"https://www.ycombinator.com/launches/IyP-peerdb-modern-etl-in-minutes-with-sql","share_image_url":"//bookface-static.ycombinator.com/assets/ycdc/yc-og-image-c440a0ad1dacfb86eeeb343717479cc54d256614449b4ef719977a0a451f8bc8.png","company":{"id":28659,"name":"PeerDB","slug":"peerdb","url":"https://peerdb.io/","logo":"https://bookface-images.s3.amazonaws.com/small_logos/4a83477f0ae90481f26400baea36bd4c2fed890a.png","batch":"Summer 2023","industry":"B2B","tags":["Developer Tools","Open Source","Data Engineering","Enterprise Software","Databases"],"search_path":"https://bookface.ycombinator.com/company/28659"}}