Bring joy to data pipelines again. Data. It's everywhere- used for everything. It helps us make intelligent business decisions, makes our teams more efficient and transparent, trains our AI and machine learning models, or allows users to access and change their account info. Unless you're a DevOps engineer, you may not realize that these pipelines- that ferry this critical information around- are exceptionally fragile. Our engineering teams face several complex challenges when updates or new features are needed. So much so that, in many instances, they lose the joy and fun out of their jobs. Some of these problems include costly data outages, inaccurate reporting or metrics affecting the business outcomes, and delayed deliverables due to complex communication chains between engineering and stakeholders. Because of these problems, engineers and data practitioners often resort to building new pipelines and datasets instead of changing existing ones, escalating technical debt, operational costs, and increasing complexity. At the heart of the open-source project SQLMesh, we have created a novel data pipeline development workflow from our combined experience at companies like Airbnb, Apple, Google, and Netflix. We designed this data transformation tool to handle complexities associated with evolving data pipelines at the internet scale. In this session, we will talk about the challenges data practitioners face today, how they hinder team agility, core concepts of SQLMesh, including virtual data environments, automatic detection of breaking changes, continuous testing and accurate previews of changes and end with a demo/workshop. Join Marisa if you are an agile facilitator working with data engineers or a data engineer yourself looking to find free open-source tools that will help your team speed development, stay aligned and minimize errors in production. This session will have something for everyone!
- How SQLMesh open source project can help data teams and pipelines be more efficient, scale effectively, create transparency and increase data correctness
- Implement SQLMesh in their data pipelines for POC’s with their own companies