Data Story @ Bricks

-

 A few years ago, building a data platform felt like managing a crowded marketplace. 



There was a data lake sitting quietly in object storage. A warehouse lived somewhere else for dashboards. ETL pipelines ran in their own tool. Streaming had another engine. Machine learning experiments happened in separate notebooks. Each team had its space. Each system did its job. But they didn’t naturally work together.

Now picture a fast-growing retail company expanding across cities. Sales data flows in daily. Engineers load raw files into cloud storage. Analysts copy pieces into a warehouse for reports. Data scientists request extracts to build models. Meanwhile, governance teams try to answer simple questions like, “Who accessed this table?” The answers aren’t always clear.

Nothing is completely broken. But everything feels stitched together.

Databricks entered this story with a different idea. Instead of improving the stitching, it asked: What if the lake itself could act like a warehouse?

That idea became the Lakehouse.

With Delta Lake, cloud storage gained reliability — transactions, schema checks, even the ability to “time travel” to previous versions of data. Suddenly, the lake wasn’t just cheap storage. It became dependable. Pipelines broke less often. Data teams worried less about overwritten files.

Performance improved too, with an optimized runtime and the Photon engine quietly speeding things up behind the scenes.

But the deeper shift wasn’t technical — it was architectural.

Batch and streaming ran on the same engine. SQL analytics and Python machine learning lived in the same workspace. Engineers and data scientists collaborated in shared notebooks. Governance worked across teams through Unity Catalog.


Instead of moving data between systems, teams stayed in one environment.

As AI became central to enterprise strategy, this mattered even more. Feature stores, experiment tracking, model registries, vector search, and LLM integrations became part of the same platform. The AI capabilities didn’t sit somewhere else. They lived where the data already lived.


And that reduced friction.


In enterprise systems, friction is expensive — in time, in integration effort, in operational risk. By reducing fragmentation, the focus shifts. The conversation is no longer, “Which tool should we add?” It becomes, “How do we design this Lakehouse well?”


In a world of exploding data and accelerating AI, simplicity becomes strategic. Sometimes innovation isn’t about adding another component.


It’s about removing the unnecessary ones.

Popular posts from this blog

Building an AI-Driven Ops Command Center with Power BI

-
Image
  Building an AI-Driven Ops Command Center with Power BI Over the last few years, I have been working on a practical framework to move operations teams from reactive monitoring to AI-augmented operations. This article shares a simple, real-world approach to building a governed Ops & Reliability Analytics platform using Power BI – something SREs, DBAs, and platform teams can actually use daily.  1. Architecture First: Don’t Just Connect, Model Properly: One common mistake I see is connecting Power BI directly to raw telemetry tables and expecting magic. With millions of rows, reports quickly become slow and confusing. The solution is a Unified Star Schema.  You should centralise core dimensions like Date, Asset, Service, and Database. On the fact side, bring in telemetry (5‑minute or hourly), incidents, change records, and ML-based risk scores. Keep relationships simple and use single‑direction filters from dimensions to facts. Avoid bi‑directional filters unless absol...
Read more

DLT vs Non-DLT in Databricks: An attempt to come up with guidelines for Oracle and Traditional ETL Teams

-
Image
  The first time I heard people talking seriously about DLT in Databricks, I noticed something interesting. The room was full of smart people, but everyone seemed to mean slightly different things when they said “DLT.” Some people were talking about automation. Some were talking about data quality. Some were treating it like just another pipeline feature. And people coming from Oracle or traditional ETL backgrounds often had the same silent question: “Fine, but what is it really? And how is it different from the way we already build pipelines?” I could relate to that question immediately. Because if you have spent years around Oracle, PL/SQL, scheduler jobs, ETL tools, control tables, recovery scripts, and operational dashboards, then modern cloud data platform language can sometimes sound more complicated than it needs to be. In the older world, even when things were complex, the mental model was clear. You had jobs. You had dependencies. You had scheduling. You had ...
Read more