Dremio Story

-

 Dremio Story



 



 

Dremio is a SQL-based data lakehouse query engine that allows analytics, BI, and AI workloads to run directly on cloud object storage such as S3 and ADLS without copying data into a warehouse.

In traditional architectures, data moves from storage to ETL pipelines and then into warehouses before BI or ML can use it. This creates latency, duplication, and cost. Dremio removes this by allowing all tools to query the lake directly. Dremio provides a semantic layer, data virtualization, and a high-performance SQL engine optimized for Apache Iceberg and Parquet. Its key innovation is Reflections – optimized materializations similar to indexed materialized views that accelerate queries without moving data.

 




In modern data fabric architectures, Dremio sits between storage and consumption layers such as Power BI, Tableau, Python, Databricks, and even LLM-based RAG systems. Databricks on the other hand is a full Lakehouse OS that includes ML, notebooks, pipelines, and governance via Unity Catalog. Unity Catalog governs Delta tables, ML features, models, and access – but only within Databricks.

Dremio offers an open equivalent using Dremio Catalog + Apache Iceberg + Project Nessie. Nessie brings Git-style versioning, branching, and rollback for data – enabling CI/CD for data.  Unlike Unity Catalog, which only works inside Databricks, Dremio’s catalog can be used by Power BI, Spark, Trino, Python, and LLM pipelines simultaneously.

In enterprise patterns, Databricks is used for AI and ML training while Dremio is used as the governed data access and BI layer across tools.  This makes Dremio ideal for Data Fabric and Data Mesh architectures where multiple engines need to work on the same governed data.

Popular posts from this blog

Building an AI-Driven Ops Command Center with Power BI

-
Image
  Building an AI-Driven Ops Command Center with Power BI Over the last few years, I have been working on a practical framework to move operations teams from reactive monitoring to AI-augmented operations. This article shares a simple, real-world approach to building a governed Ops & Reliability Analytics platform using Power BI – something SREs, DBAs, and platform teams can actually use daily.  1. Architecture First: Don’t Just Connect, Model Properly: One common mistake I see is connecting Power BI directly to raw telemetry tables and expecting magic. With millions of rows, reports quickly become slow and confusing. The solution is a Unified Star Schema.  You should centralise core dimensions like Date, Asset, Service, and Database. On the fact side, bring in telemetry (5‑minute or hourly), incidents, change records, and ML-based risk scores. Keep relationships simple and use single‑direction filters from dimensions to facts. Avoid bi‑directional filters unless absol...
Read more

AzureSQL Elastic Pool: Why Scaling?

-
Image
   AzureSQL Elastic Pool: Why Scaling?                    Once upon a sprint review in a glass‑walled Bengaluru meeting room, two heroes walked into the architecture slide deck: “Native Scaling” and “Auto‑Scaling Automation”.   One wore a crisp white shirt, spoke politely, and waited for instructions.   The other had a hoodie, a monitoring dashboard open, coffee in one hand, and alarms already ringing in Azure Monitor.   And that, my friends, is how most real‑world cloud conversations start 😄 .   Let’s decode this with a story instead of dry documentation.   Imagine your Azure SQL Elastic Pool is like a shared PG apartment in HSR Layout.  Multiple tenants. Different habits. Some cook Maggi at midnight, others run full biryani feasts at quarter‑end.   Native scaling is when the landlord increases the electricity meter *after* tenants complain.   Auto‑scaling is when s...
Read more

Databricks Prerequisites – From Real Project to Real Platforms

-
Image
  Databricks Prerequisites – From Real Project to Real Platforms As I am wrapping up my current enterprise data platform project and moving into deep Databricks integration work, I realised something very clearly – most failures in Databricks programs do not come from the tool, they come from missing foundations. So before teams jump into notebooks and pipelines, I want to share what really matters, based on what I saw on the ground. Databricks is NOT a replacement for ADF or Airflow. Tools like Azure Data Factory are your ingestion and orchestration layer. Databricks is your heavy‑duty processing, analytics, and AI engine. Think of it like this: Sources → ADF / Airflow → Databricks → BI / ML / APIs Databricks runs on Apache Spark. If you do not understand partitions, shuffles, executors, and joins, you will burn money and get slow pipelines. Databricks does not remove the need to understand distributed systems – it only makes them easier to run. Clusters are not just ‘...
Read more