Python, Spark, PySpark — a small tech story

-

 




Monday morning. Typical data team standup.
Ravi says “Got the dataset. Around 2 GB. I’ll handle it with Python.”

No drama. He opens his laptop, writes a quick Python script, uses pandas, does some transformations — done. Fast, clean, local.

A few weeks later…

Same team, same Ravi — different situation.
“Guys… the data is now 2 TB.”

He tries the same Python script.
Laptop slows down. Memory crashes. Everything stalls.

This is where Apache Spark enters.

Now Ravi realizes — this is no longer a single-machine problem.
Spark distributes the data across multiple machines and processes it in parallel.

But there’s a catch.

Ravi is comfortable in Python. Not Scala. Not Java.

So instead of switching languages, he uses PySpark.

Now he writes code in Python style, but under the hood, Spark runs it across a cluster.

So what’s really different?

AspectPythonSparkPySpark
TypeProgramming languageDistributed processing enginePython API for Spark
Best forSmall to medium dataMassive datasets (GB → PB)Large data with Python
ExecutionSingle machineCluster (multiple nodes)Cluster (via Spark)
Ease of useVery easyMedium (needs setup, cluster)Easy if you know Python
PerformanceLimited by local resourcesHighly scalableScalable + Python-friendly
Typical toolspandas, NumPySpark Core, Spark SQLPySpark DataFrames
Use casesScripting, ML, APIsETL, big data pipelinesData engineering on Spark





Quick reality check

If you process a CSV on your laptop with pandas → mostly Python
If you process 10 TB of logs across a cluster → Spark
If you write that Spark job in Python → PySpark

Closing thought

Start with Python.
Move to Spark when scale demands it.
Use PySpark when you want scale without leaving Python.


https://www.linkedin.com/pulse/python-spark-pyspark-small-tech-story-saptarshi-biswas-hg7pc

Popular posts from this blog

Building an AI-Driven Ops Command Center with Power BI

-
Image
  Building an AI-Driven Ops Command Center with Power BI Over the last few years, I have been working on a practical framework to move operations teams from reactive monitoring to AI-augmented operations. This article shares a simple, real-world approach to building a governed Ops & Reliability Analytics platform using Power BI – something SREs, DBAs, and platform teams can actually use daily.  1. Architecture First: Don’t Just Connect, Model Properly: One common mistake I see is connecting Power BI directly to raw telemetry tables and expecting magic. With millions of rows, reports quickly become slow and confusing. The solution is a Unified Star Schema.  You should centralise core dimensions like Date, Asset, Service, and Database. On the fact side, bring in telemetry (5‑minute or hourly), incidents, change records, and ML-based risk scores. Keep relationships simple and use single‑direction filters from dimensions to facts. Avoid bi‑directional filters unless absol...
Read more

DLT vs Non-DLT in Databricks: An attempt to come up with guidelines for Oracle and Traditional ETL Teams

-
Image
  The first time I heard people talking seriously about DLT in Databricks, I noticed something interesting. The room was full of smart people, but everyone seemed to mean slightly different things when they said “DLT.” Some people were talking about automation. Some were talking about data quality. Some were treating it like just another pipeline feature. And people coming from Oracle or traditional ETL backgrounds often had the same silent question: “Fine, but what is it really? And how is it different from the way we already build pipelines?” I could relate to that question immediately. Because if you have spent years around Oracle, PL/SQL, scheduler jobs, ETL tools, control tables, recovery scripts, and operational dashboards, then modern cloud data platform language can sometimes sound more complicated than it needs to be. In the older world, even when things were complex, the mental model was clear. You had jobs. You had dependencies. You had scheduling. You had ...
Read more

Data Story @ Bricks

-
Image
 A few years ago, building a data platform felt like managing a crowded marketplace.  There was a data lake sitting quietly in object storage. A warehouse lived somewhere else for dashboards. ETL pipelines ran in their own tool. Streaming had another engine. Machine learning experiments happened in separate notebooks. Each team had its space. Each system did its job. But they didn’t naturally work together. Now picture a fast-growing retail company expanding across cities. Sales data flows in daily. Engineers load raw files into cloud storage. Analysts copy pieces into a warehouse for reports. Data scientists request extracts to build models. Meanwhile, governance teams try to answer simple questions like, “Who accessed this table?” The answers aren’t always clear. Nothing is completely broken. But everything feels stitched together. Databricks entered this story with a different idea. Instead of improving the stitching, it asked: What if the lake itself could act like a wareh...
Read more