Apache Spark & PySpark for Data Engineers
Distributed processing with DataFrames, Spark SQL, partitioning, joins and performance tuning on realistic datasets.
- ✓ Instructor-led live sessions, online or in Bengaluru
- ✓ Hands-on labs in every module
- ✓ Capstone project with trainer review
- ✓ Curriculum customisable for corporate teams
- ✓ Certificate of completion
Next batch: Sat, 09 Jan 2027 · 32 hours · 4 days
Data EngineeringApache Spark & PySpark for Data Engineers
Duration: 32 hours · 4 days
Next batch: Sat, 09 Jan 2027
Format: Online live · Bengaluru classroom
Modules: 8 · Topics: 38
Level: Intermediate
Fee: ₹14,999
Trusted by leading experts
Upcoming batches
Open batches for individuals; corporate batches on your dates.
| Programme | Starts | Mode | Schedule | Duration | |
|---|---|---|---|---|---|
| Apache Spark & PySpark for Data Engineers | Sat, 09 Jan 2027 | Online live | Weekend (Sat) | 5 Saturdays | Enquire |
| Apache Spark & PySpark for Data Engineers | Mon, 08 Mar 2027 | Online live | Weekday | 4 days | Enquire |
Soaring demand, and where it takes you
Spark remains one of the most widely used engines for large-scale data processing, and tuning skills are particularly valued.
Become an expert
Roles this programme prepares you for:
Become a Big Data Engineer
Builds large-scale batch and streaming jobs.
- Explain Spark's architecture and execution model
- Transform data with DataFrames and Spark SQL
- Choose join strategies and handle skew
About the course
Distributed data processing with Apache Spark and PySpark: DataFrames, Spark SQL, joins, partitioning, caching and performance tuning, plus Structured Streaming basics. Labs use realistic data volumes.
Distributed processing with DataFrames, Spark SQL, partitioning, joins and performance tuning on realistic datasets.
Why should I attend?
Data engineers
Python developers handling large data
ETL developers modernising pipelines
Data scientists working at scale
Prerequisites
Python and SQL; basic data-processing experience
Level
Intermediate
Delivery modes, duration and fee
Online live
Instructor-led virtual classroom; weekday (full-day) or weekend (Saturday) batches
Bengaluru classroom
In-person batches at our Bengaluru training venue
On-site at your office
Private corporate batch anywhere in India, scheduled around your teams
Blended
Live sessions plus guided self-paced labs and mentor check-ins
Duration: 32 hours · 4 days
Course fee (open batch): ₹14,999
Corporate / team batches: per-participant or flat batch pricing on request, based on group size, format and customisation.
What will you learn
- Explain Spark's architecture and execution model
- Transform data with DataFrames and Spark SQL
- Choose join strategies and handle skew
- Partition and cache data effectively
- Read the Spark UI to diagnose slow jobs
- Build a simple Structured Streaming job
Skills you will gain
Processing
Performance
Operations
Tools and platforms covered
Apache SparkPySpark
Python
Databricks
Parquet
Delta Lake
JupyterCourse curriculum
M01 Spark architecture
- Driver and executors
- Lazy evaluation
- Jobs, stages, tasks
- Cluster managers
- Spark UI walkthrough
🧪 Hands-on: Explore the Spark UI
M02 DataFrames and Spark SQL
- Transformations
- Spark SQL
- UDFs vs built-ins
- Window functions
- Pandas API on Spark
🧪 Hands-on: Clean a large dataset
M03 Joins and aggregations
- Join strategies
- Broadcast joins
- Data skew
- Sort-merge vs hash joins
- Salting skewed keys
🧪 Hands-on: Fix a skewed join
M04 Performance tuning
- Partitioning
- Caching
- Adaptive query execution
- Shuffle tuning
- Memory configuration
🧪 Hands-on: Tune a slow job
M05 File formats and storage
- Parquet
- Delta Lake
- Partitioned tables
- Compaction and small files
- Z-ordering concepts
🧪 Hands-on: Write optimised tables
M06 Structured Streaming
- Streaming model
- Sources and sinks
- Watermarks
- Exactly-once semantics
- Stateful aggregations
🧪 Hands-on: Build a streaming aggregation
M07 Testing and packaging
- Unit tests with pytest
- Data assertions
- Packaging jobs
- CI basics
🧪 Hands-on: Add tests and CI to a PySpark job
M08 Capstone: optimise a Spark pipeline
- Baseline run
- Diagnose with Spark UI
- Optimise
- Report
🧪 Hands-on: Cut the runtime of a provided slow pipeline and document why
Hands-on projects
Large-scale transformation job
A tuned PySpark pipeline with before-and-after performance analysis.
Why train with us
Built around your stack
Labs and examples adapted to the tools, cloud and data your team actually uses.
Practice-first
Every module ends in a lab, and the course ends with a reviewed capstone.
Practitioner-led
Delivered by Karthick Selvam and vetted specialist trainers from our network.
Visible progress
Attendance, lab completion and assessment results shared with your L&D team.
Certificate of completion
Participants who complete the labs and capstone receive a Learning Dots certificate of completion for Apache Spark & PySpark for Data Engineers. [confirm certificate wording / sample image]
Learner reviews
“[Verified learner review for this course]”
“[Verified learner review for this course]”
Meet your trainer
Karthick Selvam
Founder · Data Engineering Corporate Trainer, Bengaluru
[Trainer bio, experience and areas of expertise]
Frequently asked questions
Where do labs run?
On Databricks or a provided Spark environment.
Can the curriculum be customised for our team?
Yes. Modules, depth, duration and labs can be tailored to your stack and goals.
What delivery formats are available?
Online live instructor-led sessions, Bengaluru classroom, or on-site at your office.
How do we enquire or book?
Use the enquiry form on this page or the Enquire button next to any batch.
What does it cost?
The open-batch fee is ₹14,999 per participant plus GST. Corporate batches are priced on group size, format and customisation.
How long is the course?
32 hours · 4 days. Corporate batches can be split into half-day sessions.
A closer look
About this course
Distributed data processing with Apache Spark and PySpark: DataFrames, Spark SQL, joins, partitioning, caching and performance tuning, plus Structured Streaming basics. Labs use realistic data volumes.
What is Apache Spark & PySpark for Data Engineers?
Distributed processing with DataFrames, Spark SQL, partitioning, joins and performance tuning on realistic datasets.
Who this course is for
Data engineers, Python developers handling large data, ETL developers modernising pipelines, Data scientists working at scale.
What you will be able to do
Explain Spark's architecture and execution model. Transform data with DataFrames and Spark SQL. Choose join strategies and handle skew. Partition and cache data effectively. Read the Spark UI to diagnose slow jobs. Build a simple Structured Streaming job.
Career paths
Roles this course supports include Big Data Engineer, Spark Developer, Data Platform Engineer. Spark remains one of the most widely used engines for large-scale data processing, and tuning skills are particularly valued.
Related courses
Data Engineering Foundations
Pipelines, warehouses, lakes and the modern data stack explained hands-on, from first principles to a working end-to-end pipeline.
Databricks & Lakehouse Engineering
Delta Lake, medallion architecture, workflows, Unity Catalog concepts and production-grade lakehouse patterns.
Data EngineeringWorkflow Orchestration with Apache Airflow
DAG design, scheduling, retries, sensors, testing and running reliable pipelines in production.
Data Engineering guides
What Is Data Engineering? A Plain-English Guide
Data engineering is the discipline of collecting, moving, transforming and storing data so analysts, applications and AI can use it reliably.
ETL vs ELT: What Is the Difference?
Two patterns for moving and transforming data, differing in where the transformation happens.
Data Lake vs Data Warehouse vs Lakehouse
Three architectures for storing analytical data.
What Is Apache Spark and When Should You Use It?
Apache Spark is an open-source engine for processing large datasets in parallel across a cluster.