Course Outline
Databricks Platform and Lakehouse Fundamentals
- Databricks Lakehouse architecture and core components.
- Organizing workspaces and catalogs.
Databricks Workspace and Notebooks
- Workspace navigation and notebook-based development.
- Structuring code into reusable notebooks.
Apache Spark Architecture and Execution
- Spark runtime architecture and execution model.
- Lazy evaluation and the job DAG (Directed Acyclic Graph).
PySpark DataFrames and the DataFrame API
- DataFrame abstractions and schemas.
- Core DataFrame operations and column expressions.
Translating SQL to PySpark DataFrames
- Converting core SQL clauses into DataFrame operations.
- Implementing window functions and aggregations in PySpark.
Reading and Writing Data in Databricks
- Reading from common file formats and database sources.
- Writing and partitioning data within the Lakehouse.
Delta Lake and Table Management
- Delta tables and ACID transactions.
- Time travel and schema evolution features.
Data Cleaning and Transformation Patterns
- Data cleaning techniques and type conversion.
- Building reusable transformation logic.
User-Defined Functions and Modular Code
- Python UDFs (User-Defined Functions) and pandas UDFs.
- Modularizing procedural logic into functions.
Performance Tuning and Optimization
- Partitioning and caching strategies.
- Diagnosing bottlenecks using the Spark UI.
Structured Streaming Fundamentals
- Differences between batch and streaming processing models.
- Streaming DataFrames and basic aggregations.
Databricks Jobs and Workflow Orchestration
- Scheduling notebooks as jobs and tasks.
- Constructing multi-step workflows with dependencies.
Unity Catalog and Data Governance
- Unity Catalog architecture and namespaces.
- Access control and data lineage management.
Testing, Debugging, and Production Practices
- Unit testing PySpark logic.
- Debugging techniques and code quality standards.
End-to-End Financial Services Use Cases
- Developing an end-to-end banking ETL pipeline.
- Translating legacy SQL processes to PySpark.
Migrating SQL Workloads to PySpark
- Migration strategy and planning patterns.
- Incremental conversion of SQL workflows to PySpark.
Requirements
- Experience with Python programming, specifically functions and data types.
- Understanding of SQL, including joins, aggregations, and subqueries.
- No prior experience with Databricks or PySpark is necessary.
Audience
- Data engineers, data analysts, and other data professionals.
- Teams seeking to migrate existing SQL-based workflows to Databricks and PySpark.
Testimonials (1)
I liked that it was practical. Loved to apply the theoretical knowledge with practical examples.
