onlincourse

Introduction to Data Engineering with Microsoft Azure 2

Further your knowledge of Microsoft Azure services and improve your data engineering skills with this online course from Microsoft.

Duration

6 weeks

Weekly study

4 hours

100% online

How it works

Unlimited subscription

Learn more

Prepare for the DP-203: Data Engineering on Microsoft Azure exam

This course has been created in partnership with Microsoft.

Building on your learning from Introduction to Data Engineering with Microsoft Azure 1, this course will develop your understanding of data engineering processes in Microsoft Azure, further preparing you to take the DP-203 exam and kickstart your career in data engineering.

Explore data services within Microsoft Azure

Using Azure data services and tools, you’ll be able to implement, develop, and optimise data storage, processing and security operations within your organisation.

You’ll be introduced to tools including Azure Synapse, Databricks and Azure Data Lake Storage, learning how each can improve and streamline your processes.

Design hybrid transactional and analytical processing (HTAP) patterns

As businesses continue to move to digital processes, they recognise the value of making faster, well-informed decisions and the impact this can have on gaining a competitive advantage.

You’ll be guided through HTAP architecture and learn how to design HTAP using Azure Synapse Analytics.

With this knowledge, you’ll be able to run analytics in near-real-time, giving you the ability to respond to opportunities at speed.

Discover data operations in Azure Databricks

Azure Databricks, a cloud-based big data and machine learning platform, empowers developers by simplifying enterprise-grade data application production.

You’ll identify the advantages of Azure Databricks over other Big Data platforms, and learn how to spend more time building apps and less time managing infrastructure.

You’ll finish this course understanding how Microsoft Azure can be used to optimise data engineering operations. Having completed both courses, you’ll be equipped to take the DP-203 exam and develop a career as a data professional.

Week 1
Work with Hybrid Transactional and Analytical Processing Solutions using Azure Synapse Analytics
- Plan hybrid transactional and analytical processing using Azure Synapse Analytics
  In this activity, you will learn about planning hybrid transactional and analytical processing using Azure Synapse Analytics.
- Configure Azure Synapse Link with Azure Cosmos DB
  During this week, you will learn how to configure Azure Synapse link with Azure Cosmos DB.
- Query Azure Cosmos DB with Apache Spark for Azure Synapse Analytics
  In this activity, you will learn how to query Azure Cosmos DB with Apache Spark for Azure Synapse Analytics.
- Query Azure Cosmos DB with SQL Serverless for Azure Synapse Analytics
  In this activity, you will learn about querying Azure Cosmos DB with SQL Serverless for Azure Synapse Analytics.
Week 2
Data engineering with Azure Databricks Part 1
- Describe Azure Databricks
  In this activity, you will describe Azure Databricks.
- Spark architecture fundamentals
  In this activity, you will learn about spark architecture fundamentals.
- Read and write data in Azure Databricks
  During this week, you will learn about reading and writing data in Azure Databricks.
- Work with DataFrames in Azure Databricks
  During this week, you will learn how to work with DataFrames in Azure Databricks.
- Work with DataFrames columns in Azure Databricks
  During this week, you will learn how to work with DataFrames columns in Azure Databricks.
Week 3
Data engineering with Azure Databricks Part 2
- Describe lazy evaluation and other performance features in Azure Databricks
  During this week, you will learn how to describe lazy evaluation and other performance features in Azure Databricks.
- Work with DataFrames advanced methods in Azure Databricks
  In this activity, you will learn how to work with DataFrames advanced methods in Azure Databricks.
- Describe platform architecture, security, and data protection in Azure Databricks
  During this week, you will learn how to describe platform architecture, security, and data protection in Azure Databricks.
- Build and query a Delta Lake
  Learn how to use Delta Lake to create, append, and upsert data to Apache Spark tables, taking advantage of built-in reliability and optimizations.
- Process streaming data with Azure Databricks structured streaming
  Learn how Structured Streaming helps you process streaming data in real time, and how you can aggregate data over windows of time.
Week 4
Data engineering with Azure Databricks Part 3
- Describe Azure Databricks Delta Lake architecture
  Use Delta Lakes as an optimization layer on top of blob storage to ensure reliability and low latency within unified Streaming and batch data pipelines.
- Create production workloads on Azure Databricks with Azure Data Factory
  Azure Data Factory helps you create workflows that orchestrate data movement and transformation at scale. Integrate Azure Databricks into your production pipelines by calling notebooks and libraries.
- Implement CI/CD with Azure DevOps
  CI/CID isn't just for developers. Learn how to put Azure Databricks notebooks under version control in an Azure DevOps repo and build deployment pipelines to manage your release process.
- Integrate Azure Databricks with Azure Synapse
  Azure Databricks is just one of many powerful data services in Azure. Learn how to integrate with Azure Synapse Analytics as part of your data architecture.
- Describe Azure Databricks best practices
  Learn best practices for workspace administration, security, tools, integration, databricks runtime, HA/DR, and clusters in Azure Databricks.
Week 5
Large-Scale Data Processing with Azure Data Lake Storage Gen2
- Introduction to Azure Data Lake storage
  Learn how Azure Data Lake Storage provides a cloud storage service that is highly available, secure, durable, scalable, and redundant and brings new efficiencies to processing big data analytics workloads.
- Upload data to Azure Data Lake Storage
  Learn various ways to upload data to Data Lake Storage Gen 2. Upload data through the Azure portal, Azure Storage Explorer, or .NET. Or copy the data in Azure Data Factory.
- Secure your Azure Storage account
  Learn how Azure Storage provides multilayered security to protect your data. Find out how to use access keys, to secure networks, and to use Advanced Threat Protection to proactively monitor your system.
Week 6
Implement a Data Streaming Solution with Azure Streaming Analytics
- Work with data streams by using Azure Stream Analytics
  Explore how Azure Stream Analytics integrates with your applications or Internet of Things (IoT) devices to gain insights with real-time streaming data. Learn how to consume and analyze data streams and derive actionable results.
- Enable reliable messaging for Big Data applications using Azure Event Hubs
  Connect sending and receiving applications with Event Hubs so you can handle extremely high loads without losing data.
- Ingest data streams with Azure Stream Analytics
  Learn how to create Azure Stream Analytics jobs to process input data, transform it with a query, and return results.