DE303: Real-Time & API-Based Data Integration

by

SFIA Reference: INCA Level 4/5

Data Engineer (Grade 3)

Competency Description

Develops secure, efficient, and responsive integration methods for streaming or API-driven data.

Learning Outcomes

  • Integrate with REST APIs securely and efficiently.
  • Implement streaming and event-driven ingestion using Kafka/Event Hub.
  • Apply Change Data Capture (CDC) to near real-time workloads.

Evidence Requirements

Level 4:

  • Developed an API-based pipeline that integrates external data.
  • Built or maintained a working CDC implementation.

Level 5:

  • Designed or refactored an event-driven integration framework.
  • Reduced latency or improved reliability of streaming integrations.

Suggested Learning Activities

  • Internal challenge: CDC setup using Databricks Auto Loader.
  • Create an async API integration for Silver ingestion.
  • Write a comparison of polling vs streaming for a use case.

Recommended Resources

  • Microsoft Learn: Azure Event-Driven Architectures
  • KnowHow: CDC Patterns
  • API Security Guidelines (Internal)

Leave a Comment