Azure Data Factory (ADF) Engineer
Summary
Title: | Azure Data Factory (ADF) Engineer |
---|---|
ID: | 10272 |
Department: | Information Technology |
Location : | Washington DC |
Description
Clearance: Active Secret of above
Requirements Gathering
- Collaborate with business analysts, data architects, and stakeholders to gather requirements and ensure data integrity throughout the migration lifecycle.
- Ability to create Data mapping documents (Source to Target) by understanding business rules.
Data Understanding/Profiling
- Perform data profiling, cleansing, and quality checks to ensure accuracy and completeness of migrated data.
- Analyze and map source Oracle database schemas to target Dataverse tables/entities, defining transformation and cleansing logic as required.
Technical Requirements for Development
- Write and optimize SQL queries, stored procedures, and scripts for Oracle databases to support data migration and validation.
- Implement data transformation processes to ensure data compatibility with Dataverse data types, relationships, and constraints.
- Experience with Dataverse data types and CRUD operations
- Experience configuring data pipelines in ADF (to extract, transform, load). Experience using .csv files as sources of data for the ADF pipelines
- Experience using Oracle CDCs as sources of data for the ADF pipelines. Experience logging errors (audit framework) from an ADF pipeline to a Dataverse table
- Experience pushing data to Dataverse table(s) including necessary transformations to match target field types (i.e. Dataverse lookups, choices, date/time, etc.)
- Experience configuring setting for pipelines to improve concurrency/performance (i.e., ADF cores, memory, etc.)
- Experience with additional Azure services (e.g., Logic Apps, Data Lake, Azure Functions, Azure SQL).
- Experience/exposure in python/shell scripting is a plus
Data Validation Post migration
- Ability to generate a reconcile report of the migrated data using SQL or scripts
Code Migration
- Must be familiar with Git or source code repository & CI/CD process.