Azure Data Factory (ADF) Engineer
Summary
| Title: | Azure Data Factory (ADF) Engineer |
|---|---|
| ID: | 10272 |
| Department: | Information Technology |
| Location : | Remote |
Description
Clearance: Active Secret or above required
Role Description:
- Collaborate with business analysts, data architects, and stakeholders to gather requirements and ensure data integrity throughout the migration lifecycle.
- Ability to create Data mapping documents (Source to Target) by understanding business rules.
- Perform data profiling, cleansing, and quality checks to ensure accuracy and completeness of migrated data.
- Analyze and map source Oracle database schemas to target Dataverse tables/entities, defining transformation and cleansing logic as required.
- Write and optimize SQL queries, stored procedures, & scripts for Oracle databases to support data migration
- Implement data transformation processes to ensure data compatibility with Dataverse data types, relationships, and constraints.
- Experience with Dataverse data types and CRUD operations
- Experience configuring data pipelines in ADF (to extract, transform, load). Experience using .csv files as sources of data for the ADF pipelines
- Experience using Oracle CDCs as sources of data for the ADF pipelines. Experience logging errors (audit framework) from an ADF pipeline to a Dataverse table
- Experience pushing data to Dataverse table(s) including necessary transformations to match target field types (i.e. Dataverse lookups, choices, date/time, etc.)
- Experience configuring setting for pipelines to improve concurrency/performance (i.e., ADF cores, memory, etc.)
- Experience with additional Azure services (e.g., Logic Apps, Data Lake, Azure Functions, Azure SQL).
- Experience/exposure in python/shell scripting is a plus
- Ability to generate a reconcile report of the migrated data using SQL or scripts
- Must be familiar with Git or source code repository & CI/CD process.
- Experience developing ETL packages with SSIS
- Experience creating, maintaining DB objects in SQL Server
- Experience with migrating data from .CSV exports
Mandatory Skills
- Data Migration Expert: Advanced
- Data Migration: Advanced
- Microsoft Azure: Intermediate
- Oracle: Advanced
- SQL: Advanced
Preferred Skills
- MICROSOFT AZURE SQL DB: Intermediate
- MS Azure Data Factory: Intermediate
- MS Azure Data Lake Analytics: Intermediate
- MS Azure Functions: Intermediate
- MS Azure Logic Apps: Intermediate

