Job Openings >> Azure Data Factory (ADF) Engineer
Azure Data Factory (ADF) Engineer
Summary
Title:Azure Data Factory (ADF) Engineer
ID:10272
Department:Information Technology
Location :Washington DC
Description

Clearance: Active Secret of above 

Requirements Gathering 

  • Collaborate with business analysts, data architects, and stakeholders to gather requirements and ensure data integrity throughout the migration lifecycle.
  • Ability to create Data mapping documents (Source to Target) by understanding business rules.

Data Understanding/Profiling

  • Perform data profiling, cleansing, and quality checks to ensure accuracy and completeness of migrated data.
  • Analyze and map source Oracle database schemas to target Dataverse tables/entities, defining transformation and cleansing logic as required.

Technical Requirements for Development

  • Write and optimize SQL queries, stored procedures, and scripts for Oracle databases to support data migration and validation.
  • Implement data transformation processes to ensure data compatibility with Dataverse data types, relationships, and constraints.
  • Experience with Dataverse data types and CRUD operations
  • Experience configuring data pipelines in ADF (to extract, transform, load). Experience using .csv files as sources of data for the ADF pipelines
  • Experience using Oracle CDCs as sources of data for the ADF pipelines. Experience logging errors (audit framework) from an ADF pipeline to a Dataverse table
  • Experience pushing data to Dataverse table(s) including necessary transformations to match target field types (i.e. Dataverse lookups, choices, date/time, etc.)
  • Experience configuring setting for pipelines to improve concurrency/performance (i.e., ADF cores, memory, etc.)
  • Experience with additional Azure services (e.g., Logic Apps, Data Lake, Azure Functions, Azure SQL).
  • Experience/exposure in python/shell scripting is a plus

Data Validation Post migration

  • Ability to generate a reconcile report of the migrated data using SQL or scripts

Code Migration

  • Must be familiar with Git or source code repository & CI/CD process.
ApplicantStack powered by Swipeclock