About the Position

This role will be responsible for designing, developing, implementing, and maintaining scalable, secure, and reliable data pipelines and data products within the customer’s modern cloud-based data environment. The Data Engineer will work closely with Data Architects, Data Scientists, business stakeholders, application teams, and analytics teams to ingest, integrate, transform, govern, and deliver trusted enterprise data for analytics, reporting, operational, and artificial intelligence use cases.

The role will leverage technologies including Microsoft Azure, Azure SQL, Databricks, Delta Lake, Unity Catalog, PySpark, Python, SQL and Databricks AI/BI, aligning directly with the customer’s technology landscape.

‍

‍

What you bring

• Deep hands-on expertise with SQL and Python, including practical experience developing production data pipelines.
• Strong hands-on experience with Databricks, Apache Spark, PySpark, Delta Lake, and Unity Catalog.
• Experience developing enterprise-scale data solutions on Microsoft Azure.
• Hands-on experience with data integration such as Lakeflow connect for data ingestion and Spark Declarative Pipelines.
• Strong knowledge of relational databases, data warehouses, data lakes, and Lakehouse architectures.
• Experience working with database technologies such as Azure SQL, SQL Server, Databricks SQL, and NoSQL technologies.
• Experience developing ETL/ELT pipelines for structured, semi-structured, and unstructured data.
• Strong understanding of data modeling concepts, including dimensional, relational, normalized, denormalized, and analytical models.
• Experience integrating data using REST APIs, JDBC/ODBC, files, messaging, and cloud-based integration patterns.
• Experience implementing data quality, data lineage, metadata management, data governance, and security controls.
• Understanding of cloud identity, role-based access control, secrets management, encryption, and secure data access patterns.
• Experience with Git, CI/CD, automated deployment, testing, and DevOps practices for data solutions.
• Ability to diagnose complex data processing, performance, and integration problems.
• Ability to manage multiple assignments simultaneously, including unstructured assignments requiring autonomy and independent judgment.
• Ability to clearly articulate technical designs, trade-offs, risks, and recommendations using sound logic and supporting evidence.
• Strong interpersonal and communication skills with the ability to collaborate with architects, developers, analysts, data scientists, cybersecurity teams, and business stakeholders.
• Ability to effectively represent internal and external constituencies.
• Experience implementing enterprise Lakehouse or medallion architectures using Databricks.
• Experience with Databricks serverless compute, workflows/jobs, Delta Live Tables or Lakeflow, Unity Catalog, Databricks SQL, and related platform capabilities.
• Experience supporting datasets and engineering pipelines for machine learning, Generative AI, RAG, and AI applications.
• Familiarity with Informatica Cloud Data Governance, Microsoft Purview or comparable metadata and data governance platforms.
• Experience with infrastructure-as-code technologies and automated cloud provisioning.
• Experience working in regulated, government, financial, audit, or other highly governed environments.
• Familiarity with agile methods and processes and experience supporting agile delivery teams.
• Professional certification in Databricks Data Engineering, Microsoft Azure, cloud data engineering, or related technologies

Nice to have

• Experience in regulated or confidential data environments.

Working with us

• You will work inside a secure, private cloud environment with highly confidential data.
• You must be authorized to work in the United States and able to pass a background check.
• We are looking for people who can start within a few weeks.

‍

What You'll Do

• Design, develop, deploy, maintain, and optimize scalable data pipelines supporting batch and near-real-time data processing.
• Collaborate with customer’s Divisions and Offices to understand business and technical requirements and translate them into reliable data engineering solutions.
• Work closely with Data Architects to implement data solutions consistent with customer’s enterprise data architecture, standards, patterns, and strategic roadmaps.
• Develop all aspects of ETL and ELT processes necessary to ingest, validate, cleanse, transform, enrich, store, and deliver data for operational, reporting, analytics, and AI use cases.
• Build and maintain data pipelines using Databricks, Python, PySpark, and SQL, as appropriate.
• Design and implement scalable data processing solutions using Databricks, Apache Spark, Delta Lake, and Azure cloud services.
• Develop and maintain curated enterprise data products and reusable datasets for consumption by reporting, analytics, data science, application, and AI workloads.
• Implement data models and physical database structures based on architectural and analytical requirements.
• Build pipelines to integrate data from enterprise applications, relational databases, APIs, flat files, cloud storage, SaaS platforms, and external data sources.
• Implement reusable patterns and frameworks for data ingestion, data transformation, change data capture, schema evolution, orchestration, and data delivery.
• Develop automated data validation, reconciliation, and data quality processes to ensure accuracy, completeness, consistency, and reliability.
• Implement monitoring, logging, alerting, error handling, and recoverability capabilities for production data pipelines.
• Troubleshoot and resolve performance, reliability, data quality, and integration issues across the data platform.
• Optimize storage, Spark workloads, SQL queries, pipeline execution, and compute utilization to improve platform performance and cost efficiency.
• Support the rationalization and consolidation of data sources to reduce redundant data movement, replication, and organizational data silos.
• Implement enterprise data governance capabilities addressing metadata, lineage, access control, security, data classification, and lifecycle management.
• Implement and administer appropriate data access controls and governance patterns using Databricks Unity Catalog and related enterprise security capabilities.
• Develop and maintain technical documentation covering data sources, mappings, transformations, schemas, lineage, dependencies, pipeline design, and operational support procedures.
• Partner with Data Scientists to develop data pipelines, features, and datasets required for machine learning, advanced analytics, Generative AI, and AI agent workloads.
• Partner with reporting and analytics teams to provide curated and optimized datasets for Power BI dashboards and other analytical applications.
• Participate in solution design and code reviews to ensure engineering implementations conform to customer architecture, security, development, and operational standards.
• Support CI/CD, source control, automated testing, infrastructure automation, and DevOps practices for data engineering solutions.
• Research emerging data engineering technologies and recommend improvements to data platform capabilities and engineering practices.