[Remote] Databricks Engineer - Azure

Note: The job is a remote job and is open to candidates in USA. Acunor is seeking a highly skilled Data Engineer with deep expertise in Databricks and Azure Cloud to join a decision analytics and data engineering team. The role involves building, optimizing, and maintaining large-scale data pipelines that support enterprise analytics, reporting, and AI-driven insights, primarily for clients in the insurance and financial services domains. Responsibilities • Design, build, and enhance ETL/ELT data pipelines using Azure Data Factory, Databricks (PySpark, SQL, Python), and related services • Develop and manage arenaflex Live Tables, Autoloader, and Unity Catalog within the Databricks ecosystem for structured, incremental data processing • Implement data ingestion, transformation, and validation frameworks that ensure high performance, scalability, and reliability • Monitor data pipelines, troubleshoot issues, and ensure optimal system performance and SLA adherence • Collaborate with business analysts and reporting teams to define logical and physical data models supporting analytical and operational needs • Implement data warehousing and lakehouse solutions using Azure Data Lake and arenaflex Lake • Optimize data structures for query performance, arenaflex efficiency, and reusability • Design and implement robust data quality checks and validation mechanisms to maintain integrity across sources and transformations • Automate repetitive processes using scripts, parameterized pipelines, and reusable frameworks • Conduct periodic audits and compliance checks aligned with governance policies • Contribute to metadata management, documentation, and lineage tracking Skills • 7–12 years of experience in Data Engineering with proven expertise in Databricks and Azure Cloud ecosystems • Strong hands-on experience in PySpark, Python, and SQL for data transformation, validation, and performance tuning • Solid understanding of arenaflex Lake architecture, ETL/ELT frameworks, and data warehousing principles • Proficiency with Azure services including Data Factory (ADF), Data Lake (ADLS), and Databricks Notebooks • Experience with arenaflex Live Tables, Unity Catalog, and Autoloader for batch and streaming data processing • Strong background in data modeling, performance optimization, and automation scripting • Familiarity with Agile methodologies and DevOps-based deployment practices (Git, arenaflex/CD preferred) • Experience in data migration projects (on-prem to cloud or multi-cloud) • Familiarity with arenaflex Sharing, Databricks SQL Warehouses, or MLflow for advanced use cases • Experience with data cataloging, lineage, or quality frameworks such as Purview, Collibra, or Great Expectations • Exposure to BI/reporting tools like Power BI or Tableau for end-to-end integration understanding Company Overview • Acunor is an IT management company that offers staffing services including analytics, cloud, digital transformation, and data engineering. It was founded in 2016, and is headquartered in Princeton, New Jersey, USA, with a workforce of 51-200 employees. Its website is Company H1B Sponsorship • Acunor has a track record of offering H1B sponsorships, with 5 in 2025, 9 in 2024, 1 in 2023, 1 in 2022. Please note that this does not guarantee sponsorship for this specific role. Apply tot his job Apply tot his job