We are seeking a highly skilled and motivated Data Engineer with experience in Databricks, Azure Data Explorer (ADX), Azure IoT Hub, real-time data streams, Kusto Query Language (KQL) and Microsoft Fabric. The successful candidate will play a key role in building and maintaining scalable data pipelines, working on real-time streaming solutions, and ensuring efficient data storage and querying across our cloud platform.

Key Responsibilities:

Design, develop, and maintain data pipelines that ingest, process, and transform data from various real-time sources (IoT devices, sensors, etc.) into Databricks delta lake.
Integrate and manage real-time streaming data from Azure IoT Hub into the data platform using Azure Stream Analytics or other real-time data processing services.
Build and manage real-time data pipelines for use cases requiring immediate analytics and processing.
Write complex and optimized Kusto Query Language (KQL) scripts for querying and analysing data in ADX.
Leverage Azure Databricks for data transformation, machine learning, and advanced analytics workflows.
Collaborate with cross-functional teams, including data scientists, software engineers, and product managers, to deliver data-driven solutions.
Ensure data quality, reliability, and consistency by implementing appropriate validation and monitoring tools.
Implement and manage Microsoft Fabric solutions to create unified and scalable data environments for efficient data exploration and analysis.
Optimize data storage, retrieval, and processing, ensuring that all pipelines are efficient, scalable, and cost-effective.
Work with Azure DevOps for version control, CI/CD pipeline setup, and automation for deployment of data solutions.

Required Skills and Qualifications:

Bachelor’s degree in computer science, Data Engineering.
6+ years of experience as a Data Engineer working with cloud-based data platforms.
Strong experience with Python, Azure Data Explorer (ADX) and proficiency in Kusto Query Language (KQL).
Experience in setting up and managing Azure IoT Hub for real-time streaming data ingestion.
Solid understanding of real-time streaming data architectures and tools like Azure Stream Analytics, Event Hubs, or Apache Kafka.
Hands-on experience with Azure Databricks for building scalable data transformation pipelines and advanced analytics workflows.
Familiarity with Microsoft Fabric for data integration, pipeline management, and optimization and real-time dashboards.
Strong knowledge of SQL, Python, data modelling, and data warehousing principles.
Familiarity with Azure Synapse Analytics, Azure Data Factory, and other Azure data services.
Experience with CI/CD pipelines and Azure DevOps for version control and automation.
Excellent problem-solving skills and attention to detail.
Strong communication and collaboration skills.

Preferred Skills:

Experience with machine learning models and AI integrations in Databricks.
Familiarity with Scala, or R for data processing.
Knowledge of data security best practices and experience implementing data governance policies.
Certifications in Azure Data Engineering or related fields are a plus.

If you are passionate about working with cutting-edge cloud data technologies and thrive in fast-paced environments, we encourage you to apply and join our dynamic team!

Remote Scouter