Description
We are seeking a highly skilled and motivated Data Engineer with experience in Databricks, Azure Data Explorer (ADX), Azure IoT Hub, real-time data streams, Kusto Query Language (KQL) and Microsoft Fabric. The successful candidate will play a key role in building and maintaining scalable data pipelines, working on real-time streaming solutions, and ensuring efficient data storage and querying across our cloud platform.
Key Responsibilities:
- Design, develop, and maintain data pipelines that ingest, process, and transform data from various real-time sources (IoT devices, sensors, etc.) into Databricks delta lake.
- Integrate and manage real-time streaming data from Azure IoT Hub into the data platform using Azure Stream Analytics or other real-time data processing services.
- Build and manage real-time data pipelines for use cases requiring immediate analytics and processing.
- Write complex and optimized Kusto Query Language (KQL) scripts for querying and analysing data in ADX.
- Leverage Azure Databricks for data transformation, machine learning, and advanced analytics workflows.
- Collaborate with cross-functional teams, including data scientists, software engineers, and product managers, to deliver data-driven solutions.
- Ensure data quality, reliability, and consistency by implementing appropriate validation and monitoring tools.
- Implement and manage Microsoft Fabric solutions to create unified and scalable data environments for efficient data exploration and analysis.
- Optimize data storage, retrieval, and processing, ensuring that all pipelines are efficient, scalable, and cost-effective.
- Work with Azure DevOps for version control, CI/CD pipeline setup, and automation for deployment of data solutions.
Required Skills and Qualifications:
- Bachelor’s degree in computer science, Data Engineering.
- 6+ years of experience as a Data Engineer working with cloud-based data platforms.
- Strong experience with Python, Azure Data Explorer (ADX) and proficiency in Kusto Query Language (KQL).
- Experience in setting up and managing Azure IoT Hub for real-time streaming data ingestion.
- Solid understanding of real-time streaming data architectures and tools like Azure Stream Analytics, Event Hubs, or Apache Kafka.
- Hands-on experience with Azure Databricks for building scalable data transformation pipelines and advanced analytics workflows.
- Familiarity with Microsoft Fabric for data integration, pipeline management, and optimization and real-time dashboards.
- Strong knowledge of SQL, Python, data modelling, and data warehousing principles.
- Familiarity with Azure Synapse Analytics, Azure Data Factory, and other Azure data services.
- Experience with CI/CD pipelines and Azure DevOps for version control and automation.
- Excellent problem-solving skills and attention to detail.
- Strong communication and collaboration skills.
Preferred Skills:
- Experience with machine learning models and AI integrations in Databricks.
- Familiarity with Scala, or R for data processing.
- Knowledge of data security best practices and experience implementing data governance policies.
- Certifications in Azure Data Engineering or related fields are a plus.
If you are passionate about working with cutting-edge cloud data technologies and thrive in fast-paced environments, we encourage you to apply and join our dynamic team!
