Data Engineer - DataOps

As a Data Engineer - DataOps, you are responsible for creating and maintaining software development lifecycle activities pertaining to the business applications. You will design, code, test, debug, and document existing and new enhancements.

Responsibilities: 

• Partner with data engineering, and business teams to set KPIs, SLAs, and SLIs for the quality and availability of data.
• Validate the data pipeline executions to ensure that they are completed within the SLA’s.
• Troubleshoot issues related to data pipelines, data integrations, bug fixes, performance bottlenecks.
• Work with multiple cross functional teams like Data Engineering, Business Analytics, IT and Vendor Support to resolve any incidents / problems.
• Continuously monitor and alert for any anomalies in the data.
• Implement role-based security, including AD integration, security policies, and auditing across the data platform environment.
• Test data to make sure it matches business logic and meets basic operational thresholds.
• Work with Data Scientists to support the AI / ML model development environments and operationalization of the models.
• Closely monitor and optimize the data platform infrastructure cost.
• On-call support for P1 issues on weekends / holidays on rotation basis.
• A desire to work in a collaborative, intellectually curious environment.

Qualifications:

• A bachelor's degree in computer science or equivalent
• 5+ years of experience as a Data Engineer,
• Experience with Azure big data tools: Azure Databricks, Azure Synapse, HDInsight, ADLS
• Experience with relational SQL and NoSQL databases.
• Excellent problem solving and analytic skills associated with working on structured and unstructured datasets using Azure bigdata tools.
• Experience with data pipeline and workflow management tools: ADF and Logic Apps
• Experience with Azure cloud services: VM, Azure Databricks, Azure SQL DB, Azure Synapse
• Experience with stream-processing systems: Azure Streams
• Experience with scripting languages Python. and have strong knowledge in developing Python notebooks.