Databricks, Data Factory Role Big Data Engineer Responsibilities Implement Data integration and Data Warehouse based solutions using Big Data Technologies. Should be highly proficient in the use of Big Data / Open Source Technologies and standard techniques of Data Integration, Data Manipulation with hands-on contribution. Should be able to develop cost efficient and performant data pipelines in the cloud platform In-depth understanding of modern big data technology, including Data modeling and machine learning skills Knowledge of real time data streaming and aggregation architectural patterns and practice Essential Skills: 3+ Years hands on knowledge on SQL as well as SQL/NoSQL databases Proficient in programming languages such as Python and PySpark/Scala/Java Experience with Apache Spark in On-prem / Cloud (Databricks / EMR / DataProc HDInsight) Working knowledge of API development Experience in building data pipelines in ETL/ELT methods Experience with integration of data from multiple data sources Experience in one of the NoSQL databases, such as HBase, Cassandra, MongoDB Knowledge of real time/streaming/event based data engineering frameworks, such as Flume, Pub-Sub, Kinesis, Eventhub Experience with various messaging systems, such as Kafka or RabbitMQ Experience with building stream-processing systems, using solutions such as Storm or Spark-Streaming Working knowledge and experience in Big data services in one of the Cloud Provider will be good (AWS or Azure or GCP) The Role offers Opportunity to join a global team to do meaningful work that contributes to global strategy and individual development An outstanding opportunity to re-imagine, redesign, and apply technology to add value to the business and operations Gives an opportunity to showcase candidates’ strong analytical skills and problem-solving ability Grow into a Technical architect role over a period
Want to receive frequent updates by email? Subscribe to our automatic job service!