Back to job search

Big Data Engineer

  • Location:


  • Job type:


  • Salary:

    Up to £0.00 per annum + (NEG)-Excellent Package on offer

  • Contact:

    Lyle Ritchie

  • Contact email:

  • Job ref:


  • Published:

    24 days ago

  • Expiry date:


  • Client:


I am currently looking for a Big Data Engineer to join a Software Services Company in Glasgow. Working within the Data Science team you will help design, implement and optimise processes to ingest data from a variety of sources including IoT devices using big data technologies

This role will be working with high velocity data from some of the world's largest retailers, supporting ground-breaking BI and Artificial Intelligence use cases

Key Responsibilities include

*Design and implement pipelines to ingest data into and consume data from the data lake
*Source and transform data from historical and streaming data sources
*Implement automated monitoring and alerting on pipelines to ensure consistent processing of data
*Design systems to allow real time analytics on high velocity data
*Proactively ensure optimum performance by revisiting design choices to check they are still fit for purpose
*Collaborate with Data Scientists and BI Developers to ensure the wider team's objectives are being met by data lake design
*Support wider team in consuming data from the data lake
*Be able to provide technical leadership in the use and design of big data technologies

In order to be successful in the role candidates need to have

Experience utilising the following technologies, preferably in the Azure platform:

*Azure Data Lake
*Azure Data Factory
*Python/PySpark or Scala
*Data pipeline design, and optimisation
*Optimising data lake file storage
*Optimising data lakes for real time advanced analytics
*Time management, planning and organisation

If you would like to find out more, then please send across an up-to-date CV at your earliest convenience