This role will be working with high velocity data from some of the world's largest retailers, supporting ground-breaking BI and Artificial Intelligence use cases
Key Responsibilities include
*Design and implement pipelines to ingest data into and consume data from the data lake
*Source and transform data from historical and streaming data sources
*Implement automated monitoring and alerting on pipelines to ensure consistent processing of data
*Design systems to allow real time analytics on high velocity data
*Proactively ensure optimum performance by revisiting design choices to check they are still fit for purpose
*Collaborate with Data Scientists and BI Developers to ensure the wider team's objectives are being met by data lake design
*Support wider team in consuming data from the data lake
*Be able to provide technical leadership in the use and design of big data technologies
In order to be successful in the role candidates need to have
Experience utilising the following technologies, preferably in the Azure platform:
*Azure Data Lake
*Azure Data Factory
*Databricks
*Python/PySpark or Scala
*Data pipeline design, and optimisation
*Optimising data lake file storage
*Optimising data lakes for real time advanced analytics
*Time management, planning and organisation
If you would like to find out more, then please send across an up-to-date CV at your earliest convenience
