Senior Big Data Engineer ETL, Spark & Cloud

Keywords : Spark, azure data factory, airflow,ETL, Python, Dutch native speaker

Context:

We wish to build up customer knowledge at our client in order to further improve our service. Both outbound in intelligent interactions with our customers, as well as business in a more targeted use of investment resources to us better match supply to demand

 

Description

You contribute to the expansion of our data pipelines within our big data platform. You will join an agile team with the mentality of a startup. In addition to the technical development of pipelines, you can also understand how to extract value from these data. You can first gain explorative insights and build production-worthy models. Open constructive content and technical adjustment is in your nature. We therefore expect more than just technical implementation.

You have experience with technical integrations and you can think / build in this (API, event based, pipeline sheduling and transaction handling…). You think along with you functionally. You are available at least 4/5 to be on-site in Mechelen team.

 

Profile:

  • Minimum 10 years of professional experience in IT
  • Experience in Design, build and deploy a set of batch data applications in spark / azure batch
  • Knowledge of data science and machine learning algorithms to ensure that your data deliveries are ideal for building models.
  • Experience in workflow mgmt and job scheduling of data loads (airflow, azure data factory)
  • Experience in real time data streaming and processing is a plus (azure functions / event hubs / streaming analytics)
  • Experience in cloud Azure is a plus. General cloud experience is a must
  • Knowledge of ETL for large data sets on distributed systems
  • Advanced knowledge in programming languages ​​(python, sql, hcl, …) and in exposing data (API, sql server, event streams, ..)
  • Experience with data exploration: Jupyter notebooks, dashboarding … and getting value from data.
  • We expect that you can identify business opportunities or service improvements through your own analyzes and insights.
  • You have enough courage to bring forward new assumptions. You have demonstrable experience in business rules and to be able to weigh complexities • Experience with ETLs with aggregations, deduplication and linking to identities
  • Experience in build automation and devops CI / CD tooling: git, Terraform / ARM templates, Circle CI
  • Experience with web environments that use big data intel / recommendations. Javascript, Java, Node Js, API with data exposition

 

For more information: Contact us now with your CV