Our data engineering team extends and maintain the various data pipelines that are at the heart of our business.
We are a data-driven company which collects and processes more than 500GB of raw data daily. We leverage big data technologies such as Spark on AWS EMR to crunch these volumes of data and make it queryable.
In this role you will ensure uptime, quality, and build upon our mission-critical Spark and Scala/Python frameworks that turns the HTML of millions of retail web pages into structured JSON, CSV & ORC formats.
In order to do this, you’ll need to jump head first into this state of the art data lake designed and implemented by Belgium’s top data engineering firm DataMinded.
Our data engineers are responsible for the following topics:
- DataLake expansion & maintenance.
- Required skills: Spark, Scala, AWS (EMR, Lambda, Elastic Search, Athena, …) , Apache Airflow.
- Infrastructure setup for data processing pipelines.
- Required skills: EMR, Spark, Airflow, Automation, Docker, …
- Design and build big data processing architectures to build further use-cases (such as AI and machine learning) upon our data.
- Required skills: AWS solutions/architecture design, Python, cost-awareness.
- Data Mart/Warehouse design to make our data more accessible for BI tools, marketing and analytics.
- Required skills: SQL database/data warehouse design, Python, data modeling.
On top of all this you’ll make sure that Daltix stays competitive in terms of data processing by using the latest & most suitable data processing technologies throughout our stack.
About the stack
Daltix is using big data technologies such as Spark, Airflow, Amazon Athena (Presto), Elastic Search & Snowflake to cope with the big amounts of data that it has to process & make accessible for analytics & the data science team on a daily basis. This is not an easy task as the volume of the data grows on a daily basis.
Being able to make huge sets of data easily analyzable & available for different use-cases one of our main challenges here as well as building tools to monitor & guard the quality of the data.
- 3 years of relevant experience & exposure to Python programming.
- Have a background in computer engineering (or relevant computer engineering degree).
- Have experience building on top of Amazon Web Services.
- Good understanding of databases/data warehouses/data lakes.
- Highly proficient in spoken and written English.
- You are passionate about data engineering and you can demonstrate a hands on experience with data analysis tools.
Ideally you also…
- Have 3 years of programming experience in Python.
- Experience with big data technologies (such as ElasticSearch, Spark, Hadoop, Airflow, Cassandra).
- Strong AWS architecture skills.
- Deep understanding of databases/data warehouses/data lakes.
- Have a deep understanding of cloud possibilities and limitations in the areas of distributed systems, load balancing and networking, massive data storage, and security.
- Get energy from working in a highly complex and challenging startup environment with a high tech product.
What can Daltix offer you?
Daltix’ offers a competitive wage (including various benefits etc) and a young, dynamic and international (we have offices in Belgium and Portugal) atmosphere to work in.
You will also receive the possibility to work from home if you prefer (even if you live in Lisbon).
When you start working at Daltix, you will get a deep dive experience. You learn all you need to know about us, our journey, your future colleagues, the tools we work with, etc.
Going beyond, is coded in our company DNA. As soon as you start working, we expect a hands-on approach, with an entrepreneurial mentality.
You will also be able to participate in relevant trainings to stay at the top of this field.
Besides developing your technical skills you will also have the opportunity to grow into the following skill sets:
- Technical/architectural lead.
- SW project management.
- Team leading & coaching.