{"id":15118,"date":"2022-10-04T14:58:37","date_gmt":"2022-10-04T14:58:37","guid":{"rendered":"https:\/\/www.daltix.com\/?p=15118"},"modified":"2022-11-15T12:29:22","modified_gmt":"2022-11-15T12:29:22","slug":"introducing-the-daltix-data-architecture","status":"publish","type":"post","link":"https:\/\/www.daltix.com\/en\/news\/introducing-the-daltix-data-architecture\/","title":{"rendered":"Introducing the Daltix Data Architecture"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\t
<\/div>\n\t\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

The Daltix Data Architecture Series<\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/header>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

4 october 2022<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Introducing the Daltix data architecture<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Here at Daltix we do data, and lots of it: we scrape hundreds of websites – collecting, standardising and enriching 33.5M data records and over 100M data points every single day. We started the company with the idea that a lot of value was stored in the pricing data on webshops and online grocery retailers. It didn\u2019t take us long to prove this, and more: we soon realised that product data – the name, weight, brand, etc. of a product – is also valuable. To get the value out of that data though, we needed to store it, transform it, and be able to analyse it; clearly the Microsoft Access database that our CEO started with (true story!) wasn\u2019t going to cut it. Since then, Daltix has evolved and we\u2019ve built what we think is a pretty solid data architecture. <\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t

\n\t\t\t\tWe obviously know that data is important, but we also truly believe that transparency is crucial to creating a good team, product, and company. \t\t\t<\/p>\n\t\t\t\t\t<\/blockquote>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

That is why, in this series of blog posts, we want to share our data architecture with you. Maybe one of these posts can help you in your data journey, or perhaps inspire you to join ours?<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

What does Daltix do?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Before we dive into our data architecture though, it\u2019ll be useful to give a quick overview of what it is we do. In a nutshell, we collect online retail pricing, product and promotion information, clean it and provide it in a structured fashion to our customers. The bulk of the data we download – our source data – is HTML pages and JSON API responses. We\u2019ll take this data in, clean it, standardise it and enrich it to form a core of usable data. This data then supports our three main products. First, in the simplest one, the data is shipped to the customer directly to ingest in their own data infrastructure. From there our customers use their own analytical tools to come to insights that help drive business decisions and pricing strategies. Second, it also sits behind our web-based matching application where competitor\u2019s products are matched to our customers\u2019 own products. And finally, if needed, it is also used by our standard dashboards that provide some ready-made insights to our customers.\u00a0<\/span><\/p>

As you can see, data collection and transformation is a major cornerstore in what we do. Over the years we\u2019ve built our data infrastructure to make sure we can keep processing the amounts of data that we need to, and ensure that we have the flexibility we need to accommodate new uses of the data that may be valuable to us or our customers. This scalability and flexibility have driven a lot of the decisions that have brought us to our current architecture, and which technologies sit in it.<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t
Our data architecture at a glance:<\/h5>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The image above shows our architecture in function of our data flows: from source data collection to customer delivery. We use AWS Lambda, Dask, and Snowflake for data processing and transformations, depending on the use case. Through these, the data gets transformed in stages, and each stage is stored separately: we use S3 for our early storage layers, Snowflake for the main repository for analysis-ready data, and OpenSearch and Amazon Aurora for quick-access storage. <\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

This staged design, storing data in a certain format in distinct places, is a common pattern in data architectures (where a 3-stage design is sometimes referred to as a \u2018<\/span>medallion architecture\u2019<\/span><\/a>). To understand how the data moves through our systems in particular, it\u2019s good to know which layers we have and why. Each of these layers will be discussed in more detail throughout this series, but in a nutshell we have:<\/span><\/p>