A Formal Framework for Data Lakes Based on Category Theory - Université de Bourgogne Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

A Formal Framework for Data Lakes Based on Category Theory

Résumé

The management of Big Data requires flexible systems to handle the heterogeneity of data models as well as the complexity of analytical workflows. Traditional systems like data warehouses have reached their limits due to their rigid schema-on-write paradigm, that requires well identified and defined use cases to ingest data. Data lakes, with their schema-on-read paradigm, have been proposed as more flexible systems in which raw data are directly stored in their original format associated with metadata, to be accessed and transformed only when users need to process or analyze them. Thus, it is necessary to define and control the different levels of abstraction and the dependencies among functionalities of a data lake to use it efficiently. In this article, we present a formal framework aiming to define a data lake pattern and to unify the interactions among the functionalities. We use the category theory as theoretical foundations to benefit from its high level of abstraction and its compositionality. By relying on different categories and functors, we ensure the navigation among the functionalities and allow the composition of multiples operations, while keeping track of the entire lineage of data. We also show how our framework can be applied on a simple example of data lake.
Fichier principal
Vignette du fichier
IDEAS_2022_DataLakes_FINAL_HAL (1).pdf (926.03 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03889812 , version 1 (30-01-2023)

Identifiants

Citer

Alexis Guyot, Annabelle Gillet, Eric Leclercq, Nadine Cullot. A Formal Framework for Data Lakes Based on Category Theory. International Database Engineered Applications Symposium (IDEAS'22), Aug 2022, Budapest, Hungary. ⟨10.1145/3548785.3548797⟩. ⟨hal-03889812⟩
17 Consultations
133 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More