TY  - JOUR
A1  - Jiang, Lan
A1  - Naumann, Felix
T1  - Holistic primary key and foreign key detection
JF  - Journal of intelligent information systems : JIIS
N2  - Primary keys (PKs) and foreign keys (FKs) are important elements of relational schemata in various applications, such as query optimization and data integration. However, in many cases, these constraints are unknown or not documented. Detecting them manually is time-consuming and even infeasible in large-scale datasets. We study the problem of discovering primary keys and foreign keys automatically and propose an algorithm to detect both, namely Holistic Primary Key and Foreign Key Detection (HoPF). PKs and FKs are subsets of the sets of unique column combinations (UCCs) and inclusion dependencies (INDs), respectively, for which efficient discovery algorithms are known. Using score functions, our approach is able to effectively extract the true PKs and FKs from the vast sets of valid UCCs and INDs. Several pruning rules are employed to speed up the procedure. We evaluate precision and recall on three benchmarks and two real-world datasets. The results show that our method is able to retrieve on average 88% of all primary keys, and 91% of all foreign keys. We compare the performance of HoPF with two baseline approaches that both assume the existence of primary keys.
KW  - Data profiling application
KW  - Primary key
KW  - Foreign key
KW  - Database
KW  - management
Y1  - 2019
U6  - https://doi.org/10.1007/s10844-019-00562-z
SN  - 0925-9902
SN  - 1573-7675
VL  - 54
IS  - 3
SP  - 439
EP  - 461
PB  - Springer
CY  - Dordrecht
ER  - 
TY  - JOUR
A1  - Vitagliano, Gerardo
A1  - Hameed, Mazhar
A1  - Jiang, Lan
A1  - Reisener, Lucas
A1  - Wu, Eugene
A1  - Naumann, Felix
T1  - Pollock: a data loading benchmark
JF  - Proceedings of the VLDB Endowment
N2  - Any system at play in a data-driven project has a fundamental requirement: the ability to load data. The de-facto standard format to distribute and consume raw data is CSV. Yet, the plain text and flexible nature of this format make such files often difficult to parse and correctly load their content, requiring cumbersome data preparation steps. We propose a benchmark to assess the robustness of systems in loading data from non-standard CSV formats and with structural inconsistencies. First, we formalize a model to describe the issues that affect real-world files and use it to derive a systematic lpollutionz process to generate dialects for any given grammar. Our benchmark leverages the pollution framework for the csv format. To guide pollution, we have surveyed thousands of real-world, publicly available csv files, recording the problems we encountered. We demonstrate the applicability of our benchmark by testing and scoring 16 different systems: popular csv parsing frameworks, relational database tools, spreadsheet systems, and a data visualization tool.
Y1  - 2023
U6  - https://doi.org/10.14778/3594512.3594518
SN  - 2150-8097
VL  - 16
IS  - 8
SP  - 1870
EP  - 1882
PB  - Association for Computing Machinery
CY  - New York
ER  -