Multivariate correlations discovery in static and streaming data

Published in VLDB 2022 (Proceedings of the VLDB Endowment 15(6)), 2022

Correlation analysis is an invaluable tool in many domains, for better understanding data and extracting salient insights. Most works to date focus on detecting high pairwise correlations. A generalization of this problem with known applications but no known efficient solutions involves the discovery of strong multivariate correlations, i.e., finding vectors (typically in the order of 3 to 5 vectors) that exhibit a strong dependence when considered altogether. In this work we propose algorithms for detecting multivariate correlations in static and streaming data. Our algorithms, which rely on novel theoretical results, support two different correlation measures, and allow for additional constraints. Our extensive experimental evaluation examines the properties of our solution and demonstrates that our algorithms outperform the state-of-the-art, typically by an order of magnitude.

Hierarchical clustering of eight time series a to h: a root cluster containing all of them, split into clusters C1, C2 and C3, which are split further into clusters C4 to C10.
The running example: time series are organised in a hierarchy of clusters, so that bounds on the correlation of whole cluster combinations can rule out or confirm many candidate correlations at once. (Figure from the paper; click to enlarge.)

The algorithms are available as the open-source Correlation Detective library. An extended version supporting four correlation measures appeared in the VLDB Journal. I presented this work at VLDB 2022 in Sydney.

Recommended citation: Koen Minartz, Jens E. d'Hondt, and Odysseas Papapetrou. 2022. Multivariate correlations discovery in static and streaming data. Proc. VLDB Endow. 15, 6 (February 2022), 1266–1278.
Download Paper | Download Slides