GammaLink

Motivated by the increasing need of linking records across various databases, we have developed a novel graphical model based classifier that uses a mixture of Poisson distributions with latent variables to achieve this. This helps create a single source of truth - data lake after correcting duplicate entries. The idea is to derive insight into each pair of hypothesis records that match by inferring its underlying latent rate of error using Bayesian Modelling techniques. The novel approach of using gamma priors for learning the latent variables along with supervised labels is unique and allows for active learning. The naïve assumption is to not undermine the hierarchical dependencies that could be present in different scenarios but rather to illustrate a broader theory. This classifier is able to work with sparse data, streaming data as well as behave as an active learning system that could work with data dumps. The application to record linkage is able to meet several challenges of sparsity, data streams and varying nature of the datasets. The accuracy obtained is much better than the conventional approach of fuzzy linking algorithms.