#déverminage

  • HoloClean - A Machine Learning System for Data Enrichment
    http://www.holoclean.io

    HoloClean is a statistical inference engine to impute, clean, and enrich data. As a weakly supervised machine learning system, HoloClean leverages available quality rules, value correlations, reference data, and multiple other signals to build a probabilistic model that accurately captures the data generation process, and uses the model in a variety of data curation tasks.

    l’installation a l’air bien compliquée, j’essaierai si j’en ai besoin mais pas avant

    • #imputation #nettoyage #déverminage

      articles en bibliographie intéressants

      A Formal Framework for Probabilistic Unclean Databases
      https://arxiv.org/pdf/1801.06750.pdf

      Abstract
      Most theoretical frameworks that focus on data errors and inconsistencies follow logic-based reasoning. Yet, practical data cleaning tools need to incorporate statistical reasoning to be effective in real-world data cleaning tasks. Motivated by these empirical successes, we propose a formal framework for unclean databases, where two types of statistical knowledge are incorporated:
      – the first represents a belief of how intended (clean) data is generated, and
      – the second represents a belief of how noise is introduced in the actual observed database instance.

      To capture this noisy channel model, we introduce the concept of a Probabilistic Unclean Database (PUD), a triple that consists of a probabilistic database that we call the intention, a probabilistic data transformator that we call the realization and captures how noise is introduced, and a dirty observed database instance that we call the observation.

      We define three computational problems in the PUD framework: cleaning (infer the most probable clean instance given a PUD), probabilistic query answering (compute the probability of an answer tuple over the unclean observed instance), and learning (estimate the most likely intention and realization models of a PUD given a collection of training data). We illustrate the PUD framework on concrete representations of the intention and realization, show that they generalize traditional concepts of repairs such as cardinality and value repairs, draw connection to consistent query answering, and prove tractability results. We further show that parameters can be learned in practical instantiations, and in fact, prove that under certain conditions we can learn a PUD directly from a single dirty database instance without any need for clean examples.

    • HoloClean: Holistic Data Repairs with Probabilistic Inference
      https://arxiv.org/pdf/1702.00820.pdf

      ABSTRACT
      We introduce HoloClean, a framework for holistic data repairing driven by probabilistic inference. HoloClean unifies existing quali- tative data repairing approaches, which rely on integrity constraints or external data sources, with quantitative data repairing methods, which leverage statistical properties of the input data. Given an inconsistent dataset as input, HoloClean automatically generates a probabilistic program that performs data repairing. Inspired by re- cent theoretical advances in probabilistic inference, we introduce a series of optimizations which ensure that inference over HoloClean’s probabilistic model scales to instances with millions of tuples. We show that HoloClean scales to instances with millions of tuples and find data repairs with an average precision of ∼ 90% and an average recall of above ∼ 76% across a diverse array of datasets exhibiting different types of errors. This yields an average F1 improvement of more than 2× against state-of-the-art methods.