Research on information network data cleaning and fusion technology based on Feature Engineering

Abstract

The on-board terminal, roadside unit, business system and other nodes are coordinated. The data presents the characteristics of multi-source heterogeneity, high noise and strong timing. The collection often introduces omissions, abnormalities, repetitions and inconsistencies, which affects the perception and decision-making of the upper level. This article takes the multi-source data cleaning and integration of the information network as the object, and proposes a data governance framework with feature engineering as the main line. The framework takes the four-dimensional quality evaluation of integrity, accuracy, consistency and timeliness as the starting point to locate defects; time alignment, missing interpolation, abnormal detection and repair and other operations are carried out in turn. Among them, the interpolation adopts the EM algorithm based on the expectation of multivariate Gausssian conditions, and the abnormal detection adopts the 3σ criterion of de-trend residual. After cleaning, the special The variance inverse weighting and principal component analysis are used for grading. Experiments show that the feature engineering-oriented cleaning and fusion method can significantly improve the data quality of the information network and provide a consistent and usable data basis for subsequent analysis.