Semi-supervised data organization for interactive anomaly analysis

Semi-supervised data organization for interactive anomaly analysis
Aslam, Javed A. (Author)
Bratus, Sergey (Author)
Pavlu, Virgil (Author)
Copyright date:
Type of resource:
Digital origin:
born digital
We consider the problem of interactive iterative analysis of datasets that consist of a large number of records represented as feature vectors. The record set is known to contain a number of anomalous records that the analyst desires to locate and describe in a short and comprehensive manner. The nature of the anomaly is not known in advance (in particular, it is not known, which features or feature values identify the anomalous records, and which are irrelevant to the search), and becomes clear only in the process of analysis, as the description of the target subset is gradually refined. This situation is common in computer intrusion analysis, when a forensic analyst browses the logs to locate traces of an intrusion of unknown nature and origin, and extends to other tasks and data sets. To facilitate such "browsing for anomalies", we propose an unsupervised data organization technique for initial summarization and representation of data sets, and a semi-supervised learning technique for iterative modifications of the latter representation. Our approach is based on information content and Jensen-Shannon divergence and is related to information bottleneck methods. We have implemented it as a part of the Kerf log analysis toolkit.
Originally published in Proceedings of the 5th International Conference on Machine Learning and Applications, 2006.
Subjects and keywords:
Anomaly detection (Computer security)
Iterative methods (Mathematics)
semisupervised learning technique
data organization
interactive anomaly analysis
interactive iterative analysis
feature vectors
anomalous record sets
computer intrusion analysis
forensic analyst
information content
Jensen-Shannon divergence
Kerf log analysis toolkit
Computer Sciences
Permanent Link: