Document Clustering Using Cosine Similarity

Abstract

Clustering or Cluster Analysis is a process of grouping similar objects in such a way that the objects in the group (cluster) are similar to each other than the objects in other groups (clusters). Clustering is an unsupervised machine learning technique where only the input data is served 3 (unlike as in supervised, a set of sample input and output pair is provided) to the system corresponding to which it recognizes a pattern and predicts the output automatically, Hence complete automation is achieved here. In specific to our work that is Document clustering is organizing the text files into clusters containing similar files (File Content). High precise clustering algorithms like K-means play an important role in data storage, data manipulation and information retrieval systems. Search engines like Google, Yahoo, Bing etc. uses Document clustering in addition to high-end processors and servers to retrieve the information in response to the various search queries. The most commonly used clustering technique is K-means, where the objects are divided into ‘k’ number of clusters with similar objects in it. The present work is focused on Document clustering using ‘Cosine Similarity’ where the pre-processing work is carried out by a readymade Java library known as ‘Apache Lucene’. The texts in the documents are broken down into strings, and the extracted strings is fed to the Apache Lucene which pre-processes the data, the number of repetitions of each word and gives the output as JSON objects. Then the cosine similarity is calculated with these indexed words. The final result of this work outputs the documents that are similar to each other, that are exactly similar to each other (copy documents) and the ones which are unique (outlier). The applications of document clustering include mining useful data in large datasets, web page clustering, search engines, anti-plagiarism checkers etc.