Web Crawler : Review of Different Types of Web Crawler, Its Issues, Applications and Research Opportunities

Keyur Desai, Virala Devulapalli, Prof. Smita Agrawal, Prof. Preeti Kathiria, Dr. Atul Patel

Abstract


Today's search engines are equipped with dedicated agents known as “web crawlers” keen to crawling large web contents online which are analyzed and indexed and make the content available to users. Crawlers act together with thousands of web servers over periods expanding from weeks to several years. These crawlers visits several thousands of pages every second, includes a high-performance fault manager, are platform independent or dependent and are able to get used to a wide range of configurations without including additional hardware. This paper is focused on prerequisites of crawler, process of crawling and different types of crawlers. This paper give review about some potential issues related to crawler, applications and research area of web crawler.

Keywords: search engine, web crawler, www, Indexing, website analysis,

Full Text:

PDF


DOI: https://doi.org/10.26483/ijarcs.v8i3.3190

Refbacks

  • There are currently no refbacks.




Copyright (c) 2017 International Journal of Advanced Research in Computer Science