30-01-2012, 04:38 PM
Web Crawler
jignesh seminar.ppt (Size: 259 KB / Downloads: 112)
INTRODUCTION
Definition:A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner.
The Role of crawlers is to collect web content
Types of crawler
Batch crawler : its sanpsnot of their crawl space,unlit reaching a certain size or time limit,certain number of pages are crawled
Incremental crawler : continuously crawl their crawl space,revisiting URLs to ensure freshness
Focused crawler : attempt to crawl pages pertaining to some topic,while minimizing number of off-topic pages that are collected
Features of a crawler
-Robustness: spider traps
-Infinitely deep directory structures
-Pages filled a large number of characters
conclusions
All the serch engines/companies employ research staff which are also academically involved: sit on PCs referee journal papers,present at conferences
jignesh seminar.ppt (Size: 259 KB / Downloads: 112)
INTRODUCTION
Definition:A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner.
The Role of crawlers is to collect web content
Types of crawler
Batch crawler : its sanpsnot of their crawl space,unlit reaching a certain size or time limit,certain number of pages are crawled
Incremental crawler : continuously crawl their crawl space,revisiting URLs to ensure freshness
Focused crawler : attempt to crawl pages pertaining to some topic,while minimizing number of off-topic pages that are collected
Features of a crawler
-Robustness: spider traps
-Infinitely deep directory structures
-Pages filled a large number of characters
conclusions
All the serch engines/companies employ research staff which are also academically involved: sit on PCs referee journal papers,present at conferences