Friday, June 20, 2014

What is the Web crawler?

web crawler (also known as an automatic indexerbotWeb spiderWeb robot) is a software program which visits Web pages in a methodical, automated manner.
This process is called Web crawling or spidering, and the resulting data is used for various purposes, including building indexes for search engines, validating that ads are being displayed in the appropriate context, and detecting malicious code on compromised web servers.
Many web crawlers will politely identify themselves via their user-agent string, which provides a reliable way of excluding a significant amount of non-human traffic from advertising metrics. The IAB (in conjunction with ABCe) maintains a list of known user-agent strings as the Spiders and Bots list. However, those web crawlers attempting to discover malicious code often must attempt to appear to be human traffic, which requires secondary, behavioral filtering to detect.
Most web crawlers will respect a file called robots.txt, hosted in the root of a web site. This file informs the web crawler which directories should and shouldn't be indexed, but does not enact any actual access restrictions.
Technically, a web crawler is a specific type of bot, or software agent.


  1. They’ve grown as a developer during their five-year relationship, Silicon Valley website design constantly learning from each new project and implementing that knowledge whenever possible.

  2. They’re flexible with changing requirements and needs, and the team proactively solves problems when they arise.
    best branding agency

  3. Presenting their work and tracking their app design agencies own progress, the team cultivated an environment promoting open communication.

  4. The site feels modern and is fully responsive. Project management could be improved by defined scheduling
    user interface design firms

  5. They became a valuable business partner by finding innovative solutions and making money-saving suggestions.
    top user interface designs

  6. It’s my first time to visit this site & I’m really surprised to see such impressive stuff out there.
    company logo designer

  7. It's not my very first time to visit this blog; I’m visiting this daily and acquire superb info from here day by day.
    logo design services

  8. The gorgeous post learned a great deal Thanks greatly!
    top web design firms

  9. Every week-end I used to pay a fast visit this site, because I’d like enjoyment, because this web site conations certainly fussy material.
    best website designers

  10. This post is really valuable that designed for the new visitors. Pleasing work, keep on writing.
    design firms

  11. If you really desire to get such type of information, visit this blog quickly.
    UI design firm