What is the Web crawler?

A web crawler (also known as an automatic indexer, bot, Web spider, Web robot) is a software program which visits Web pages in a methodical, automated manner.

This process is called Web crawling or spidering, and the resulting data is used for various purposes, including building indexes for search engines, validating that ads are being displayed in the appropriate context, and detecting malicious code on compromised web servers.

Many web crawlers will politely identify themselves via their user-agent string, which provides a reliable way of excluding a significant amount of non-human traffic from advertising metrics. The IAB (in conjunction with ABCe) maintains a list of known user-agent strings as the Spiders and Bots list. However, those web crawlers attempting to discover malicious code often must attempt to appear to be human traffic, which requires secondary, behavioral filtering to detect.

Most web crawlers will respect a file called robots.txt, hosted in the root of a web site. This file informs the web crawler which directories should and shouldn't be indexed, but does not enact any actual access restrictions.

Technically, a web crawler is a specific type of bot, or software agent.

Ad Serving Options

Ad Serving Options 

There are two methods by which the winning bidder can return ad markup to the exchange. In either case, the ad markup is either XHTML if the bidder is responding with a banner or VAST XML if responding with a VAST video.

1 Ad Served on the Win Notice

In this method, ad markup is returned to the exchange is via the win notice. In this case, the

response body of the win notice call (e.g., invoking the “nurl” attribute) contains the ad markup

and only the ad markup; there must be no other structured data in the response body. Using

this method, the “adm” attribute in the “bid” object must be omitted.

2 Ad Served in the Bid

In this method, ad markup is returned directly in the bid itself. This is accomplished via the

“adm” attribute in the “bid” object. If both the “adm” attribute and win notice return data, the

“adm” contents will take precedence.

3 Comparison of Ad Serving Approaches

Each of the ad serving methods has its own advantages that may be of varying importance to

either the exchange or the bidder.

            3.1 Ad Served in the Bid

                   1   Potential Concurrency: The exchange can choose to return that ad markup and
                        call the win notice concurrently, thereby improving user experience.

                   2  Reduced Risk of Forfeiture: A forfeit is the scenario in which a bidder wins, but

                       forfeits due to technical failure serving the ad. This can occur when serving on

                       the win notice (e.g., win notice call failure), but is mitigated by including the ad

                       in the bid.

               

            3.2 Ad Served on the Win Notice

1 Reduced Bandwidth Costs: Serving ad markup only upon winning can save large

                    amounts of bandwidth usage, the costs for which can mount up over high volumes.

2 Additional Bidder Flexibility: Bidders may typically know the ad they will serve

                           at the time of bid, but this provides an additional optional decision point after

                           the settlement price has been established.

Mobile Interstitial

Banners are small ads that when touched typically take the user to some form of full-screen in-app browsing experience.

Interstitials, on the other hand, immediately present rich HTML5 experiences or "web apps" at natural app transition points such as launch, video pre-roll or game level load. Web apps are in-app browsing experiences with a simple close button rather than any navigation bar—the content provides its own internal navigation scheme. Interstitial ads are typically more expensive and subject to impression constraints.