Submitted by Chen Wang, 8 months ago

(well not really sure if this is the place to submit competition entry) Contest_Submision_dabsused is a crawler made for one of my pet projects dabsused.com which tracks price changes of used stocks on several websites run by a popular UK e-commerce company, Dabs. I wrote existing crawler in Java a few months ago and it's been in production since; This 80App is to replicate what the crawler does for checking prices. It's not a general purpose crawler, rather it's designed specifically for currently layout of those websites, which share a common backend system. I managed to get very similar, definitely usable results from creating a 80App crawler but it's not plain sailing either. I've put together a few slides to express my opinions (see attachment) and hopefully that's going to help someone or 80legs :) The crawler project's source is available for download too. 

Submitted by Diederik van Liere, 8 months ago

Companies are increasingly aware that they need to monitor the blogosphere to see what is being said about their brands and products. Given the fact that there are too many blogs to follow individually, which blogs are important to monitor? Obviously, looking at the number of incoming links is a known metric for impact but we need additional measures to determine who is influential in the blogosphere and who is not.

Blogcrawler is an 80legs.com hosted application to measure the community strength of a blog. Community strength is defined as the number of returning commenters divided by the total number of commenters and is scaled by the number of posts of the blog and total number of comments. This measure is best used when comparing multiple blogs and traffic stats for these blogs are unknown. Returning commenters are an indication that a blogger has the capability to attract and retain an audience.

Blogcrawler is a generic tool for blogs as it supports generic parsers for most blogs and can be extended with custom parsers if needed to crawl a specific blog.

After the crawler has crawled all required blogs, then a simple Python script is used to parse the results and calculate community strength for each blog. Community strength can be calculated for different time windows: last week, last month or even last year.

Detailed output is also possible, how many articles were posted in a given time window, how many comments were made and how many unique commenters participated.

The default parsers are configured for a number of Canadian political blogs including: eaves.ca, terahertzatheist.ca, djkelly.ca and stageleft.info.

You can reach me at @drdee_is_wired

Submitted by Shawn Simister, 8 months ago

AdverSpider is a next-generation web crawler that specializes in detecting and analyzing online advertising. AdverSpider takes a list of URLs, visits each page and figures out how many ads are being displayed, which ad networks are being used and which sizes of ads are available. This data is returned to you as XML which makes it easy to process however you'd like.

For more details see: http://shawn.simister.ca/AdverSpider/