First scraper tutorial (Python)

Write a real scraper by copying and pasting code, for programmers or non-programmers (30 minutes).

1. Make a new scraper

We’re going to scrape the average number of years children spend in school in different countries from this page, which was once on a UN site but has since been replaced with an Excel spreadsheet.

Create a dataset, choose the "Code in your browser" tool, then choose Python as the language. You’ll get a web based code editor.

Put in a few lines of code to show it runs, and click the “Run” button or type Ctrl+Return.

print "Hello, coding in the cloud!"

(As we go through this tutorial, you can copy and paste each block of code onto the end of your growing scraper, and run it each time.)

The code runs on ScraperWiki's servers. You can see any output you printed in the console at the bottom of the editor.

2. Download HTML from the web

You can use any normal Python library to crawl the web, such as urllib2 or Mechanize. There is also a simple built in ScraperWiki library which may be easier to use.

import scraperwiki
html = scraperwiki.scrape("http://web.archive.org/web/20110514112442/http://unstats.un.org/unsd/demographic/products/socind/education.htm")
print html

3. Parsing the HTML to get your content

lxml is the best library for extracting content from HTML.

import lxml.html
root = lxml.html.fromstring(html)
for tr in root.cssselect("div[align='left'] tr"):
    tds = tr.cssselect("td")
    if len(tds)==12:
        data = {
            'country' : tds[0].text_content(),
            'years_in_school' : int(tds[4].text_content())
        }
        print data

The bits of code like div and td are CSS selectors, just like those used to style HTML. Here we use them to select all the table rows. And then, for each of those rows, we select the individual cells, and if there are 12 of them (ie: we are in the main table body, rather than in one of the header rows), we extract the country name and schooling statistic.

4. Saving to the ScraperWiki datastore

The datastore is a magic SQL store, one where you don't need to make a schema up front.

Replace print data in the lxml loop with this save command (make sure you keep it indented with spaces at the start like this):

        scraperwiki.sqlite.save(unique_keys=['country'], data=data)

The unique keys (just country in this case) identify each piece of data. When the scraper runs again, existing data with the same values for the unique keys is replaced.

Go to the "View in a table" tool to see the data loading in. Wait until it has finished.

5. Getting the data out again

If you haven't done so yet, edit the title of your scraper.

Now, you can use other tools. For example choose "Download as spreadsheet".

For more complex queries "Query with SQL". Try this query in the SQL query box.

select * from swdata order by years_in_school desc limit 10

It gives you the records for the ten countries where children spend the most years at school.

What next?

If you have a scraper you want to write, and feel ready, then get going. Otherwise try the other tutorials.