root / PyCon07 / material / tutorial_9.txt

====================================================
Creating a local copy of a website with tagged words
====================================================

:abstract:
  
  This tutorial will explain you how to create a local copy of web pages,
  with tagged words. There's no particular prerequisites for this tutorial.

.. contents: Toc

The job is done in three parts:

- reading the configuration file
- getting the content
- extracting the words to tag and write the files

Reading the configuration file
==============================

Python provides a module that knows how to read *ini* files, that have 
sections. Such file can look like this::

    [webpages]
    python=http://python.org
    pycon=http://us.pycon.org

    [options]
    path=/tmp

    [words]
    cool
    fun
    neat

`ConfigParser` knows how to handle these files, and can be used to provide
a simple access to them.

-> learn `how to use ConfigParser`

Getting the content
===================

The configuration is now read. Let's get the content over the web. `urllib2`
provides such a service, and let you suck a distant page content to a 
workable string.

-> learn `how to use urllib2`

Extracting the words
====================

The workable string can be parsed with a regular expression matcher, that 
let you substitute a matching string with a value. In our case, the expression
would be each word listed in the `words` section of the file.

-> learn `how to substitute a value in a string with *re*`

Linking everything
==================

Our program works in 4 phases:

- reads the `webpages` section of the configuration file
- get all web pages
- for each of them, modify the words pointed by the configuration list by 
  a bolded word
- save them in text files, in the `path` given.