root / atomisator / doc / README.txt

==========
atomisator
==========

This program takes feeds and inject them into a database,
after a filtering.

The first step is to read data::

    >>> from sources import get_entries
    >>> sources = (('rss', rss_file),)
    >>> feeds = get_entries(sources)
    >>> sample_feed = feeds.next()


And to link to datas contained into the database::

    >>> sqluri = 'sqlite:///tests/test.db'
    >>> from filters import bayes
    >>> bayes.SQLURI = sqluri

    >>> from entries import Entries
    >>> entries = Entries(sqluri)

Then to run filters on it::

    >>> from filters import run_filters
    >>> qualified = [entry for entry in sample_feed
    ...              if run_filters(entry, entries)]

To decide wheter to inject them into the database::

    >> entries.insert_entries(qualified)

Then provide an output::

    >> from outputs import write_output
    >> print write_output('summary', entries).encode('utf8')
    2007-05-30: Xavier Darcos programme la finde la carte scolaire
    ...

The cool thing about atomisator is the filtering. The defaut filterings
run over the entries are:

- levensthein, to avoid similar entries, even if they differ a little bit;
- html remover, to avoid storing html tags into the database.
- bayesian classifier, to decide whether an entry is interesting.

The classifier can be trained (after un-htmlization)::

    >>> from filters.bayes import bayesian_learn, bayesian
    >>> from filters.unhtml import descape
    >>> entries = list(get_entries(sources).next())
    >>> for entry in entries:
    ...     res = descape(entry, None)
    ...
    >>> bayesian_learn(entries[0], sqluri=None, answer='y')
    >>> bayesian_learn(entries[1], sqluri=None, answer='n')

Then guessed::

    >>> bayesian(entries[0], None, sqluri=None)
    True
    >>> bayesian(entries[1], None, sqluri=None)
    False