Changeset 222:c07c73ab5741 for xap/trunk/tokenizer

Timestamp:

03/21/08 17:38:57 (8 months ago)

Author:

Lafaye Philippe (RAGE2000) <lafaye@…>

Message:

The character table use now unicode

Files:

• xap/trunk/tokenizer/filters.py (modified) (1 diff)

xap/trunk/tokenizer/filters.py

@@ -194 +194 @@
         for word in words:
             splited = word.split()
-            result[splited[0]] = splited[1]
+            result[splited[0].decode('utf8')] = splited[1]
         return result