frenchtext

NLP library to process french text.

In this early pre-version, the library provides :

• datasets to train business-oriented french text models
• a characters normalization pipeline tailored for french text

Install

pip install frenchtext

Dependencies

• pandas
• pyarrow
• requests
• fastprogress

Licence

APACHE licence 2.0 : https://www.apache.org/licenses/LICENSE-2.0

How to use

The detailed documentation for each module is available through the menu on the left side of this page.

You will find below an overview of the library.

French datasets

Data sources

The text content of the main french websites in the domain of finance and business (+ wikipedia) was extracted in september 2019 using nlptextdoc.

This extraction was done as "politely" as possible:

• extract only freely and publicly available content
• respect the robots.txt directives of each website (pages forbidden for indexing, maximum extraction rate)
• detect when websites use tools to prevent indexing (like Datadome) and abort the crawl

IMPORTANT: The original authors of the websites own the copyright on all text blocks in this dataset.

To be able to link each text block to its original author, we track the origin URL of each text block throughout the whole process.

YOU CAN'T REUSE THE TEXT BLOCKS FOR ANY PURPOSE EXCEPT TRAINING A NATURAL LANGUAGE PROCESSING MODEL.

See the new European copyright rules : European Parliament approves new copyright rules for the internet

"The directive aims to make it easier for copyrighted material to be used freely through text and data mining, thereby removing a significant competitive disadvantage that European researchers currently face."

=> 131 websites and 2 564 755 HTML pages

Data preparation

The text blocks were then:

• deduplicated to keep only distinct text blocks for each website (forgetting part of the original document structure),
• tagged (but not filtered) by language (using https://fasttext.cc/docs/en/language-identification.html),
• grouped in categories according to the main theme of the original website,
• split in Pandas dataframes of size < 2 GB.

=> 10 categories: 'Assurance', 'Banque', 'Bourse', 'Comparateur', 'Crédit', 'Forum', 'Institution', 'Presse', 'SiteInfo', 'Wikipedia'

In each dataframe, the text blocks were additionnaly SHUFFLED IN A RANDOM ORDER to make it very difficult to reconstruct the original articles (safety measure to help protect the copyrights of the authors).

The results of this second step can be downloaded in the config.datasets directory, as dataframes serialized in the feather format, in files named according to the 'DatasetFile' column of the datasets table.

=> 19 dataset files: 'assurance', 'banque', 'bourse', 'comparateur', 'crédit', 'forum', 'institution', 'presse-1', 'presse-2', 'presse-3', 'presse-4', 'presse-5', 'presse-6', 'siteinfo', 'wikipedia-1', 'wikipedia-2', 'wikipedia-3', 'wikipedia-4', 'wikipedia-5'

Dataset size

The number of words in each text block was computed using the default french tokenizer from spaCy v2.1.

This business-oriented dataset contains 2 billion french words.

Here is a summary of the number of words contributed by each category in millions:

• Assurance : 12
• Banque : 20
• Bourse : 26
• Comparateur : 20
• Crédit : 1
• Forum : 152
• Institution : 4
• Presse : 963
• SiteInfo : 78
• Wikipedia : 727

Dataset files

from frenchtext.core import *
from frenchtext.datasets import *

List available dataset files :

datasetfiles = list_dataset_files()
datasetfiles

['assurance',
 'banque',
 'bourse',
 'comparateur',
 'crédit',
 'forum',
 'institution',
 'presse-1',
 'presse-2',
 'presse-3',
 'presse-4',
 'presse-5',
 'presse-6',
 'siteinfo',
 'wikipedia-1',
 'wikipedia-2',
 'wikipedia-3',
 'wikipedia-4',
 'wikipedia-5']

Source websites and number of words in each dataset file :

datasetsdf = list_datasets()
datasetsdf[["DatasetFile","Url","Pages","Words"]].iloc[80:100]

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

Download dataset files

download_dataset_file("assurance")

Downloading dataset file : assurance (17 MB)

download_all_datasets()

Downloading dataset file : assurance (17 MB)
Downloading dataset file : banque (28 MB)
Downloading dataset file : bourse (38 MB)
Downloading dataset file : comparateur (28 MB)
Downloading dataset file : crédit (2 MB)
Downloading dataset file : forum (220 MB)
Downloading dataset file : institution (5 MB)
Downloading dataset file : presse-1 (218 MB)
Downloading dataset file : presse-2 (196 MB)
Downloading dataset file : presse-3 (190 MB)
Downloading dataset file : presse-4 (234 MB)
Downloading dataset file : presse-5 (269 MB)
Downloading dataset file : presse-6 (334 MB)
Downloading dataset file : siteinfo (116 MB)
Downloading dataset file : wikipedia-1 (131 MB)
Downloading dataset file : wikipedia-2 (182 MB)
Downloading dataset file : wikipedia-3 (263 MB)
Downloading dataset file : wikipedia-4 (269 MB)
Downloading dataset file : wikipedia-5 (267 MB)

You can change the local directory where the dataset files are downloaded :

config.datasets

PosixPath('/home/laurent/.frenchtext/datasets')

config["datasets_path"] = "/tmp/datasets"
config.datasets.mkdir(parents=True, exist_ok=True)

config.datasets

PosixPath('/tmp/datasets')

Read dataset files

datasetdf = read_dataset_file("assurance")
datasetdf

Loaded dataframe for dataset assurance : 563613 text blocks

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

Access text blocks in dataset files

Filter and iterate over the rows of a dataset file :

rowsiterator = get_rows_from_datasetdf(datasetdf, minwords=None, maxwords=5, lang="?")
show_first_rows(rowsiterator,10)

12 - COORDONNEES
41 - 01 30 41 67 33
49 - Dmitriy G.
57 - Les atouts du Multisupport CONFIANCE
74 - 01XXL meribel hiver
76 - Garantie en cas de vol
87 - Par AXA, le 01/08/2016
96 - mgr@enderby.eu
127 - 18 place De Strasbourg
131 - Saint Gaudens

Filter and iterate over the text blocks of a full dataset (across multiple files) :

textiterator = get_textblocks_from_dataset("Assurance", minwords=None, maxwords=10, lang="fr")
show_first_textblocks(textiterator,skip=2000,count=10)

Loaded dataframe for dataset assurance : 563613 text blocks
2001 - Rééquipement à neuf à vie
2002 - Définition Conducteur secondaire- Lexique
2003 - Comment éviter les fraudes
2004 - Comment demander un remboursement santé - GENERALI
2005 - Simulateur pour connaître les obligations de votre accord de branche
2006 - Complémentaire Epargne retraite des indépendants et TNS - Malakoff Médéric
2007 - Experts-Comptables, découvrez la mission épargne salariale
2008 - Vous n’êtes pas encore client :
2009 - Actualités (Page 6) | ameli.fr | Pharmacien
2010 - Dépression : quelle prise en charge ? - Matmut

Access a specific row :

get_text_from_rowindex(datasetdf,100)

'Les inondations de plaine : débordement de cours d’eau avec une durée d’immersion longue (prévisibles plusieurs jours ou heures à l’avance).'

Find text blocks with a specific char or substring :

find_textblocks_with_chars(datasetdf,"rétroviseur",count=20,ctxsize=15)

350594     ore dans notre rétroviseur gauche lorsque 
149029     de glace ? Les rétroviseurs ainsi que les 
51349      ace. Quant aux rétroviseurs, ils le sont d
310354     vant, arrière, rétroviseurs et vitres laté
489866    \naussi dans le rétroviseur pour ne pas se 
364550     ôté ou sous le rétroviseur intérieur de vo
560539     tionnement des rétroviseurs.              
560700     é (pare-brise, rétroviseurs…),            
223621     riorations des rétroviseurs et des phares.
543903     es miroirs des rétroviseurs lorsqu’ils peu
502075      logo dans son rétroviseur et par un signa
53237      vous cassez le rétroviseur d’une voiture. 
310456      éraflures, un rétroviseur abîmé, ou un au
375158     ant, moteur de rétroviseurs…              
539914     nt et arrière, rétroviseurs intérieurs et 
171367     t utilisez vos rétroviseurs               
485058      ainsi que les rétroviseurs ne sont pas ga
277390     ant, moteur de rétroviseurs...            
20222      sont offerts : rétroviseurs électriques, c
317634     res, y compris rétroviseurs et feux       
Name: Text, dtype: object

find_textblocks_with_chars(datasetdf,64257,count=10,wrap=True)

175413    x besoins de diversi[fi]cation des placements
337398    e 30 villes ont béné[fi]cié de ces animations
265114    nt règlementaire et [fi]nancier, nous accompa
74267          La Fondation a [fi]nancé depuis 2009, l’
424584    tion de l’équilibre [fi]nancier des régimes d
219195    d, Jérôme Powell con[fi]rmera que, dans l’att
489511    s besoins de diversi[fi]cation de la clientèl
517563    si en présence d’un [fi]nancement par crédit,
479694    nt règlementaire et [fi]nancier, La Mondiale 
252202    n de disponibilités [fi]nancières mais aussi,
Name: Text, dtype: object

Track the source URL for each text block

Optionally download and read urls file to track the origin of each text block :

urlsdf = read_urls_file()
urlsdf.head()

Loaded datasets urls : 2668787 urls

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

get_text_from_rowindex(datasetdf,100)

'Les inondations de plaine : débordement de cours d’eau avec une durée d’immersion longue (prévisibles plusieurs jours ou heures à l’avance).'

get_url_from_rowindex(datasetdf, 100)

'https://www.maif.fr/conseils-prevention/risques-majeurs/inondation.html'

Characters normalization pipeline

Motivation

French datasets often contain several thousands distinct Unicode characters.

Characters stats in Wikipedia dataset :

• 35.6 billion chars
• 13 502 distinct Unicode chars

Characters stats in Business dataset :

• 27.5 billion chars
• 3 763 distinct Unicode chars

We need to reduce the number of distinct characters fed to our natural language processing applications, for three reasons :

• chars considered by the user as visually equivalent will often produce a different application behavior : this is a huge problem for the user experience
• with so many chars, the designer of the NLP application will not be able to reason about all possible combinations : this could harm the explainability of the system
• this huge number of distinct characters brings a significant amount complexity the NLP models will have to deal with

Characters stats in Wikipedia dataset :

• Only 1316 chars more frequent than 1 in 100 million
• 99.9987 % of Wikipedia chars would be preserved if we only kept the frequent chars

Characters stats in Business dataset :

• Only 531 chars more frequent than 1 in 100 million
• 99.9996 % of Business chars would be preserved if we only kept the frequent chars

We can be smarter than that and replace rare chars with equivalent (or mostly equivalent) more frequent chars to preserve a maximum of information.

Target characters set

After a detailed study of all the frequent chars, the goal is to design a noramization pipeline which can retain as much information as possible while greatly reducing the number of dinstinct chars.

We saw before that it is possible to preserve 99.9996% of the original chars while keeping only 500 distinct chars. By being clever and replacing equivalent chars, we can divide this number by 2 and still retain the same amount of information.

It may then be useful to limit the number of distinct characters after normalization to 255 distinct characters :

• if needed, french text chars can then be encoded with a single byte
• the list of supported chars can be memorized by NLP application developers and users

from frenchtext.core import *
from frenchtext.chars import *

255 supported characters after normalization :

import pandas as pd
dfcharsnorm = pd.read_csv(chardatadir / "charset-fr.csv", sep=";")
dfcharsnorm

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

The table below shows the number of chars in each category (after normalization) per 100 million characters :

dfblocks = dfcharsnorm.groupby(by=["Category","SubCategory"]).agg({"Char":["count","sum"],"CountBusiness":"sum"})
dfblocks["CountBusiness"] = (dfblocks["CountBusiness"] / 27577304956 * 100000000).astype(int)
dfblocks

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead tr th {
    text-align: left;
}

.dataframe thead tr:last-of-type th {
    text-align: right;
}

Normalization pipeline overview

The normalization pipeline applies the following 14 steps, which are explained and illustrated in the sections below.

• Fix encoding errors
  • fix windows1252 text read as iso8859-1
  • fix utf8 text read as windows1252
  • fix windows1252 text read as utf8
  • merge Unicode combining chars
  • ignore control chars
• Remove display attributes
  • replace latin letter symbols
  • replace latin letter ligatures
  • replace latin number symbols
• Normalize visually equivalent chars
  • replace equivalent chars
  • replace cyrillic and greek chars looking like latin letters
• Encode infrequent chars while losing a little bit of information
  • replace infrequent latin letters with diacritics
  • replace infrequent chars from other scripts
  • replace infrequent symbols
  • ignore remaining chars with no glyph

The statistics below count the number of chars normalized for 1 million chars in 4 distinct parts of the french datasets : business websites, forums, news, wikipedia.

The first line of the table below shows that :

• in 1 million chars extracted from forum pages (raw users input), 41.8 chars will be encoding errors (windows1252 read as iso8859-1)
• in 1 million chars extracted from wikipedia (curated content), only 0.006 chars will be encoding errors

These numbers show that characters normalization is much more important in real world applications than in academic papers based on clean wikipedia text.

normstats = pd.read_csv(chardatadir / "stats" / "normalization.total.stats.csv")
normstats[["Transform","FreqBusiness","FreqForum","FreqPresse","FreqWikipedia"]]

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

Most frequent chars replaced from equivalent characters :

replacestats = pd.read_csv(chardatadir / "stats" / "normalization.layer8.stats.csv")
replacestats[["Char","CharName","FreqBusiness","FreqForum","FreqPresse","FreqWikipedia"]].head(20)

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

For example, list of all Unicode chars wich will be projected to a regular 'apostrophe' :

replacechars = pd.read_csv(chardatadir / "normalizedchars.csv", sep=';')
replacechars[replacechars["NormChar"]=="'"][["Code","Char","CharName"]]

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

Frequency of characters from other scripts (chinese, arabic, cyrillic ...) :

scriptsstats = pd.read_csv(chardatadir / "stats" / "normalization.layer11.stats.csv")
scriptsstats[["CharFamily","FreqBusiness","FreqForum","FreqPresse","FreqWikipedia"]]

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

Normalization pipeline API

Initialize a text normalizer :

%time norm = TextNormalizer()
norm

CPU times: user 1.83 s, sys: 15.6 ms, total: 1.84 s
Wall time: 2 s





1 - Fix encoding errors : windows1252 read as iso8859-1
2 - Fix encoding errors : utf8 read as windows1252
3 - Fix encoding errors :  windows1252 read as utf8
4 - Merge Unicode combining chars
5 - Ignore control chars
6 - Replace latin letter symbols
7 - Replace latin letter ligatures
8 - Replace latin number symbols
9 - Normalize equivalent chars
10 - Replace cyrillic and greek chars looking like latin letters
11 - Replace infrequent chars : latin letters with diacritics
12 - Replace infrequent chars : other scripts
13 - Replace infrequent chars : symbols
14 - Replace infrequent chars : chars to ignore

Normalize text :

teststring = chr(127995)+"① l`"+chr(156)+"uv"+chr(127)+"re est¨ "+chr(147)+"belle"+chr(148)+"¸ à  ½ € énième ‰ "+chr(133)+" ⁽🇪ffic🇦ce⁾ ！"
teststring

'🏻① l`\x9cuv\x7fre est¨ \x93belle\x94¸ à  ½ € énième ‰ \x85 ⁽🇪ffic🇦ce⁾ ！'

result = norm(teststring)
result

(1) l'oeuvre est «belle», Ã  1/2 € énième ‰ … (EfficAce) !

Describe the changes applied by the normalization pipeline :

print(result.describeChanges())

Fix encoding errors : windows1252 read as iso8859-1
 < 🏻① l` [�] uv�re est¨  [�] belle [�] ¸ à  ½ € énième ‰  [�]  ⁽🇪ffic🇦ce⁾ ！
 < 🏻① l` [œ] uv�re est¨  [“] belle [”] ¸ à  ½ € énième ‰  […]  ⁽🇪ffic🇦ce⁾ ！
Fix encoding errors : utf8 read as windows1252
 < 🏻① l`œuv�re est¨ “belle”¸ à   [½]   [€]  énième  [‰]  … ⁽🇪ffic🇦ce⁾ ！
 < 🏻① l`œuv�re est¨ “belle”¸ à   [½_]   [€__]  énième  [‰__]  … ⁽🇪ffic🇦ce⁾ ！
Merge Unicode combining chars
 < 🏻① l`œuv�re est¨ “belle”¸ à  ½ €  [é] ni [è] me ‰ … ⁽🇪ffic🇦ce⁾ ！
 < 🏻① l`œuv�re est¨ “belle”¸ à  ½ €  [é_] ni [è_] me ‰ … ⁽🇪ffic🇦ce⁾ ！
Ignore control chars
 <  [🏻] ① l`œuv [�] re est [¨]  “belle”¸ à  ½ € énième ‰ … ⁽🇪ffic🇦ce⁾ ！
 <  [_] ① l`œuv [_] re est [_]  “belle”¸ à  ½ € énième ‰ … ⁽🇪ffic🇦ce⁾ ！
Replace latin letter symbols
 < ① l`œuvre est “belle”¸ à  ½ € énième ‰ … ⁽ [🇪] ffic [🇦] ce⁾ ！
 < ① l`œuvre est “belle”¸ à  ½ € énième ‰ … ⁽ [E] ffic [A] ce⁾ ！
Replace latin letter ligatures
 < ① l` [œ ] uvre est “belle”¸ à  ½ € énième ‰ … ⁽E [ffi  ] cAce⁾ ！
 < ① l` [oe] uvre est “belle”¸ à  ½ € énième ‰ … ⁽E [ffi] cAce⁾ ！
Replace latin number symbols
 <  [①  ]  l`oeuvre est “belle”¸ à   [½  ]  € énième ‰ … ⁽EfficAce⁾ ！
 <  [(1)]  l`oeuvre est “belle”¸ à   [1/2]  € énième ‰ … ⁽EfficAce⁾ ！
Normalize equivalent chars
 < (1) l [`] oeuvre est  [“] belle [”]  [¸]  Ã  1/2 € énième ‰ …  [⁽] EfficAce [⁾]   [！] 
 < (1) l ['] oeuvre est  [«] belle [»]  [,]  Ã  1/2 € énième ‰ …  [(] EfficAce [)]   [!] 

Compute spans for equivalent substrings before and after normalization :

result.output[0:12]

"(1) l'oeuvre"

result.input[result.mapOutputIndexToInput(0):result.mapOutputIndexToInput(12)]

'🏻① l`\x9cuv\x7fre'

result.output[3:10]

" l'oeuv"

result.input[result.mapOutputIndexToInput(3):result.mapOutputIndexToInput(10)]

' l`\x9cuv\x7f'

Performance test : 2500 sentences per second => fast enough but will be optimized in a later version.

%timeit -n100 norm(teststring)

397 µs ± 89.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Appendix : Unicode utility functions

Unicode characters properties :

charname("🙂")

'Slightly Smiling Face'

charcategory("🙂")

'Symbol'

charsubcategory("🙂")

'Other'

charblock("🙂")

'Emoticons'

blockfamily('Emoticons')

'Symbols'