factsumm

Release 0.1.5

FactSumm: Factual Consistency Scorer for Abstractive Summarization

Homepage PyPI Python

Keywords
abstractive-summarization, factual-consistency, natural-language-processing, summarization
License
Apache-2.0
Install
pip install factsumm==0.1.5

Documentation

FactSumm: Factual Consistency Scorer for Abstractive Summarization

GitHub release Apache 2.0 Issues

FactSumm is a toolkit that scores Factualy Consistency for Abstract Summarization

Without fine-tuning, you can simply apply a variety of downstream tasks to both the source article and the generated abstractive summary

For example, by extracting fact triples from source articles and generated summaries, you can verify that generated summaries correctly reflect source-based facts ( See image above )


Installation

With and Python 3 (>= 3.8), you can install factsumm simply using pip:

pip install factsumm

Or you can install FactSumm from source repository:

git clone https://github.com/huffon/factsumm
cd factsumm
pip install .

Usage

>>> from factsumm import FactSumm
>>> factsumm = FactSumm()
>>> article = "Lionel Andrés Messi (born 24 June 1987) is an Argentine professional footballer who plays as a forward and captains both Spanish club Barcelona and the Argentina national team. Often considered as the best player in the world and widely regarded as one of the greatest players of all time, Messi has won a record six Ballon d'Or awards, a record six European Golden Shoes, and in 2020 was named to the Ballon d'Or Dream Team."
>>> summary = "Lionel Andrés Messi (born 24 Aug 1997) is an Spanish professional footballer who plays as a forward and captains both Spanish club Barcelona and the Spanish national team."
>>> factsumm(article, summary, verbose=True)
<Source Entities>
Line No.1: [[('Lionel Andrés Messi', 'PERSON'), ('24 June 1987', 'DATE'), ('Argentine', 'NORP'), ('Spanish', 'NORP'), ('Barcelona', 'ORG'), ('Argentina', 'GPE')]]
Line No.2: [[('one', 'CARDINAL'), ('Messi', 'PERSON'), ('six', 'CARDINAL'), ("Ballon d'Or", 'WORK_OF_ART'), ('European Golden Shoes', 'WORK_OF_ART'), ('2020', 'DATE'), ("Ballon d'Or Dream Team", 'WORK_OF_ART')]]

<Summary Entities>
Line No.1: [[('Lionel Andrés Messi', 'PERSON'), ('24 Aug 1997', 'DATE'), ('Spanish', 'NORP'), ('Barcelona', 'ORG')]]

<Source Facts>
('Lionel Andrés Messi', 'per:origin', 'Argentine')
('Lionel Andrés Messi', 'per:employee_of', 'Barcelona')
('Barcelona', 'org:country_of_headquarters', 'Spanish')
('Lionel Andrés Messi', 'per:date_of_birth', '24 June 1987')
('Lionel Andrés Messi', 'per:countries_of_residence', 'Argentina')
('Spanish', 'org:top_members/employees', 'Lionel Andrés Messi')
('Barcelona', 'org:top_members/employees', 'Lionel Andrés Messi')

<Summary Facts>
('Lionel Andrés Messi', 'per:employee_of', 'Barcelona')
('Lionel Andrés Messi', 'per:date_of_birth', '24 Aug 1997')
('Barcelona', 'org:country_of_headquarters', 'Spanish')
('Spanish', 'org:top_members/employees', 'Lionel Andrés Messi')
('Barcelona', 'org:top_members/employees', 'Lionel Andrés Messi')
('Lionel Andrés Messi', 'per:origin', 'Spanish')
('Lionel Andrés Messi', 'per:countries_of_residence', 'Spanish')

<Common Facts>
('Barcelona', 'org:top_members/employees', 'Lionel Andrés Messi')
('Lionel Andrés Messi', 'per:employee_of', 'Barcelona')
('Barcelona', 'org:country_of_headquarters', 'Spanish')
('Spanish', 'org:top_members/employees', 'Lionel Andrés Messi')

<Diff Facts>
('Lionel Andrés Messi', 'per:date_of_birth', '24 Aug 1997')
('Lionel Andrés Messi', 'per:origin', 'Spanish')
('Lionel Andrés Messi', 'per:countries_of_residence', 'Spanish')

Fact Score: 0.5714285714285714
Answers based on Source (Questions are generated from Summary)
[Q] Who is the captain of the Spanish national team?	[Pred] <unanswerable>
[Q] When was Lionel Andrés Messi born?	[Pred] 24 June 1987
[Q] Lionel Andrés Messi is a professional footballer of what nationality?	[Pred] Argentine
[Q] Lionel Messi is a captain of which Spanish club?	[Pred] Barcelona

Answers based on Summary (Questions are generated from Summary)
[Q] Who is the captain of the Spanish national team?	[Pred] Lionel Andrés Messi
[Q] When was Lionel Andrés Messi born?	[Pred] 24 Aug 1997
[Q] Lionel Andrés Messi is a professional footballer of what nationality?	[Pred] Spanish
[Q] Lionel Messi is a captain of which Spanish club?	[Pred] Barcelona

QAGS Score: 0.3333333333333333

Avg. ROUGE-1: 0.4415584415584415
Avg. ROUGE-2: 0.3287671232876712
Avg. ROUGE-L: 0.4415584415584415
<BERTScore Score>
Precision: 0.9760397672653198
Recall: 0.9778039455413818
F1: 0.9769210815429688

You can use the GPU with the device. If you want to use GPU, pass cuda (default is cpu)

>>> factsumm(article, summary, device="cuda")

Sub-modules

From here, you can find various way to score Factual Consistency level with Unsupervised methods


Triple-based Module ( closed-scheme )

>>> from factsumm import FactSumm
>>> factsumm = FactSumm()
>>> factsumm.extract_facts(article, summary, verbose=True)
<Source Entities>
Line No.1: [[('Lionel Andrés Messi', 'PERSON'), ('24 June 1987', 'DATE'), ('Argentine', 'NORP'), ('Spanish', 'NORP'), ('Barcelona', 'ORG'), ('Argentina', 'GPE')]]
Line No.2: [[('one', 'CARDINAL'), ('Messi', 'PERSON'), ('six', 'CARDINAL'), ("Ballon d'Or", 'WORK_OF_ART'), ('European Golden Shoes', 'WORK_OF_ART'), ('2020', 'DATE'), ("Ballon d'Or Dream Team", 'WORK_OF_ART')]]

<Summary Entities>
Line No.1: [[('Lionel Andrés Messi', 'PERSON'), ('24 Aug 1997', 'DATE'), ('Spanish', 'NORP'), ('Barcelona', 'ORG')]]

<Source Facts>
('Lionel Andrés Messi', 'per:origin', 'Argentine')
('Lionel Andrés Messi', 'per:employee_of', 'Barcelona')
('Barcelona', 'org:country_of_headquarters', 'Spanish')
('Lionel Andrés Messi', 'per:date_of_birth', '24 June 1987')
('Lionel Andrés Messi', 'per:countries_of_residence', 'Argentina')
('Spanish', 'org:top_members/employees', 'Lionel Andrés Messi')
('Barcelona', 'org:top_members/employees', 'Lionel Andrés Messi')

<Summary Facts>
('Lionel Andrés Messi', 'per:employee_of', 'Barcelona')
('Lionel Andrés Messi', 'per:date_of_birth', '24 Aug 1997')
('Barcelona', 'org:country_of_headquarters', 'Spanish')
('Spanish', 'org:top_members/employees', 'Lionel Andrés Messi')
('Barcelona', 'org:top_members/employees', 'Lionel Andrés Messi')
('Lionel Andrés Messi', 'per:origin', 'Spanish')
('Lionel Andrés Messi', 'per:countries_of_residence', 'Spanish')

<Common Facts>
('Barcelona', 'org:top_members/employees', 'Lionel Andrés Messi')
('Lionel Andrés Messi', 'per:employee_of', 'Barcelona')
('Barcelona', 'org:country_of_headquarters', 'Spanish')
('Spanish', 'org:top_members/employees', 'Lionel Andrés Messi')

<Diff Facts>
('Lionel Andrés Messi', 'per:date_of_birth', '24 Aug 1997')
('Lionel Andrés Messi', 'per:origin', 'Spanish')
('Lionel Andrés Messi', 'per:countries_of_residence', 'Spanish')

Fact Score: 0.5714285714285714

The triple-based module counts the overlap of fact triples between the generated summary and the source document.


QA-based Module

If you ask questions about the summary and the source document, you will get a similar answer if the summary realistically matches the source document

>>> from factsumm import FactSumm
>>> factsumm = FactSumm()
>>> factsumm.extract_qas(article, summary, verbose=True)
Answers based on Source (Questions are generated from Summary)
[Q] Who is the captain of the Spanish national team?	[Pred] <unanswerable>
[Q] When was Lionel Andrés Messi born?	[Pred] 24 June 1987
[Q] Lionel Andrés Messi is a professional footballer of what nationality?	[Pred] Argentine
[Q] Lionel Messi is a captain of which Spanish club?	[Pred] Barcelona

Answers based on Summary (Questions are generated from Summary)
[Q] Who is the captain of the Spanish national team?	[Pred] Lionel Andrés Messi
[Q] When was Lionel Andrés Messi born?	[Pred] 24 Aug 1997
[Q] Lionel Andrés Messi is a professional footballer of what nationality?	[Pred] Spanish
[Q] Lionel Messi is a captain of which Spanish club?	[Pred] Barcelona

QAGS Score: 0.3333333333333333

ROUGE-based Module

>>> from factsumm import FactSumm
>>> factsumm = FactSumm()
>>> factsumm.calculate_rouge(article, summary)
Avg. ROUGE-1: 0.4415584415584415
Avg. ROUGE-2: 0.3287671232876712
Avg. ROUGE-L: 0.4415584415584415

Simple but effective word-level overlap ROUGE score


BERTScore Module

>>> from factsumm import FactSumm
>>> factsumm = FactSumm()
>>> factsumm.calculate_bert_score(article, summary)
<BERTScore Score>
Precision: 0.9760397672653198
Recall: 0.9778039455413818
F1: 0.9769210815429688

BERTScore can be used to calculate the similarity between each source sentence and the summary sentence


Citation

If you apply this library to any project, please cite:

@misc{factsumm,
  author       = {Heo, Hoon},
  title        = {FactSumm: Factual Consistency Scorer for Abstractive Summarization},
  howpublished = {\url{https://github.com/Huffon/factsumm}},
  year         = {2021},
}

References

Stats

Dependencies
7
Dependent packages
0
Dependent repositories
0
Total releases
6
Latest release
First release
Stars
102
Forks
8
Watchers
1
Contributors
2
Repository size
830 KB
SourceRank
10

Development practices

Source repo 2FA enabled
TEXT!
Package manager 2FA enabled
TEXT!
Is security responsive
TEXT!
Dependencies are managed
TEXT!
Issue-free release available
TEXT!
Succession plan available
TEXT!
Package manager 2FA enabled
TEXT!

The Tidelift Subscription provides access to a continuously curated stream of human-researched and maintainer-verified data on open source packages and their licenses, releases, vulnerabilities, and development practices.

Learn more →

Releases

0.1.5
0.1.4
0.1.2
0.1.1
0.1.0
0.0.1

Contributors

devfon karter-liner


See all contributors

Something wrong with this page? Make a suggestion

Export .ABOUT file for this package

Last synced: 2024-01-01 16:45:00 UTC

Login to resync this project