Simplification Data: Difference between revisions

From Clarin K-Centre
Jump to navigation Jump to search
VisualWikitext
(Created page with "==ASSET Simplification Corpus== The ASSET simplification corpus (Alva-Manchego et al, 2020) was automatically translated to Dutch (Seidl et al., 2023), and is freely available. *[https://github.com/tsei902/simplify_dutch/tree/main/resources/datasets/asset Github download] * <small>Alva-Manchego, F., Martin, L., Bordes, A., Scarton, C., Sagot, B., & Specia, L. (2020). ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting tra...")
 
No edit summary
 
(12 intermediate revisions by 2 users not shown)
Line 1: Line 1:
==ASSET Simplification Corpus==
<languages/>
The ASSET simplification corpus (Alva-Manchego et al, 2020) was automatically translated to Dutch (Seidl et al., 2023), and is freely available.
<translate>


==Automatically translated datasets== <!--T:1-->
===ASSET Simplification Corpus=== <!--T:2-->
<!--T:3-->
The Abstractive Sentence Simplification Evaluation and Tuning (ASSET) Dataset (Alva-Manchego et al, 2020) was automatically translated to Dutch (Seidl et al., 2023), and is freely available.
<!--T:4-->
*[https://github.com/tsei902/simplify_dutch/tree/main/resources/datasets/asset Github download]
*[https://github.com/tsei902/simplify_dutch/tree/main/resources/datasets/asset Github download]
* <small>Alva-Manchego, F., Martin, L., Bordes, A., Scarton, C., Sagot, B., & Specia, L. (2020). ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. arXiv preprint arXiv:2005.00481.</small>
* <small>Alva-Manchego, F., Martin, L., Bordes, A., Scarton, C., Sagot, B., & Specia, L. (2020). ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. arXiv preprint arXiv:2005.00481.</small>
* <small>Seidl, T., Vandeghinste, V., & Van de Cruys, T. (2023). [https://kuleuven.limo.libis.be/discovery/fulldisplay?docid=alma9993527112601488&context=L&vid=32KUL_KUL:KULeuven&lang=en&search_scope=All_Content&adaptor=Local%20Search%20Engine&tab=all_content_tab&query=any,contains,seidl%20theresa&offset=0 Controllable Sentence Simplification in Dutch]. KU Leuven. Faculteit Ingenieurswetenschappen.</small>
* <small>Seidl, T., Vandeghinste, V., & Van de Cruys, T. (2023). [https://kuleuven.limo.libis.be/discovery/fulldisplay?docid=alma9993527112601488&context=L&vid=32KUL_KUL:KULeuven&lang=en&search_scope=All_Content&adaptor=Local%20Search%20Engine&tab=all_content_tab&query=any,contains,seidl%20theresa&offset=0 Controllable Sentence Simplification in Dutch]. KU Leuven. Faculteit Ingenieurswetenschappen.</small>


==Wikilarge Dataset==
===Wikilarge Dataset=== <!--T:5-->
 
<!--T:6-->
Automatic translation of the Wikilarge dataset, useful for automatic simplification (Seidl et al., 2023), freely available. Original dataset from Zhang & Lapata
Automatic translation of the Wikilarge dataset, useful for automatic simplification (Seidl et al., 2023), freely available. Original dataset from Zhang & Lapata


<!--T:7-->
*[https://github.com/tsei902/simplify_dutch/tree/main/resources/datasets/wikilarge Github download]
*[https://github.com/tsei902/simplify_dutch/tree/main/resources/datasets/wikilarge Github download]
* <small>Seidl, T., Vandeghinste, V., & Van de Cruys, T. (2023). [https://kuleuven.limo.libis.be/discovery/fulldisplay?docid=alma9993527112601488&context=L&vid=32KUL_KUL:KULeuven&lang=en&search_scope=All_Content&adaptor=Local%20Search%20Engine&tab=all_content_tab&query=any,contains,seidl%20theresa&offset=0 Controllable Sentence Simplification in Dutch]. KU Leuven. Faculteit Ingenieurswetenschappen.</small>
* <small>Seidl, T., Vandeghinste, V., & Van de Cruys, T. (2023). [https://kuleuven.limo.libis.be/discovery/fulldisplay?docid=alma9993527112601488&context=L&vid=32KUL_KUL:KULeuven&lang=en&search_scope=All_Content&adaptor=Local%20Search%20Engine&tab=all_content_tab&query=any,contains,seidl%20theresa&offset=0 Controllable Sentence Simplification in Dutch]. KU Leuven. Faculteit Ingenieurswetenschappen.</small>
* <small>Zhang, X. & Lapata, M. (2017). Sentence Simplification with Deep Reinforcement Learning. In ''Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing'', pages 584–594, Copenhagen, Denmark. Association for Computational Linguistics.</small>
* <small>Zhang, X. & Lapata, M. (2017). Sentence Simplification with Deep Reinforcement Learning. In ''Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing'', pages 584–594, Copenhagen, Denmark. Association for Computational Linguistics.</small>


==Comparable Corpus Wablieft De Standaard==
===NFI SimpleWiki dataset=== <!--T:8-->
 
<!--T:9-->
Translated dataset created by Netherlands Forensic Institute using Meta's [https://ai.meta.com/research/no-language-left-behind/ No Language Left Behind model]. It comprises 167,000 aligned sentence pairs and serves as a Dutch translation of the SimpleWiki [https://cs.pomona.edu/~dkauchak/simplification/ dataset].
 
<!--T:10-->
* 8.67 MB
* [https://huggingface.co/datasets/NetherlandsForensicInstitute/simplewiki-translated-nl Download dataset]
 
==Comparable Corpus Wablieft De Standaard== <!--T:11-->
 
<!--T:12-->
Corpus created by Nick Vanackere. It contains a comparable corpus of 12,687 Wablieft articles between 2012-2017 from 206,466 De Standaard articles from 2013-2017. To ensure comparability, only articles from 08/01/2013 till 16/11/2017 were considered, resulting in 8,744 Wablieft articles and 202,284 De Standaard articles. The difference in the number of articles is due to the publication frequency, with Wablieft being weekly and De Standaard daily.
 
<!--T:13-->
*[https://github.com/nivack/comparable_corpus_Wablieft_deStandaard Github]
*[https://github.com/nivack/comparable_corpus_Wablieft_deStandaard Github]
*[https://kuleuven.limo.libis.be/discovery/fulldisplay?docid=alma9993153812401488&context=L&vid=32KUL_KUL:KULeuven&lang=en&search_scope=All_Content&adaptor=Local%20Search%20Engine&tab=all_content_tab&query=any,contains,nick%20vanackere&offset=0 Vanackere, N., & Vandeghinste, V. (2022). Building a comparable corpus between easy-to-read Dutch Wablieft and De Standaard. KU Leuven. Faculteit Ingenieurswetenschappen.]
*<small>[https://kuleuven.limo.libis.be/discovery/fulldisplay?docid=alma9993153812401488&context=L&vid=32KUL_KUL:KULeuven&lang=en&search_scope=All_Content&adaptor=Local%20Search%20Engine&tab=all_content_tab&query=any,contains,nick%20vanackere&offset=0 Vanackere, N., & Vandeghinste, V. (2022). Building a comparable corpus between easy-to-read Dutch Wablieft and De Standaard. KU Leuven. Faculteit Ingenieurswetenschappen.]</small>
 
==Synthetic datasets== <!--T:14-->
 
===UWV Leesplank NL wikipedia=== <!--T:15-->


==UWV Leesplank NL wikipedia==
<!--T:16-->
The set contains 2,391,206 pragraphs of prompt/result combinations, where the prompt is a paragraph from Dutch Wikipedia and the result is a simplified text, which could include more than one paragraph. This dataset was created by UWV, as a part of project "Leesplank", an effort to generate datasets that are ethically and legally sound.
The set contains 2,391,206 pragraphs of prompt/result combinations, where the prompt is a paragraph from Dutch Wikipedia and the result is a simplified text, which could include more than one paragraph. This dataset was created by UWV, as a part of project "Leesplank", an effort to generate datasets that are ethically and legally sound.


<!--T:17-->
* [https://huggingface.co/datasets/UWV/Leesplank_NL_wikipedia_simplifications/blob/main/README.md HuggingFace ReadMe file]
* [https://huggingface.co/datasets/UWV/Leesplank_NL_wikipedia_simplifications/blob/main/README.md HuggingFace ReadMe file]


<!--T:18-->
*[https://huggingface.co/datasets/UWV/Leesplank_NL_wikipedia_simplifications Dataset]
*[https://huggingface.co/datasets/UWV/Leesplank_NL_wikipedia_simplifications Dataset]
<!--T:19-->
A more extended version of this dataset was made by Michiel Buisman and Bram Vanroy. This datasets contains a first, small set of variations of Wikipedia paragraphs in different styles (jargon, official, archaïc language, technical, academic, and poetic).
<!--T:20-->
* 3.02 MB
* [https://huggingface.co/datasets/UWV/veringewikkelderingen Download page]
===ChatGPT generated dataset by Van de Velde=== <!--T:21-->
<!--T:22-->
Created in light of a master thesis by Charlotte Van de Velde. The dataset contains Dutch source sentences and aligned simplified sentences, generated with ChatGPT. All splits combined, the dataset consists of 1267 entries.
<!--T:23-->
# Training = 1013 sentences (262 KB)
# Validation = 126 sentences (32.6 KB)
# Test = 128 sentences (33 KB)
<!--T:24-->
* [https://huggingface.co/datasets/BramVanroy/chatgpt-dutch-simplification Download page (CSV files)]
==Manually simplified== <!--T:25-->
===Dutch municipal data=== <!--T:26-->
<!--T:27-->
The Dutch municipal corpus is a parallel monolingual corpus for the evaluation of sentence-level simplification in the Dutch municipal domain. The corpus was created by Amsterdam Intelligence. It contains 1,311 translated parallel sentence pairs that were automatically aligned. The sentence pairs originate from 50 documents from the Communications Department of the City of Amsterdam that were manually simplified to evaluate simplification for Dutch. 
<!--T:28-->
*265 KB
*[https://github.com/Amsterdam-AI-Team/dutch-municipal-text-simplification Github]
</translate>

Latest revision as of 11:38, 2 September 2024

Other languages:

Automatically translated datasets

ASSET Simplification Corpus

The Abstractive Sentence Simplification Evaluation and Tuning (ASSET) Dataset (Alva-Manchego et al, 2020) was automatically translated to Dutch (Seidl et al., 2023), and is freely available.

  • Github download
  • Alva-Manchego, F., Martin, L., Bordes, A., Scarton, C., Sagot, B., & Specia, L. (2020). ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. arXiv preprint arXiv:2005.00481.
  • Seidl, T., Vandeghinste, V., & Van de Cruys, T. (2023). Controllable Sentence Simplification in Dutch. KU Leuven. Faculteit Ingenieurswetenschappen.

Wikilarge Dataset

Automatic translation of the Wikilarge dataset, useful for automatic simplification (Seidl et al., 2023), freely available. Original dataset from Zhang & Lapata

  • Github download
  • Seidl, T., Vandeghinste, V., & Van de Cruys, T. (2023). Controllable Sentence Simplification in Dutch. KU Leuven. Faculteit Ingenieurswetenschappen.
  • Zhang, X. & Lapata, M. (2017). Sentence Simplification with Deep Reinforcement Learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 584–594, Copenhagen, Denmark. Association for Computational Linguistics.

NFI SimpleWiki dataset

Translated dataset created by Netherlands Forensic Institute using Meta's No Language Left Behind model. It comprises 167,000 aligned sentence pairs and serves as a Dutch translation of the SimpleWiki dataset.

Comparable Corpus Wablieft De Standaard

Corpus created by Nick Vanackere. It contains a comparable corpus of 12,687 Wablieft articles between 2012-2017 from 206,466 De Standaard articles from 2013-2017. To ensure comparability, only articles from 08/01/2013 till 16/11/2017 were considered, resulting in 8,744 Wablieft articles and 202,284 De Standaard articles. The difference in the number of articles is due to the publication frequency, with Wablieft being weekly and De Standaard daily.

Synthetic datasets

UWV Leesplank NL wikipedia

The set contains 2,391,206 pragraphs of prompt/result combinations, where the prompt is a paragraph from Dutch Wikipedia and the result is a simplified text, which could include more than one paragraph. This dataset was created by UWV, as a part of project "Leesplank", an effort to generate datasets that are ethically and legally sound.

A more extended version of this dataset was made by Michiel Buisman and Bram Vanroy. This datasets contains a first, small set of variations of Wikipedia paragraphs in different styles (jargon, official, archaïc language, technical, academic, and poetic).

ChatGPT generated dataset by Van de Velde

Created in light of a master thesis by Charlotte Van de Velde. The dataset contains Dutch source sentences and aligned simplified sentences, generated with ChatGPT. All splits combined, the dataset consists of 1267 entries.

  1. Training = 1013 sentences (262 KB)
  2. Validation = 126 sentences (32.6 KB)
  3. Test = 128 sentences (33 KB)


Manually simplified

Dutch municipal data

The Dutch municipal corpus is a parallel monolingual corpus for the evaluation of sentence-level simplification in the Dutch municipal domain. The corpus was created by Amsterdam Intelligence. It contains 1,311 translated parallel sentence pairs that were automatically aligned. The sentence pairs originate from 50 documents from the Communications Department of the City of Amsterdam that were manually simplified to evaluate simplification for Dutch.


Retrieved from "https://kdutch.ivdnt.org/mediawiki/index.php?title=Simplification_Data&oldid=10674"