Parallel Monolingual Corpora: Difference between revisions
Jump to navigation
Jump to search
| (4 intermediate revisions by the same user not shown) | |||
| Line 11: | Line 11: | ||
*[https://spraakbanken.gu.se/en/resources/openedges OpenEdges Download] | *[https://spraakbanken.gu.se/en/resources/openedges OpenEdges Download] | ||
==ASSET Simplification Corpus== | ==Simplification Data== | ||
===ASSET Simplification Corpus=== | |||
The ASSET simplification corpus (Alva-Manchego et al, 2020) was automatically translated to Dutch (Seidl et al., 2023), and is freely available. | The ASSET simplification corpus (Alva-Manchego et al, 2020) was automatically translated to Dutch (Seidl et al., 2023), and is freely available. | ||
| Line 18: | Line 19: | ||
* <small>Seidl, T., Vandeghinste, V., & Van de Cruys, T. (2023). [https://kuleuven.limo.libis.be/discovery/fulldisplay?docid=alma9993527112601488&context=L&vid=32KUL_KUL:KULeuven&lang=en&search_scope=All_Content&adaptor=Local%20Search%20Engine&tab=all_content_tab&query=any,contains,seidl%20theresa&offset=0 Controllable Sentence Simplification in Dutch]. KU Leuven. Faculteit Ingenieurswetenschappen.</small> | * <small>Seidl, T., Vandeghinste, V., & Van de Cruys, T. (2023). [https://kuleuven.limo.libis.be/discovery/fulldisplay?docid=alma9993527112601488&context=L&vid=32KUL_KUL:KULeuven&lang=en&search_scope=All_Content&adaptor=Local%20Search%20Engine&tab=all_content_tab&query=any,contains,seidl%20theresa&offset=0 Controllable Sentence Simplification in Dutch]. KU Leuven. Faculteit Ingenieurswetenschappen.</small> | ||
==Wikilarge Dataset== | ===Wikilarge Dataset=== | ||
Automatic translation of the Wikilarge dataset, useful for automatic simplification (Seidl et al., 2023), freely available. Original dataset from Zhang & Lapata | Automatic translation of the Wikilarge dataset, useful for automatic simplification (Seidl et al., 2023), freely available. Original dataset from Zhang & Lapata | ||
| Line 24: | Line 25: | ||
* <small>Seidl, T., Vandeghinste, V., & Van de Cruys, T. (2023). [https://kuleuven.limo.libis.be/discovery/fulldisplay?docid=alma9993527112601488&context=L&vid=32KUL_KUL:KULeuven&lang=en&search_scope=All_Content&adaptor=Local%20Search%20Engine&tab=all_content_tab&query=any,contains,seidl%20theresa&offset=0 Controllable Sentence Simplification in Dutch]. KU Leuven. Faculteit Ingenieurswetenschappen.</small> | * <small>Seidl, T., Vandeghinste, V., & Van de Cruys, T. (2023). [https://kuleuven.limo.libis.be/discovery/fulldisplay?docid=alma9993527112601488&context=L&vid=32KUL_KUL:KULeuven&lang=en&search_scope=All_Content&adaptor=Local%20Search%20Engine&tab=all_content_tab&query=any,contains,seidl%20theresa&offset=0 Controllable Sentence Simplification in Dutch]. KU Leuven. Faculteit Ingenieurswetenschappen.</small> | ||
* <small>Zhang, X. & Lapata, M. (2017). Sentence Simplification with Deep Reinforcement Learning. In ''Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing'', pages 584–594, Copenhagen, Denmark. Association for Computational Linguistics.</small> | * <small>Zhang, X. & Lapata, M. (2017). Sentence Simplification with Deep Reinforcement Learning. In ''Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing'', pages 584–594, Copenhagen, Denmark. Association for Computational Linguistics.</small> | ||
===Comparable Corpus Wablieft De Standaard=== | |||
*[https://github.com/nivack/comparable_corpus_Wablieft_deStandaard Github] | |||
*[https://kuleuven.limo.libis.be/discovery/fulldisplay?docid=alma9993153812401488&context=L&vid=32KUL_KUL:KULeuven&lang=en&search_scope=All_Content&adaptor=Local%20Search%20Engine&tab=all_content_tab&query=any,contains,nick%20vanackere&offset=0 Vanackere, N., & Vandeghinste, V. (2022). Building a comparable corpus between easy-to-read Dutch Wablieft and De Standaard. KU Leuven. Faculteit Ingenieurswetenschappen.] | |||
===UWV Leesplank NL wikipedia=== | |||
The set contains 2,391,206 pragraphs of prompt/result combinations, where the prompt is a paragraph from Dutch Wikipedia and the result is a simplified text, which could include more than one paragraph. This dataset was created by UWV, as a part of project "Leesplank", an effort to generate datasets that are ethically and legally sound. | |||
* [https://huggingface.co/datasets/UWV/Leesplank_NL_wikipedia_simplifications/blob/main/README.md HuggingFace ReadMe file] | |||
*[https://huggingface.co/datasets/UWV/Leesplank_NL_wikipedia_simplifications Dataset] | |||
Latest revision as of 10:30, 20 March 2024
DAESO Corpus
The DAESO Corpus is a parallel monolingual treebank of Dutch texts and the corpus contains more than 2.1 million words of parallel and comparable text. About 678,000 words were lined up manually and about 1.5 million words were automatically aligned. A semantic relation was added to the aligned words / phrases.
- 92.5 MB
- version 1.0 (2010)
- Download page
Bible Corpus
A diachronically and synchronically parallel corpus of Bible translations in Dutch, English, German and Swedish, with texts from the 14th century until today.
Simplification Data
ASSET Simplification Corpus
The ASSET simplification corpus (Alva-Manchego et al, 2020) was automatically translated to Dutch (Seidl et al., 2023), and is freely available.
- Github download
- Alva-Manchego, F., Martin, L., Bordes, A., Scarton, C., Sagot, B., & Specia, L. (2020). ASSET: A dataset for tuning and evaluation of sentence simplification models with multiple rewriting transformations. arXiv preprint arXiv:2005.00481.
- Seidl, T., Vandeghinste, V., & Van de Cruys, T. (2023). Controllable Sentence Simplification in Dutch. KU Leuven. Faculteit Ingenieurswetenschappen.
Wikilarge Dataset
Automatic translation of the Wikilarge dataset, useful for automatic simplification (Seidl et al., 2023), freely available. Original dataset from Zhang & Lapata
- Github download
- Seidl, T., Vandeghinste, V., & Van de Cruys, T. (2023). Controllable Sentence Simplification in Dutch. KU Leuven. Faculteit Ingenieurswetenschappen.
- Zhang, X. & Lapata, M. (2017). Sentence Simplification with Deep Reinforcement Learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 584–594, Copenhagen, Denmark. Association for Computational Linguistics.
Comparable Corpus Wablieft De Standaard
- Github
- Vanackere, N., & Vandeghinste, V. (2022). Building a comparable corpus between easy-to-read Dutch Wablieft and De Standaard. KU Leuven. Faculteit Ingenieurswetenschappen.
UWV Leesplank NL wikipedia
The set contains 2,391,206 pragraphs of prompt/result combinations, where the prompt is a paragraph from Dutch Wikipedia and the result is a simplified text, which could include more than one paragraph. This dataset was created by UWV, as a part of project "Leesplank", an effort to generate datasets that are ethically and legally sound.