Translations:Other corpora/37/en

From Clarin K-Centre
Revision as of 11:28, 21 March 2024 by FuzzyBot (talk | contribs) (Importing a new version from external source)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

CC-100 Corpus

This corpus is an attempt to recreate the dataset used for training XLM-R. This corpus comprises of monolingual data for 100+ languages and also includes data for romanized languages (indicated by *_rom). This was constructed using the urls and paragraph indices provided by the CC-Net repository by processing January-December 2018 Commoncrawl snapshots. Each file comprises of documents separated by double-newlines and paragraphs within the same document separated by a newline. The data is generated using the open source CC-Net repository. No claims of intellectual property are made on the work of preparation of the corpus. Dutch is one of the languages.

Retrieved from "https://kdutch.ivdnt.org/mediawiki/index.php?title=Translations:Other_corpora/37/en&oldid=7458"