The OSCAR project (Open Super-large Crawled Aggregated coRpus) is an Open Source project aiming to provide web-based multilingual resources and datasets for Machine Learning (ML) and Artificial Intelligence (AI) applications. The project focuses specifically in providing large quantities of unannotated raw data that is commonly used in the pre-training of large deep learning models. The OSCAR project has developed high-performance data pipelines specifically conceived to classify and filter large amounts of web data. The project has also put special attention in improving the data quality of web-based corpora as well as providing data for low-resource languages, so that these new ML/AI technologies are accessible to as many communities as possible.
Data is distributed by language in both original and deduplicated form. There are currently 166 different languages available. If you use OSCAR please consider giving us some feedback by writing to our mail address. Also consider citing our papers.
If you want to contribute to OSCAR please open a pull request!
Since 2019, The OSCAR Project has been funded by Inria (project-team ALMAnaCH), the PRAIRIE institute. Starting in 2023, DFKI and the German Federal Ministry for Economic Affairs and Climate Action (BMWK) through the project OpenGPT-X, have joined Inria, ALMAnaCH and the PRAIRIE institute in providing funding for the OSCAR Project. During 2022 and at the beginning of 2023, OSCAR was also shortly funded by The University of Mannheim.
If you are interested in OSCAR and would like to access the corpus, send us a mail using our mail address, with "OSCAR Access Request" as mail title. Please include your name, last name, affiliation, contact details, which languages do you need and a brief description of how you intend to use OSCAR.
Grab the latest OSCAR release here! 🚀
Join our Discord community here! 💬
