pipeline for pre-processing warc files from CommonCrawl
initial inspiration came from https://arxiv.org/pdf/2306.01116.pdf (The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only)
-
Notifications
You must be signed in to change notification settings - Fork 0
pipeline for pre-processing warc files from CommonCrawl
License
TPC-AI/warc_processing
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
pipeline for pre-processing warc files from CommonCrawl
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published