Currently running at a rate of 800 files per half hour, which will require 4.2 years to check all files.
Based on these numbers, it seems appropriate to identify and implement processes to increase speed to reduce the duration needed to check all files.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T114 CCC: Identify processes to increase speed and files checked per minute | |||
| Resolved | TheSandDoctor | T115 CCC: Run CCC script a second time, but in reverse order? | |||
| Resolved | TheSandDoctor | T117 CCC: Start second instance of rcworker.py |
It's hard to know exactly what is slowing things down without a profile. @TheSandDoctor, could you run corrupt.py through CProfile? I'd expect the largest source of delay to come from three places:
I don't know much about pillow, so I can't speak to how fast it is or how to speed it up. There are two ways to speed up network processes: bypassing it and parallelization. You could use a database connection to get page data instead of relying on PWB and going out over the network, which could speed things up a tiny bit. That effect, unfortunately, isn't super significant in the face of trying to download the often-large Commons files.
You could try checking a smaller thumbnail of the image for corruption instead of the full size, which would download faster.
Running a second process is probably the best way to speed things up overall.