Start a second instance of rcworker.py. Given that they pop pages off of the redis queue, they wouldn't conflict and would always have work. This would allow them to speed up in the coverage of uploaded content. Currently, there is a several hours' worth of backlog for rcworker to clear through as it crashed at some point last night before I could restart it.
This will require some changes to logging as to avoid overwrites or edit conflicts there. Perhaps refactor code?
| Restricted Diffusion Commit | |
| Restricted Diffusion Commit | |
| Restricted Diffusion Commit |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T114 CCC: Identify processes to increase speed and files checked per minute | |||
| Resolved | TheSandDoctor | T117 CCC: Start second instance of rcworker.py |
Forward works, but backwards appears to get a empty page generator. Further investigation required.
Started 5 total workers to process the queue faster and am looking into automating their deployment dependent on queue size.