Page MenuHomePhabricator
Create Task
Maniphest T130

CCC: Tagging fixed images as corrupt
Closed, ResolvedPublic
Actions

Description

CCC's rcwatcher task has been tagging fixed images as corrupt (though not updating the database entries). As a result, I have shutdown the bot entirely pending investigation.

Some investigation revealed that the redis queue is currently 490,252 entries long and currently still processing uploads from Feb 12.

The crashes really hurt the bot's effectiveness. A new check needs to be added for cases like this and -- ideally -- an automated process developed to spin up (and down) workers dynamically.

Revisions and Commits

Restricted Diffusion Commit
Restricted Diffusion Commit

Event Timeline

TheSandDoctor triaged this task as Unbreak Now! priority.Mar 3 2020, 8:44 AM
TheSandDoctor created this task.
TheSandDoctor created this object with visibility "Public (No Login Required)".
TheSandDoctor updated the task description. (Show Details)
TheSandDoctor added a comment.EditedMar 5 2020, 10:42 AM

Single (new build) worker started on the backlog. Seeing if issues re-emerge before starting feeder & other workers. With a backlog of 490,252, I don't need to worry about it running out of stock at the moment.

During the downtime, I also updated to the latest version of pywikibot and now catch the InvalidTitle exception, but make it mega clear (tagged critical) in the logs so that I can easily observe when it happens again for that issue).

TheSandDoctor added a commit: Restricted Diffusion Commit.Mar 5 2020, 10:55 AM
TheSandDoctor closed this task as Resolved.Mar 8 2020, 5:54 PM
TheSandDoctor added a commit: Restricted Diffusion Commit.Apr 8 2020, 8:03 PM