May 8 2020
May 7 2020
I added logging for the specifics of the OSError triggered when corrupt images identified. Will watch the logs closely to ensure it is capturing the sort of information that I am looking for. Hopefully this leads to some useful insight if another false positive is detected.
Restored activity, going to be adding more logging for corrupt images...want to know the type of errors being caught....
Turns out that hashes weren't an issue and were actually the change hash....that isn't the issue here by looks.
Bot shut down in meantime
It appears that the script is somehow saving the wrong hash values for every image. Investigating.
First step in triage is to revert away from using thumbnails for large image sizes.
Apr 21 2020
Apr 8 2020
Rcworker has since been rebuilt and this has not re-emerged since. Closing.
Mar 8 2020
Mar 5 2020
Single (new build) worker started on the backlog. Seeing if issues re-emerge before starting feeder & other workers. With a backlog of 490,252, I don't need to worry about it running out of stock at the moment.
Mar 3 2020
Feb 28 2020
Feb 27 2020
That was surprisingly quick. Upgraded and feeder/workers restarted. Total downtime for feeder (component that really matters for keeping up to date) was around 1 second (time it took me to Ctrl + C and then up arrow and hit enter).
Feb 24 2020
2020-02-24 08:42:16,590 __main__ : INFO File:�রপক্ষ মন্দির থেকে ভক্তদের বের হওয়াjpgশ্য..
Traceback (most recent call last):
File "rcworker.py", line 221, in <module>
main()
File "rcworker.py", line 214, in main
run_worker()
File "rcworker.py", line 63, in run_worker
file_page = pywikibot.FilePage(site, change.title)
File "/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 1744, in wrapper
return obj(*__args, **__kw)
File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 2478, in __init__
super(FilePage, self).__init__(source, title, 6)
File "/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 1744, in wrapper
return obj(*__args, **__kw)
File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 2478, in __init__
super(FilePage, self).__init__(source, title, 6)
File "/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 1744, in wrapper
return obj(*__args, **__kw)
File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 2327, in __init__
super(Page, self).__init__(source, title, ns)
File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 200, in __init__
self._link = Link(title, source=source, default_namespace=ns)
File "/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 1744, in wrapper
return obj(*__args, **__kw)
File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 6029, in __init__
raise pywikibot.Error(
pywikibot.exceptions.Error: Title contains illegal char (\uFFFD 'REPLACEMENT CHARACTER')Feb 19 2020
@AntiCompositeNumber it doesn't look like it caught the image in question, but it was whichever one was uploaded directly after File:PICT0430 - 301032 - onroerenderfgoed.jpg.
2020-02-18 18:45:16,662 __main__ : INFO File:PICT0430 - 301032 - onroerenderfgoed.jpg :Not corrupt. Stored
Traceback (most recent call last):
File "rcworker.py", line 221, in <module>
main()
File "rcworker.py", line 214, in main
run_worker()
File "rcworker.py", line 61, in run_worker
file_page = pywikibot.FilePage(site, change.title)
File "/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 1744, in wrapper
return obj(*__args, **__kw)
File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 2478, in __init__
super(FilePage, self).__init__(source, title, 6)
File "/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 1744, in wrapper
return obj(*__args, **__kw)
File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 2327, in __init__
super(Page, self).__init__(source, title, ns)
File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 200, in __init__
self._link = Link(title, source=source, default_namespace=ns)
File "/usr/local/lib/python3.8/site-packages/pywikibot/tools/__init__.py", line 1744, in wrapper
return obj(*__args, **__kw)
File "/usr/local/lib/python3.8/site-packages/pywikibot/page.py", line 6029, in __init__
raise pywikibot.Error(
pywikibot.exceptions.Error: Title contains illegal char (\uFFFD 'REPLACEMENT CHARACTER')Feb 16 2020
I got pywikibot to fail attempting to access
Feb 15 2020
Pushed to prod and workers/catalog scanner restarted with change.
Feb 14 2020
Happened again, crashing 2 or 3 of the 5 workers.
Continuously gets stuck at that entry. Further work needed to identify and address.
Feb 13 2020
Started 5 total workers to process the queue faster and am looking into automating their deployment dependent on queue size.