Now that we have resolved the issue, let's take a second to look at what we did to resolve the issue.
In order to troubleshoot our issue, we executed a scheduled cron job manually. If we circle back to previous chapters, this is a prime example of duplicating an issue and seeing it for ourselves.
In this case, the job was not performing the tasks it was supposed to. In order to identify the reason, we ran it manually.
During that manual execution, we were able to identify the following error:
IOError: [Errno 24] Too many open files: '/opt/myapp/queue/1433955823.29_0.txt'
This error is very common and is caused by the job running into user limits that prevent a single user from opening too many files. To resolve this we added custom settings to the /etc/security/limits.conf file.
These changes set the soft limitation of open files to 100000 for our user by default. We also allowed the user to increase the open files limit to 500000 on an ad-hoc basis via the hard...