Sometimes you might encounter an error message like this:

Too many open files (os error 24)

Too many open files? That seems clear enough. A bit of googling—or prodding of your preferred LLM—will tell you that this is caused by the nofiles ulimit. Easy to solve. Except you then increase that limit to something huge like a million, and the error doesn’t go away.

There is another possibility: in your application you are watching files for changes, and you are hitting a different limit. File watching on Linux (generally) uses inotify internally, and that has its own limits (which are helpfully configured in a completely different place). Those limits are max_user_instances, max_user_watches and max_queued_events (see inotify(7) man for more details).

You can manually set these limits using sysctl. For example:

sysctl -w fs.inotify.max_user_instances=8192

I ran into this because max_user_instances defaults to 128(!) on NixOS. As an ivory tower-enjoying NixOS user, you will of course instead want to set your limits like this:

boot.kernel.sysctl = {
    "fs.inotify.max_user_instances" = "8192";
};