Linux: Troubleshooting Filesystem Full issue due to open files

We face filesystem Full issue in our day to day Linux Administration.
This issue is mainly related with /var, /home filesystems. If both are in / filesystem then it will impact / filesystem as well. When we face this type of issue, our first solution is to delete or truncate log files, temp files, hidden files, unnecessary files from User’s home directory, etc. However sometime it will not solve the issue. There are chances of after deleting log files still “df” command show /var filesystem is 100% full, but when we check with “du” command, it shows the less usage of /var directory. One of the main reason behind this is deleted log files are still getting accessed by processes. Though they are not seen on filesystem /var, still they are in memory. To troubleshoot this type of issue, can try below steps:
Check filesystem usage.


df -h

If /var is showing 100% full then check actual usage of /var


du -h -x /var|tail -1   ---> It will show actual usage of  /var

If actual usage of /var is less than showing in “df -h” command then there must be issue with open log files which are deleted but still accessed by Process.

Check for deleted files which are getting accessed.


lsof +L1

lsof | grep "/var" | grep deleted

You will also get process id accessing that files. Just kill or stop these processes. And now check filesystem usage, it should show normal.

Neelesh Gurjar has written 122 articles

5 thoughts on “Linux: Troubleshooting Filesystem Full issue due to open files

  1. I have faced above issue many time during deployment of war inside tomcat container.
    It keeps log file attached with tomcat process and doesnot allows tomcat to startup again.

    Many time, that process also claim lot of memory with it. In one of the scenario it was holding around 20 GB of harddisk space taken by tomcat log files.

    Also, In one of the case, killing of the process doesnot work for freeing up the memory , we need to restart the node completely.

    Thanks,
    Rahul

  2. Ravi says:

    /dev/mapper/system–vg-opt–lv 372M 250M 0 100% /opt
    this is the file system issue any please suggest me what to do i had deleted old logs and gzip same logs
    but still showing 100% full

Leave a Reply