We face filesystem Full issue in our day to day Linux Administration.
This issue is mainly related with /var, /home filesystems. If both are in / filesystem then it will impact / filesystem as well. When we face this type of issue, our first solution is to delete or truncate log files, temp files, hidden files, unnecessary files from User’s home directory, etc. However sometime it will not solve the issue. There are chances of after deleting log files still “df” command show /var filesystem is 100% full, but when we check with “du” command, it shows the less usage of /var directory. One of the main reason behind this is deleted log files are still getting accessed by processes. Though they are not seen on filesystem /var, still they are in memory. To troubleshoot this type of issue, can try below steps:
Check filesystem usage.
If /var is showing 100% full then check actual usage of /var
du -h -x /var|tail -1 ---> It will show actual usage of /var
If actual usage of /var is less than showing in “df -h” command then there must be issue with open log files which are deleted but still accessed by Process.
Check for deleted files which are getting accessed.
lsof +L1 lsof | grep "/var" | grep deleted
You will also get process id accessing that files. Just kill or stop these processes. And now check filesystem usage, it should show normal.