Gfilui

I have FSCKFIX=yes in /etc/default/rcS, however, when I drove down to the site and connected a monitor to this server today, I saw this:

I ran fsck.ext4 -fy /dev/sda1 and rebooted and now the server is back up and running.

I understand there are situations where just blindly running fsck -fy is dangerous and can lead to data loss. However, in my case, this is a physically inaccessible server and all the data is backed up in the cloud. It is paramount that Ubuntu does any and all steps possible to try to get it online.

Is there any way to force Ubuntu to always run fsck -fy, even if there are any "UNEXPECTED INCONSISTENCY" problems?

Let me preface this with a disclaimer that if you have regular issues with unclean filesystems, even though you shutdown your system cleanly, you have grave underlying problems and its possible fsck can do more harm than good !

AFAIK there is no mechanism for an automatic fsck only if inconsistencies are found.

However, you do an fsck at every boot with some kernel parameters.

sudo nano /etc/default/grub

find the line that says

GRUB_CMDLINE_LINUX_DEFAULT

and add

fsck.mode=force fsck.repair=yes

to the existing things there.

yes here should do the same as your FSCKFIX=yes in /etc/default/rcS or a manual fsck -fy. Personally I think preen would be safer, but then again it would hang on startup if the disk needs some more fixes that fsck does not deem "safe " and wants user interaction.

run

sudo update-grub

to update grup and verify it with

grep fsck /boot/grub/grub.cfg 

or have a look with an editor in /boot/grub/grub.cfg

If you then reboot, the filesystem should be checked, you can verify the last time it was checked (should be your boot time) with

sudo dumpe2fs -h /dev/your/device | grep checked
dumpe2fs 1.43.5 (04-Aug-2017)
Last checked: Sun Feb 18 08:53:31 2018