non raid filesystem suddenly gets mounted read only

[taxtax]

Hi,

I'm facing some weird issue which I don't understand, hopefully somebody overhere can point me in the right direction. I have TS-439 Pro with the following disks in it:

bay 1: Samsung HD204UI 1AQ1
bay 2: Samsung HD204UI 1AQ1
bay 3: WDC WD60EFRX-68MYMN182.0
bay 4: WDC WD60EFRX-68MYMN182.0

My firmware version is 4.1.2 Build 20150126 (most recent)

I don't use any form of raid, they are just single volumes, and all SMART status is Good / green.

I am running a manually installed Wordpress installation off the fourth disk (/dev/sdd3), with /share/Qweb as document root, which is mounted as /share/HDD_DATA:

Code: Select all

[/share/Qweb] # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/ramdisk            139.5M    113.2M     26.2M  81% /
tmpfs                    64.0M    348.0k     63.7M   1% /tmp
tmpfs                   502.8M      4.0k    502.8M   0% /dev/shm
/dev/sda4               371.0M    231.1M    139.9M  62% /mnt/ext
/dev/md9                509.5M    149.8M    359.7M  29% /mnt/HDA_ROOT
/dev/sda3                 1.8T      1.3T    545.1G  70% /share/HDA_DATA
/dev/sdb3                 1.8T    712.6G      1.1T  39% /share/HDB_DATA
/dev/sdc3                 5.4T      4.7T    689.2G  87% /share/HDC_DATA
/dev/sdd3                 5.4T      5.3T     33.7G  99% /share/HDD_DATA
[/share/Qweb] #

Code: Select all

[/share/Qweb] # mount
/proc on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
sysfs on /sys type sysfs (rw)
tmpfs on /tmp type tmpfs (rw,size=64M)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/bus/usb type usbfs (rw)
/dev/sda4 on /mnt/ext type ext3 (rw)
/dev/md9 on /mnt/HDA_ROOT type ext3 (rw,data=ordered)
/dev/sda3 on /share/HDA_DATA type ext4 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,nodelalloc,noacl)
/dev/sdb3 on /share/HDB_DATA type ext4 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,nodelalloc,noacl)
/dev/sdc3 on /share/HDC_DATA type ext4 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,nodelalloc,noacl)
/dev/sdd3 on /share/HDD_DATA type ext4 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,nodelalloc,noacl)
none on /sys/kernel/config type configfs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
[/share/Qweb] #

Every now and then though, seems like every 2 days, not really sure, suddenly the filesystem on disk /dev/sdd3 becomes read only, causing problems for the website I'm hosting on the NAS±

Code: Select all

[/share/Qweb] # pwd
/share/Qweb
[/share/Qweb] # ls -l
drwxrwxrwx    5 admin    administ      4096 Feb  6 15:52 blog/
-rw-r--r--    1 admin    administ  80373760 Feb  1 17:55 blog.tar
-rw-r--r--    1 admin    administ       423 Feb  1 17:23 index.php
-rw-r--r--    1 admin    administ      1139 Feb  1 17:22 index.php.org.die_hier_stond
lrwxrwxrwx    1 admin    administ        43 Jan 30 20:02 phpMyAdmin -> /share/HDD_DATA/.qpkg/phpMyAdmin/phpMyAdmin/
[/share/Qweb] # touch hello
touch: hello: Read-only file system <<--- ouch!
[/share/Qweb] #

I cannot find any error messages on the host, in for example /var/log/messages (which is empty) or in the QWeb Web Console under the messages tab. Does anybody have a clue as to why this is happening, and how to solve it? Or where to look for some logging? I can imagine this is some kind of precautionary measure taken by the NAS, but would like to know what triggers it, since I can test the disk without any issues from the web console, no errors reported, and SMART status is GOOD.

The only workaround that I currently know of, is to reboot the NAS, after that the filesystem is mounted as read/write again, but that's obviously not the most elegant of solutions

[pwilson]

/dev/sdd3 5.4T 5.3T 33.7G 99% /share/HDD_DATA

It is only read-only because it is "Full".

[taxtax]

Thank you very much sir, that indeed was the issue!

[taxtax]

hmm, the filesystem is now suddenly read only again..

Code: Select all

[/share/HDD_DATA/Qweb] # touch hello
touch: hello: Read-only file system
[/share/HDD_DATA/Qweb] # df -h .
Filesystem                Size      Used Available Use% Mounted on
/dev/sdd3                 5.4T      5.3T    122.4G  98% /share/HDD_DATA
[/share/HDD_DATA/Qweb] #

exactly how much free space / or what percentage triggers this behaviour? and is there any way to override / adjust it?

[pwilson]

hmm, the filesystem is now suddenly read only again..

Code: Select all

[/share/HDD_DATA/Qweb] # touch hello
touch: hello: Read-only file system
[/share/HDD_DATA/Qweb] # df -h .
Filesystem                Size      Used Available Use% Mounted on
/dev/sdd3                 5.4T      5.3T    122.4G  98% /share/HDD_DATA
[/share/HDD_DATA/Qweb] #

exactly how much free space / or what percentage triggers this behaviour? and is there any way to override / adjust it?

No Operating System likes working with "full" drives. Linux is no exception. Ensure you always have 5-10% free space on all drives. Shame you didn't implement RAID protection on your data, as this would have provided a nice easy to manage single volume, instead of having to manage it manually like this on multiple risky single drive volumes.

10TB of data on your 6TB drives, and no RAID redundancy..... you are brave. Why so much contempt for your data?

[taxtax]

Well, I couldn't afford to buy four 6 TB drives in a single go, just the two. All data is backupped on LTO4 tapes though, I'm not that much of a gambler

About the filesystem, I am quite familiair with working on Linux, but never have I seen a filesystem that becomes read only all of the sudden because it is almost full, so I'm thinking this is some QNAP related setting. Besides, there is over 120 GB free on this disk, that's not almost full to me. Is the QNAP software doing this, based on the percentage that the disk is used? And is there any way I can alter these tresholds?

Thanks again for your help.

edit: and if not possible to alter the treshold, how much would I have to free up for the QNAP software to not do this?

[taxtax]

bump..

[pwilson]

bump..

You can bump all you want but your drives are still full. The solutions are obvious.

Either:

1. Reduce the amount of data in your storage -or-
2. Increase the amount of storage available for your data.

Check out those percentages:

Code: Select all

df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/ramdisk            139.5M    113.2M     26.2M  81% /
tmpfs                    64.0M    348.0k     63.7M   1% /tmp
tmpfs                   502.8M      4.0k    502.8M   0% /dev/shm
/dev/sda4               371.0M    231.1M    139.9M  62% /mnt/ext
/dev/md9                509.5M    149.8M    359.7M  29% /mnt/HDA_ROOT
/dev/sda3                 1.8T      1.3T    545.1G  70% /share/HDA_DATA
/dev/sdb3                 1.8T    712.6G      1.1T  39% /share/HDB_DATA
/dev/sdc3                 5.4T      4.7T    689.2G  87% /share/HDC_DATA
/dev/sdd3                 5.4T      5.3T     33.7G  99% /share/HDD_DATA

This isn't about "thresholds", they are 87% & 99% full respectively.

[taxtax]

Well, this is only the first bump I did of this thread, regarding the "you can bump all you want" remark.

You copy pasted old information, this is the current status of the disk, it is 98% full and there is > 120 GB free space, as I posted earlier:

Code: Select all

[/share/HDD_DATA/Qweb] # touch hello
touch: hello: Read-only file system
[/share/HDD_DATA/Qweb] # df -h .
Filesystem                Size      Used Available Use% Mounted on
/dev/sdd3                 5.4T      5.3T    122.4G  98% /share/HDD_DATA
[/share/HDD_DATA/Qweb] #

And still it becomes read only after a while.

Are you telling me that it´s normal behaviour for Linux to remount a filesystem in read only mode, when it´s xx% full? Because if that is the case, I'm probably better off opening a case with support, hoping to find someone that *does* know what he's talking about.

[schumaku]

Can't see any poor this file system is mounted read-only ... this is your wild imagination, nobody said that. Well, except of the shell...

Some is reserved on the file system ...

[~] # tune2fs_64 -l /dev/hdd | grep 'lock count'
Block count: nnnnnnnnn
Reserved block count: nnnnnn

---and running a U**x-like file system at 98 or 99% is crazy. Troubles are predictable. Completely unrelated to QNAP.

If you start adding data to hdd, troubles will come back.

PS. Bumping is a waste of bandwidth and does reliably lower the motivation to help.

[taxtax]

Ok, so now I'm a waste of bandwith, I feel so very welcome at this forum I bumped the thread just *once"* when no reply was given after a week, come on guys, try and not be so grumpy.

You are right about it not being mounted read only, but still the kernel was blocking access to it, that's why I created this thread.

Your reasoning regarding the percentages makes no sense. Sure, when you have for example a 40 GB disk, it being > 98% full is definitely an issue, but in this case there is over 120 GB of free space available. Also, there are plenty of free blocks, besides those reserved for the system:

Code: Select all

[/share/HDD_DATA] # tune2fs -l /dev/sdd3 |grep -i lock
Block count:              1464738423
Reserved block count:     131072
Free blocks:              31769974
First block:              0
Block size:               4096
Reserved GDT blocks:      674
Blocks per group:         32768
Inode blocks per group:   512
Flex block group size:    16
Reserved blocks uid:      0 (user admin)
Reserved blocks gid:      0 (group administrators)
Journal backup:           inode blocks
[/share/HDD_DATA] #

Just in case anyone runs into the same issue, I managed to solve this by manually running a filesystem check, after first stopping all services calling the /etc/init.d/services.sh script, and then unmounting the disk.

There were some warnings in the GUI regarding this disk which I hadn't noticed before, and although I had initially repaired the disk twice using the GUI, this apparently did not fix the issue since it kept reappearing. But running the filesystem check manually did, since all has been good now for over a week (unless the disk is broken of course, but that would be a completely different problem).

[pwilson]

Ok, so now I'm a waste of bandwith,

That isn't what he said. He said "Bumping" was a waste of Bandwidth.

You aren't listening to what we are telling you anyway, so this discussion is becoming a waste of bandwidth. We have identified the issue for you, and you are choosing to ignore the advice provided. (Which is your right to do).

Good luck with your data. Enjoy it while it lasts..... I'd recommend maintaining very regular Backups if you intend to keep your drives this "full" going forward. You will continue to have issues until you listen to our advice, but it is your data, your NAS, and your time, so you can do anything you want. Anyway, there is nothing positive I can add to this thread, that I haven't already, so I'll leave you to manage your NAS in any fashion you deem appropriate for your data.

[taxtax]

> We have identified the issue for you

Wow, this is just not true! If you won't take it from me, which you obviously won't: I've verified this with several people in my company who work professionally as UNIX / Linux administrators. You are talking nonsense. There is absolutely no reason for suddenly not being able to write to the filesystem anymore, when there is over 120 GB of free space left on it (besides it needing a e2fsck obviously, THAT is the reason why I got the read only filesystem error).

And stop lecturing me about making backups, I have those as I already wrote (and yes, I am also able to restore those backups).

But please, keep on believing that what you are saying is the truth, and that I'm just some guy that has no clue what he's talking about that won't listen to your (so called) solution, since this has been your attitude towards me from the start.

Last time I ever created a thread on this forum btw, but I'm sure you won't mind, as won't I.

[pwilson]

> We have identified the issue for you

Wow, this is just not true! If you won't take it from me, which you obviously won't: I've verified this with several people in my company who work professionally as UNIX / Linux administrators. You are talking nonsense. There is absolutely no reason for suddenly not being able to write to the filesystem anymore, when there is over 120 GB of free space left on it (besides it needing a e2fsck obviously, THAT is the reason why I got the read only filesystem error).

And stop lecturing me about making backups, I have those as I already wrote (and yes, I am also able to restore those backups).

But please, keep on believing that what you are saying is the truth, and that I'm just some guy that has no clue what he's talking about that won't listen to your (so called) solution, since this has been your attitude towards me from the start.

Last time I ever created a thread on this forum btw, but I'm sure you won't mind, as won't I.

Your HR Dept needs to find better UNIX / Linux administrators then, as they are clueless if they really told you this fantasy. Your drives are full whether you want to accept it or not. Get your company's "professional" UNIX / Linux administrators to fix this for you, if you really believe we are wrong.

Whether you choose to participate in this Forum is your choice. If you can't handle it, then by all means please feel free to not post here again. Your participation here is pointless anyway, as you aren't listening when answers are provided. Your drives are not mounted "read-only". If you don't believe me, simply simply access the command line of your NAS and run:

Code: Select all

mount

You will discover that none of the filesystems are mounted read-only. I'm sorry reality offends you. I have no attitude toward anyone. You don't like what I'm telling you, so you think I have attitude. Reality is that you don't like my answer, but you'd rather attack me that actually fix the underlying issue. If you don't like what I have to say, please simply ignore me. This won't fix anything, but you'll sleep better.