Read I/O error, ILLEGAL REQUEST

[dulcifium]

Not sure if that scan finished. Under disk test status it says "Suspend (system reset)". The system was not reset and the uptime is still several months. There's nothing in the logs about what might have happened (actually the logs are currently appearing blank but I can't see any email messages about the disk test or anything in the notifications tab on web interface).

Now nothing is responsive. Web interface barely works, no apps work, even logging in via ssh is so lagging it's been 15 mins since I entered password and I still have a blank screen. Three hours ago I got the following errors by email:

Code: Select all

 Level:  Warning
 Host: Disk 9 medium error. Please run a bad block scan on this drive
or replace it if the error persists.

 Level:  Warning
 Host: Disk 9 Read I/O error, UNRECOVERED READ ERROR sense_key=0x3,
asc=0x11, ascq=0x4, CDB=88 00 00 00 00 01 5f 2c 34 f0 00 00 00 08 00
00 .

 Level:  Warning
 Host: Disk 9 Read I/O error, ILLEGAL REQUEST sense_key=0x5, asc=0x21,
ascq=0x4, CDB=88 00 00 00 00 01 5f 2c 0b f8 00 00 02 c8 00 00 .

Level:  Warning
 Host: Disk 9 Read I/O error, ILLEGAL REQUEST sense_key=0x5, asc=0x21,
ascq=0x4, CDB=88 00 00 00 00 01 5f 2c 0a 08 00 00 01 20 00 00 .

I have the replacement disk ready but I can't even get into the web admin panel at the moment...

At this stage I'm wondering whether it's better to turn replace the disk without going through the proper steps?

[dulcifium]

I'm getting a load average of 160+ but nothing obvious showing in htop:

[Don]

Replace the bad drive following the procedure in the manual.

[dulcifium]

Replace the bad drive following the procedure in the manual.

I have checked the procedure in the manual but can't find it anywhere. The only information I can find is on this forum which says "leave it on, pull out the bad drive and replace without doing anything in admin panel".

However, the machine is still completely unresponsive and these huge load averages despite nothing showing in htop or top. Even the physical LCD display is lagging tons.

I'm terrified to put it in degraded mode while the CPU usage is so high. I'm also terrified to reboot.

No idea why on earth it's managed to break itself so badly just before I try to replace the drive. Why on earth would it have such high CPU load with nothing higher than 2%?!

Currently unable to get past password screen by ssh or even load login screen for web admin.

Do I have any option other than a hard reboot at this stage?

[P3R]

I'm terrified to put it in degraded mode while the CPU usage is so high. I'm also terrified to reboot.

Well neither is perfect but as the state of the NAS is abonormal there is no known right or wrong here. The system have since long indicated that disk 9 is faulty so I would hot swap that first and hope for the best.

If the situation doesn't improve in, say 6 hours, I don't see any other way out than a reboot. Then you'll have to take it from there.

[dulcifium]

Not even responding to ping now :/

Christ, wish me luck...

EDIT:

For the record, I crossed my fingers and pulled the disk. At this point the NAS was not accessible by the hardware buttons and LCD display, nor by ping or ssh or the web interface.

After about five minutes it beeped and noticed the disk was gone. Then the red LED came on the front panel and the LCD display came back to life!

I got the degraded notification on the LCD then inserted the new disk and hoped. There was no activity indicating a rebuild for over half an hour.

However, once it looked like the disk was recognised, I was able to ping the NAS again and log into it by ssh. Finally the CPU load had fallen from 250-300 down to a more "normal" 18.



I was even able to login to the web interface again! Here, the logs indicate that the new disk was finally recognised at some level after about ten minutes and the system started the rebuild after about half an hour.



CPU usage is still abnormally high but the rebuild seems to be going okay.

One odd thing I can't explain is that the ethernet connection seems to have dropped to 100Mb now instead of 1GbE. Any ideas?

[dulcifium]

UPDATE: Rebuild worked fine and system seems to be stable once again. My only assumption is something about the bad disk caused that huge CPU spike which almost brought the whole system down.

I'm still keeping an eye on the original bad disk.

Thanks P3R!

[P3R]

[dulcifium]

Woke up to the following warning:

Code: Select all

Type	Date	Time	Users	Source IP	Computer name	Content	
Warning	2018/12/09	04:59:01	System	127.0.0.1	localhost	Host: Disk 10 Read I/O error, ILLEGAL REQUEST sense_key=0x5, asc=0x21, ascq=0x4, CDB=88 00 00 00 00 01 32 55 39 c8 00 00 00 08 00 00 .

However, QNAP says this is just a "warning" not an "error". Is the disk okay?

Checking SMART I get this which seems the same as my other disks:

Code: Select all

[/] # get_hd_smartinfo -d 10
001 Raw_Read_Error_Rate          0        200 200 051 OK
003 Spin_Up_Time                 6625     167 167 021 OK
004 Start_Stop_Count             24       100 100 000 OK
005 Reallocated_Sector_Ct        0        200 200 140 OK
007 Seek_Error_Rate              0        200 200 000 OK
009 Power_On_Hours               7033     091 091 000 OK
010 Spin_Retry_Count             0        100 253 000 OK
011 Calibration_Retry_Count      0        100 253 000 OK
012 Power_Cycle_Count            7        100 100 000 OK
192 Power-Off_Retract_Count      4        200 200 000 OK
193 Load_Cycle_Count             59       200 200 000 OK
194 Temperature_Celsius          37       113 108 000 OK
196 Reallocated_Event_Count      0        200 200 000 OK
197 Current_Pending_Sector       0        200 200 000 OK
198 Offline_Uncorrectable        0        100 253 000 OK
199 UDMA_CRC_Error_Count         0        200 200 000 OK
200 Multi_Zone_Error_Rate        0        200 200 000 OK

Any insight?

So it's now been ten months with the original disk 10 that started this. I've since had 2 drives fail in quick succession so I'm hoping it's definitely not a hardware issue.

However, I've noticed that disk 10 still has very slow IOPS read values. All the over disks are getting >100 which is correct for a 7200rpm drive but for some reason it is lingering at 58 IPOS.

What could cause this?