Read I/O error, ILLEGAL REQUEST

Questions about SNMP, Power, System, Logs, disk, & RAID.
dulcifium
Been there, done that
Posts: 689
Joined: Fri Oct 23, 2009 6:52 pm

Re: Read I/O error, ILLEGAL REQUEST

Post by dulcifium » Wed Feb 20, 2019 7:57 pm

Not sure if that scan finished. Under disk test status it says "Suspend (system reset)". The system was not reset and the uptime is still several months. There's nothing in the logs about what might have happened (actually the logs are currently appearing blank but I can't see any email messages about the disk test or anything in the notifications tab on web interface).

Now nothing is responsive. Web interface barely works, no apps work, even logging in via ssh is so lagging it's been 15 mins since I entered password and I still have a blank screen. Three hours ago I got the following errors by email:

Code: Select all

 Level:  Warning
 Host: Disk 9 medium error. Please run a bad block scan on this drive
or replace it if the error persists.

 Level:  Warning
 Host: Disk 9 Read I/O error, UNRECOVERED READ ERROR sense_key=0x3,
asc=0x11, ascq=0x4, CDB=88 00 00 00 00 01 5f 2c 34 f0 00 00 00 08 00
00 .

 Level:  Warning
 Host: Disk 9 Read I/O error, ILLEGAL REQUEST sense_key=0x5, asc=0x21,
ascq=0x4, CDB=88 00 00 00 00 01 5f 2c 0b f8 00 00 02 c8 00 00 .

Level:  Warning
 Host: Disk 9 Read I/O error, ILLEGAL REQUEST sense_key=0x5, asc=0x21,
ascq=0x4, CDB=88 00 00 00 00 01 5f 2c 0a 08 00 00 01 20 00 00 .
I have the replacement disk ready but I can't even get into the web admin panel at the moment...

At this stage I'm wondering whether it's better to turn replace the disk without going through the proper steps?
Last edited by dulcifium on Wed Feb 20, 2019 8:50 pm, edited 2 times in total.
QNAP TS-EC1079 Pro (4.3.4.0435) | 10 x 4 TB (WD WDC WD40EFRX firmware 82.00A82) | 32 TB RAID-6
QNAP TS-639 Pro (3.8.1 Build 20121205) | 6 x 3 TB (Toshiba DT01ACA300 firmware MX6OABB0) | 12 TB RAID-6
Both shared over single port GbE and wifi router to Linux, Android, Windows, macOS and PS3

dulcifium
Been there, done that
Posts: 689
Joined: Fri Oct 23, 2009 6:52 pm

Re: Read I/O error, ILLEGAL REQUEST

Post by dulcifium » Wed Feb 20, 2019 8:45 pm

I'm getting a load average of 160+ but nothing obvious showing in htop:

Image
QNAP TS-EC1079 Pro (4.3.4.0435) | 10 x 4 TB (WD WDC WD40EFRX firmware 82.00A82) | 32 TB RAID-6
QNAP TS-639 Pro (3.8.1 Build 20121205) | 6 x 3 TB (Toshiba DT01ACA300 firmware MX6OABB0) | 12 TB RAID-6
Both shared over single port GbE and wifi router to Linux, Android, Windows, macOS and PS3

User avatar
Don
Guru
Posts: 11621
Joined: Thu Jan 03, 2008 4:56 am
Location: Long Island, New York
Contact:

Re: Read I/O error, ILLEGAL REQUEST

Post by Don » Wed Feb 20, 2019 10:35 pm

Replace the bad drive following the procedure in the manual.
Read the Online Manuals and use the forum search feature before posting.

It is a recommended to use RAID and have external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.

Submit bugs and feature requests to QNAP via their Helpdesk app.

NAS: TVS-882BR | F/W: 4.3.6.0805 | 40GB | 2 x M.2 SATA RAID 1 (System/VMs) | 4 x M.2 NMVe QM2-4P-384A RAID 5 (Cache) | 5 x 4TB HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-663 | F/W: 4.3.6.0883 | 16GB | 2 x M.2 NMVe QM2-2P RAID 1 (Cache) | 4 x 4TB RAID 5
Apps: Boinc, Squid, DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS, Entware, DLstation, +others

dulcifium
Been there, done that
Posts: 689
Joined: Fri Oct 23, 2009 6:52 pm

Re: Read I/O error, ILLEGAL REQUEST

Post by dulcifium » Wed Feb 20, 2019 10:37 pm

Don wrote:
Wed Feb 20, 2019 10:35 pm
Replace the bad drive following the procedure in the manual.
I have checked the procedure in the manual but can't find it anywhere. The only information I can find is on this forum which says "leave it on, pull out the bad drive and replace without doing anything in admin panel".

However, the machine is still completely unresponsive and these huge load averages despite nothing showing in htop or top. Even the physical LCD display is lagging tons.

I'm terrified to put it in degraded mode while the CPU usage is so high. I'm also terrified to reboot.

No idea why on earth it's managed to break itself so badly just before I try to replace the drive. Why on earth would it have such high CPU load with nothing higher than 2%?!

Currently unable to get past password screen by ssh or even load login screen for web admin.

Do I have any option other than a hard reboot at this stage?
QNAP TS-EC1079 Pro (4.3.4.0435) | 10 x 4 TB (WD WDC WD40EFRX firmware 82.00A82) | 32 TB RAID-6
QNAP TS-639 Pro (3.8.1 Build 20121205) | 6 x 3 TB (Toshiba DT01ACA300 firmware MX6OABB0) | 12 TB RAID-6
Both shared over single port GbE and wifi router to Linux, Android, Windows, macOS and PS3

P3R
Guru
Posts: 10584
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Read I/O error, ILLEGAL REQUEST

Post by P3R » Wed Feb 20, 2019 11:08 pm

dulcifium wrote:
Wed Feb 20, 2019 10:37 pm
I'm terrified to put it in degraded mode while the CPU usage is so high. I'm also terrified to reboot.
Well neither is perfect but as the state of the NAS is abonormal there is no known right or wrong here. The system have since long indicated that disk 9 is faulty so I would hot swap that first and hope for the best.

If the situation doesn't improve in, say 6 hours, I don't see any other way out than a reboot. Then you'll have to take it from there.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

dulcifium
Been there, done that
Posts: 689
Joined: Fri Oct 23, 2009 6:52 pm

Re: Read I/O error, ILLEGAL REQUEST

Post by dulcifium » Wed Feb 20, 2019 11:18 pm

Not even responding to ping now :/

Christ, wish me luck...

EDIT:

For the record, I crossed my fingers and pulled the disk. At this point the NAS was not accessible by the hardware buttons and LCD display, nor by ping or ssh or the web interface.

After about five minutes it beeped and noticed the disk was gone. Then the red LED came on the front panel and the LCD display came back to life!

I got the degraded notification on the LCD then inserted the new disk and hoped. There was no activity indicating a rebuild for over half an hour.

However, once it looked like the disk was recognised, I was able to ping the NAS again and log into it by ssh. Finally the CPU load had fallen from 250-300 down to a more "normal" 18.

Image

I was even able to login to the web interface again! Here, the logs indicate that the new disk was finally recognised at some level after about ten minutes and the system started the rebuild after about half an hour.

Image

CPU usage is still abnormally high but the rebuild seems to be going okay.

One odd thing I can't explain is that the ethernet connection seems to have dropped to 100Mb now instead of 1GbE. Any ideas?
QNAP TS-EC1079 Pro (4.3.4.0435) | 10 x 4 TB (WD WDC WD40EFRX firmware 82.00A82) | 32 TB RAID-6
QNAP TS-639 Pro (3.8.1 Build 20121205) | 6 x 3 TB (Toshiba DT01ACA300 firmware MX6OABB0) | 12 TB RAID-6
Both shared over single port GbE and wifi router to Linux, Android, Windows, macOS and PS3

dulcifium
Been there, done that
Posts: 689
Joined: Fri Oct 23, 2009 6:52 pm

Re: Read I/O error, ILLEGAL REQUEST

Post by dulcifium » Thu Feb 21, 2019 10:23 pm

UPDATE: Rebuild worked fine and system seems to be stable once again. My only assumption is something about the bad disk caused that huge CPU spike which almost brought the whole system down.

I'm still keeping an eye on the original bad disk.

Thanks P3R! :)
Last edited by dulcifium on Fri Feb 22, 2019 5:48 am, edited 1 time in total.
QNAP TS-EC1079 Pro (4.3.4.0435) | 10 x 4 TB (WD WDC WD40EFRX firmware 82.00A82) | 32 TB RAID-6
QNAP TS-639 Pro (3.8.1 Build 20121205) | 6 x 3 TB (Toshiba DT01ACA300 firmware MX6OABB0) | 12 TB RAID-6
Both shared over single port GbE and wifi router to Linux, Android, Windows, macOS and PS3

P3R
Guru
Posts: 10584
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Read I/O error, ILLEGAL REQUEST

Post by P3R » Fri Feb 22, 2019 4:38 am

:D
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

dulcifium
Been there, done that
Posts: 689
Joined: Fri Oct 23, 2009 6:52 pm

Re: Read I/O error, ILLEGAL REQUEST

Post by dulcifium » Fri Mar 01, 2019 11:01 pm

dulcifium wrote:
Mon Dec 10, 2018 4:53 am
Woke up to the following warning:

Code: Select all

Type	Date	Time	Users	Source IP	Computer name	Content	
Warning	2018/12/09	04:59:01	System	127.0.0.1	localhost	Host: Disk 10 Read I/O error, ILLEGAL REQUEST sense_key=0x5, asc=0x21, ascq=0x4, CDB=88 00 00 00 00 01 32 55 39 c8 00 00 00 08 00 00 .
However, QNAP says this is just a "warning" not an "error". Is the disk okay?

Checking SMART I get this which seems the same as my other disks:

Code: Select all

[/] # get_hd_smartinfo -d 10
001 Raw_Read_Error_Rate          0        200 200 051 OK
003 Spin_Up_Time                 6625     167 167 021 OK
004 Start_Stop_Count             24       100 100 000 OK
005 Reallocated_Sector_Ct        0        200 200 140 OK
007 Seek_Error_Rate              0        200 200 000 OK
009 Power_On_Hours               7033     091 091 000 OK
010 Spin_Retry_Count             0        100 253 000 OK
011 Calibration_Retry_Count      0        100 253 000 OK
012 Power_Cycle_Count            7        100 100 000 OK
192 Power-Off_Retract_Count      4        200 200 000 OK
193 Load_Cycle_Count             59       200 200 000 OK
194 Temperature_Celsius          37       113 108 000 OK
196 Reallocated_Event_Count      0        200 200 000 OK
197 Current_Pending_Sector       0        200 200 000 OK
198 Offline_Uncorrectable        0        100 253 000 OK
199 UDMA_CRC_Error_Count         0        200 200 000 OK
200 Multi_Zone_Error_Rate        0        200 200 000 OK
Any insight?
So it's now been ten months with the original disk 10 that started this. I've since had 2 drives fail in quick succession so I'm hoping it's definitely not a hardware issue.

However, I've noticed that disk 10 still has very slow IOPS read values. All the over disks are getting >100 which is correct for a 7200rpm drive but for some reason it is lingering at 58 IPOS.

What could cause this?
QNAP TS-EC1079 Pro (4.3.4.0435) | 10 x 4 TB (WD WDC WD40EFRX firmware 82.00A82) | 32 TB RAID-6
QNAP TS-639 Pro (3.8.1 Build 20121205) | 6 x 3 TB (Toshiba DT01ACA300 firmware MX6OABB0) | 12 TB RAID-6
Both shared over single port GbE and wifi router to Linux, Android, Windows, macOS and PS3

Post Reply

Return to “System & Disk Volume Management”