TS-EC2480U stopped responding while expanding
-
- Starting out
- Posts: 10
- Joined: Fri Feb 17, 2017 5:41 pm
TS-EC2480U stopped responding while expanding
Hi everybody,
In my company we have a TS-EC2480U with currently two RAID pools.
- RAID pool 1 as RAID5, 5x6TB drives (was about 90% full)
- RAID pool 2 as RAID6, 7x6TB drives (was about 80% full)
- Two 6TB global hot spares
- Two unused 6TB drives
Everything was working fine, but yesterday we launched a RAID pool 1 expansion, by adding a new drive from the two unused drives to the RAID5 archive.
This is nothing new as it was an operation already carried on multiple times in the past, always completed without a single glitch.
When we launched the expansion yesterday, it was slowly progressing from 0.0%: it usually took about a week to expand, but this was not a problem as the NAS used to continu to be working correctly.
This time something terrible happened: the NAS stopped responding at all, this is the situation:
- No response on ports 80, 21 and 22 (no SSH, FTP or Web)
- Only partial response from port 8080 (admin GUI): we get an HTTP error not completing a GET operation, the webserver is actually not responding...
- No response on SAMBA
- Responds to pings correctly and with no delay
- Red and Green lights are blinking (very likely means "Online RAID capacity expansion is in process." in user's manual)
- Before the problem, the statuses or the disks were perfectly good and montlhy tested (with the quick test procedure provided by the QNAP)
What to do now?
Should we wait (for days!) with the great discomfort of not being able to access the NAS?
Should we reboot the NAS hoping everything will be still accessible afterwards?
We got backups but this a tricky situation since restore time will be long, but in the meantime the NAS could respond again any time, making a restore a waste of time...
Since we have no SSH access we are unable to see what it is really doing: we can't assess if it will stay stuck forever or if it will unstuck by itself in a couple of days...
Suggestions?
Thanks!
In my company we have a TS-EC2480U with currently two RAID pools.
- RAID pool 1 as RAID5, 5x6TB drives (was about 90% full)
- RAID pool 2 as RAID6, 7x6TB drives (was about 80% full)
- Two 6TB global hot spares
- Two unused 6TB drives
Everything was working fine, but yesterday we launched a RAID pool 1 expansion, by adding a new drive from the two unused drives to the RAID5 archive.
This is nothing new as it was an operation already carried on multiple times in the past, always completed without a single glitch.
When we launched the expansion yesterday, it was slowly progressing from 0.0%: it usually took about a week to expand, but this was not a problem as the NAS used to continu to be working correctly.
This time something terrible happened: the NAS stopped responding at all, this is the situation:
- No response on ports 80, 21 and 22 (no SSH, FTP or Web)
- Only partial response from port 8080 (admin GUI): we get an HTTP error not completing a GET operation, the webserver is actually not responding...
- No response on SAMBA
- Responds to pings correctly and with no delay
- Red and Green lights are blinking (very likely means "Online RAID capacity expansion is in process." in user's manual)
- Before the problem, the statuses or the disks were perfectly good and montlhy tested (with the quick test procedure provided by the QNAP)
What to do now?
Should we wait (for days!) with the great discomfort of not being able to access the NAS?
Should we reboot the NAS hoping everything will be still accessible afterwards?
We got backups but this a tricky situation since restore time will be long, but in the meantime the NAS could respond again any time, making a restore a waste of time...
Since we have no SSH access we are unable to see what it is really doing: we can't assess if it will stay stuck forever or if it will unstuck by itself in a couple of days...
Suggestions?
Thanks!
- OneCD
- Guru
- Posts: 12143
- Joined: Sun Aug 21, 2016 10:48 am
- Location: "... there, behind that sofa!"
Re: TS-EC2480U stopped responding while expanding
Hi and welcome to the forum.
Does that model allow direct connection of a monitor? If so, you could connect a keyboard and login at the console.
Does that model allow direct connection of a monitor? If so, you could connect a keyboard and login at the console.
-
- Starting out
- Posts: 10
- Joined: Fri Feb 17, 2017 5:41 pm
Re: TS-EC2480U stopped responding while expanding
Thanks!
Unfortunately no monitor and no serial ports, this is all I got:
https://download.qnap.com/Origin/images ... ware01.png
...Anything else?
Unfortunately no monitor and no serial ports, this is all I got:
https://download.qnap.com/Origin/images ... ware01.png
...Anything else?
-
- Starting out
- Posts: 10
- Joined: Fri Feb 17, 2017 5:41 pm
Re: TS-EC2480U stopped responding while expanding
Wait... there is an HDMI port!
I'll try right now, thanks again!
I'll try right now, thanks again!
-
- Starting out
- Posts: 10
- Joined: Fri Feb 17, 2017 5:41 pm
Re: TS-EC2480U stopped responding while expanding
Nothing... I tried with an HDMI monitor and a USB keyboard (I tried all the ports) but no video signal seems to be coming from the NAS... All I see is a black screen.
Other ideas?
Other ideas?
- OneCD
- Guru
- Posts: 12143
- Joined: Sun Aug 21, 2016 10:48 am
- Location: "... there, behind that sofa!"
Re: TS-EC2480U stopped responding while expanding
Yes.
You may not be presented with a login for the default HDMI output.
But as the base OS is derived from Ubuntu, you can type CTRL+ALT+F1 through to F7 to get other console logins.
You may not be presented with a login for the default HDMI output.
But as the base OS is derived from Ubuntu, you can type CTRL+ALT+F1 through to F7 to get other console logins.
-
- Starting out
- Posts: 10
- Joined: Fri Feb 17, 2017 5:41 pm
Re: TS-EC2480U stopped responding while expanding
Still nothing, I got no signal on the monitor: the led on the monitor stays red, meaning there is no video signal coming from the NAS, apparently...
- OneCD
- Guru
- Posts: 12143
- Joined: Sun Aug 21, 2016 10:48 am
- Location: "... there, behind that sofa!"
-
- Starting out
- Posts: 10
- Joined: Fri Feb 17, 2017 5:41 pm
Re: TS-EC2480U stopped responding while expanding
Guys, I'm still stuck: what would you do in this situation?
Risk a restart or wait but risking waiting forever?
Risk a restart or wait but risking waiting forever?
- Spider99
- Experience counts
- Posts: 1951
- Joined: Fri Oct 21, 2011 11:14 pm
- Location: UK
Re: TS-EC2480U stopped responding while expanding
Why no ssh access? Without that you are caught in a bind as you have no way of knowing if its stuck or still chugging along
As you have backups i would risk a restart - i.e. press the power button for a couple of sec to get it to shutdown - my take awhile
IIRC on restart it will try and re sync the array but you should be able to get in and see what errors might be in the logs etc and turn on ssh if its off
Good Luck
As you have backups i would risk a restart - i.e. press the power button for a couple of sec to get it to shutdown - my take awhile
IIRC on restart it will try and re sync the array but you should be able to get in and see what errors might be in the logs etc and turn on ssh if its off
Good Luck
Tim
TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
-
- Starting out
- Posts: 10
- Joined: Fri Feb 17, 2017 5:41 pm
Re: TS-EC2480U stopped responding while expanding
Thanks for the help.
Just to provide a bigger picture:
SSH was ON and working until the NAS got stuck, now the only thing that is responding is ping...
Another additional information: the QTS OS resides on the same pool that is espanding (pool 1), so if the restart doesn't work as expected (i.e. if the pool 1 is compromised), tha NAS OS would not be able to start anymore, and besides data, we would have to restore also all the settings, users quota and so many settings used in our corporate environment that it could take days and days just to sort it out.
As we are still waiting for the QNAP support to reply, we are still stuck on what to do... recovering data from backups is our last resort, I would strongly avoid that!
Thanks for any other advice
Just to provide a bigger picture:
SSH was ON and working until the NAS got stuck, now the only thing that is responding is ping...
Another additional information: the QTS OS resides on the same pool that is espanding (pool 1), so if the restart doesn't work as expected (i.e. if the pool 1 is compromised), tha NAS OS would not be able to start anymore, and besides data, we would have to restore also all the settings, users quota and so many settings used in our corporate environment that it could take days and days just to sort it out.
As we are still waiting for the QNAP support to reply, we are still stuck on what to do... recovering data from backups is our last resort, I would strongly avoid that!
Thanks for any other advice
- Spider99
- Experience counts
- Posts: 1951
- Joined: Fri Oct 21, 2011 11:14 pm
- Location: UK
Re: TS-EC2480U stopped responding while expanding
Well if you cant get in via the normal methods Qnap support will not be able to help much until you can get a connection to the nas
You can wait but it might be a waste of time - are the disks being accessed at all can you hear the disks working?
My suspicion is you have a drive failure or two on the raid5 and the nas has lost its system drives and has become unresponsive
You can wait but it might be a waste of time - are the disks being accessed at all can you hear the disks working?
My suspicion is you have a drive failure or two on the raid5 and the nas has lost its system drives and has become unresponsive
Tim
TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
- MrVideo
- Experience counts
- Posts: 4742
- Joined: Fri May 03, 2013 2:26 pm
Re: TS-EC2480U stopped responding while expanding
Actually it does not. The OS is placed in a separate RAID1 partition.wh1sp3r wrote:Another additional information: the QTS OS resides on the same pool that is espanding (pool 1)
QTS MANUALS
Submit QNAP Support Ticket - QNAP Tutorials, FAQs, Downloads, Wiki - Product Support Status - Moogle's QNAP FAQ help V2
Asking a question, include the following (Thanks to Toxic17)
QNAP md_checker nasreport (release 20210309)
===============================
Model: TS-869L -- RAM: 3G -- FW: QTS 4.1.4 Build 20150522 (used for data storage)
WD60EFRX-68L0BN1(x1)/68MYMN1(x7) Red HDDs -- RAID6: 8x6TB -- Cold spare: 1x6TB
Entware
===============================
Model: TS-451A -- RAM: 2G -- FW: QTS 4.5.2 Build 20210202 (used as a video server)
WL3000GSA6472(x3) White label NAS HDDs -- RAID5: 3x3TB
Entware -- MyKodi 17.3 (default is Kodi 16)
===============================
My 2017 Total Solar Eclipse Photos | My 2019 N. Ireland Game of Thrones tour
Submit QNAP Support Ticket - QNAP Tutorials, FAQs, Downloads, Wiki - Product Support Status - Moogle's QNAP FAQ help V2
Asking a question, include the following (Thanks to Toxic17)
QNAP md_checker nasreport (release 20210309)
===============================
Model: TS-869L -- RAM: 3G -- FW: QTS 4.1.4 Build 20150522 (used for data storage)
WD60EFRX-68L0BN1(x1)/68MYMN1(x7) Red HDDs -- RAID6: 8x6TB -- Cold spare: 1x6TB
Entware
===============================
Model: TS-451A -- RAM: 2G -- FW: QTS 4.5.2 Build 20210202 (used as a video server)
WL3000GSA6472(x3) White label NAS HDDs -- RAID5: 3x3TB
Entware -- MyKodi 17.3 (default is Kodi 16)
===============================
My 2017 Total Solar Eclipse Photos | My 2019 N. Ireland Game of Thrones tour
-
- Starting out
- Posts: 10
- Joined: Fri Feb 17, 2017 5:41 pm
Re: TS-EC2480U stopped responding while expanding
Thanks a lot for the information, since I was not here when the NAS was initialized I might not have the big picture about its OS:MrVideo wrote:Actually it does not. The OS is placed in a separate RAID1 partition.
The separate RAID1 partition must be indicated before the initialization or you are talking about another separate internal memory which has nothing to do with the 24 HDD slots?
In this case, if I reboot the NAS the OS should stay clean and operate correclty, am I right?
- Spider99
- Experience counts
- Posts: 1951
- Joined: Fri Oct 21, 2011 11:14 pm
- Location: UK
Re: TS-EC2480U stopped responding while expanding
the raid1 is across all disks in the nas - its a mirror which in your case, if the nas is full, 22 spare partitions available
there are also ones for swap and apps - also raid 1's
and your data volume uses the fourth partition as raid5/6/or 10 etc
there is also a DOM which bootstraps the nas until the firmware can be loaded - although i think some nas do not have this IIRC
Are you on a recent firmware version and have you upgraded recently i.e. say just before the raid expansion?
there are also ones for swap and apps - also raid 1's
and your data volume uses the fourth partition as raid5/6/or 10 etc
there is also a DOM which bootstraps the nas until the firmware can be loaded - although i think some nas do not have this IIRC
Are you on a recent firmware version and have you upgraded recently i.e. say just before the raid expansion?
Tim
TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
TS-853A(16GB): - 4.3.4.0483 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)