TS-EC2480U stopped responding while expanding

[wh1sp3r]

Hi everybody,

In my company we have a TS-EC2480U with currently two RAID pools.
- RAID pool 1 as RAID5, 5x6TB drives (was about 90% full)
- RAID pool 2 as RAID6, 7x6TB drives (was about 80% full)
- Two 6TB global hot spares
- Two unused 6TB drives

Everything was working fine, but yesterday we launched a RAID pool 1 expansion, by adding a new drive from the two unused drives to the RAID5 archive.
This is nothing new as it was an operation already carried on multiple times in the past, always completed without a single glitch.
When we launched the expansion yesterday, it was slowly progressing from 0.0%: it usually took about a week to expand, but this was not a problem as the NAS used to continu to be working correctly.

This time something terrible happened: the NAS stopped responding at all, this is the situation:

- No response on ports 80, 21 and 22 (no SSH, FTP or Web)
- Only partial response from port 8080 (admin GUI): we get an HTTP error not completing a GET operation, the webserver is actually not responding...
- No response on SAMBA
- Responds to pings correctly and with no delay
- Red and Green lights are blinking (very likely means "Online RAID capacity expansion is in process." in user's manual)
- Before the problem, the statuses or the disks were perfectly good and montlhy tested (with the quick test procedure provided by the QNAP)


What to do now?
Should we wait (for days!) with the great discomfort of not being able to access the NAS?
Should we reboot the NAS hoping everything will be still accessible afterwards?

We got backups but this a tricky situation since restore time will be long, but in the meantime the NAS could respond again any time, making a restore a waste of time...
Since we have no SSH access we are unable to see what it is really doing: we can't assess if it will stay stuck forever or if it will unstuck by itself in a couple of days...

Suggestions?

Thanks!

[OneCD]

Hi and welcome to the forum.

Does that model allow direct connection of a monitor? If so, you could connect a keyboard and login at the console.

[wh1sp3r]

Thanks!

Unfortunately no monitor and no serial ports, this is all I got:

https://download.qnap.com/Origin/images ... ware01.png

...Anything else?

[wh1sp3r]

Wait... there is an HDMI port!

I'll try right now, thanks again!

[wh1sp3r]

Nothing... I tried with an HDMI monitor and a USB keyboard (I tried all the ports) but no video signal seems to be coming from the NAS... All I see is a black screen.

Other ideas?

[OneCD]

Yes.

You may not be presented with a login for the default HDMI output.

But as the base OS is derived from Ubuntu, you can type CTRL+ALT+F1 through to F7 to get other console logins.

[wh1sp3r]

Still nothing, I got no signal on the monitor: the led on the monitor stays red, meaning there is no video signal coming from the NAS, apparently...

[OneCD]

I'm out of ideas.

[wh1sp3r]

Guys, I'm still stuck: what would you do in this situation?

Risk a restart or wait but risking waiting forever?

[Spider99]

Why no ssh access? Without that you are caught in a bind as you have no way of knowing if its stuck or still chugging along

As you have backups i would risk a restart - i.e. press the power button for a couple of sec to get it to shutdown - my take awhile

IIRC on restart it will try and re sync the array but you should be able to get in and see what errors might be in the logs etc and turn on ssh if its off

Good Luck

[wh1sp3r]

Thanks for the help.
Just to provide a bigger picture:
SSH was ON and working until the NAS got stuck, now the only thing that is responding is ping...

Another additional information: the QTS OS resides on the same pool that is espanding (pool 1), so if the restart doesn't work as expected (i.e. if the pool 1 is compromised), tha NAS OS would not be able to start anymore, and besides data, we would have to restore also all the settings, users quota and so many settings used in our corporate environment that it could take days and days just to sort it out.

As we are still waiting for the QNAP support to reply, we are still stuck on what to do... recovering data from backups is our last resort, I would strongly avoid that!

Thanks for any other advice

[Spider99]

Well if you cant get in via the normal methods Qnap support will not be able to help much until you can get a connection to the nas

You can wait but it might be a waste of time - are the disks being accessed at all can you hear the disks working?

My suspicion is you have a drive failure or two on the raid5 and the nas has lost its system drives and has become unresponsive

[MrVideo]

Another additional information: the QTS OS resides on the same pool that is espanding (pool 1)

Actually it does not. The OS is placed in a separate RAID1 partition.

[wh1sp3r]

Actually it does not. The OS is placed in a separate RAID1 partition.

Thanks a lot for the information, since I was not here when the NAS was initialized I might not have the big picture about its OS:
The separate RAID1 partition must be indicated before the initialization or you are talking about another separate internal memory which has nothing to do with the 24 HDD slots?
In this case, if I reboot the NAS the OS should stay clean and operate correclty, am I right?

[Spider99]

the raid1 is across all disks in the nas - its a mirror which in your case, if the nas is full, 22 spare partitions available
there are also ones for swap and apps - also raid 1's
and your data volume uses the fourth partition as raid5/6/or 10 etc

there is also a DOM which bootstraps the nas until the firmware can be loaded - although i think some nas do not have this IIRC

Are you on a recent firmware version and have you upgraded recently i.e. say just before the raid expansion?