TS-EC2480U stopped responding while expanding

Questions about SNMP, Power, System, Logs, disk, & RAID.
wh1sp3r
Starting out
Posts: 10
Joined: Fri Feb 17, 2017 5:41 pm
Model: TS-ECx80U

TS-EC2480U stopped responding while expanding

Postby wh1sp3r » Fri Feb 17, 2017 5:45 pm

Hi everybody,

In my company we have a TS-EC2480U with currently two RAID pools.
- RAID pool 1 as RAID5, 5x6TB drives (was about 90% full)
- RAID pool 2 as RAID6, 7x6TB drives (was about 80% full)
- Two 6TB global hot spares
- Two unused 6TB drives

Everything was working fine, but yesterday we launched a RAID pool 1 expansion, by adding a new drive from the two unused drives to the RAID5 archive.
This is nothing new as it was an operation already carried on multiple times in the past, always completed without a single glitch.
When we launched the expansion yesterday, it was slowly progressing from 0.0%: it usually took about a week to expand, but this was not a problem as the NAS used to continu to be working correctly.

This time something terrible happened: the NAS stopped responding at all, this is the situation:

- No response on ports 80, 21 and 22 (no SSH, FTP or Web)
- Only partial response from port 8080 (admin GUI): we get an HTTP error not completing a GET operation, the webserver is actually not responding...
- No response on SAMBA
- Responds to pings correctly and with no delay
- Red and Green lights are blinking (very likely means "Online RAID capacity expansion is in process." in user's manual)
- Before the problem, the statuses or the disks were perfectly good and montlhy tested (with the quick test procedure provided by the QNAP)


What to do now?
Should we wait (for days!) with the great discomfort of not being able to access the NAS?
Should we reboot the NAS hoping everything will be still accessible afterwards?

We got backups but this a tricky situation since restore time will be long, but in the meantime the NAS could respond again any time, making a restore a waste of time...
Since we have no SSH access we are unable to see what it is really doing: we can't assess if it will stay stuck forever or if it will unstuck by itself in a couple of days...

Suggestions?

Thanks!

User avatar
OneCD
Experience counts
Posts: 2320
Joined: Sun Aug 21, 2016 10:48 am
Model: TS-x69 Pro
Location: over there, behind that sofa...

Re: TS-EC2480U stopped responding while expanding

Postby OneCD » Fri Feb 17, 2017 5:55 pm

Hi and welcome to the forum. :)

Does that model allow direct connection of a monitor? If so, you could connect a keyboard and login at the console. :geek:
Stuff I like: Apache automation BASH blocklists BTFS BusyBox CLI CouchPotato CRON CSS DD-WRT Debian DNS DNSMasq Entware-3x EXT2/3/4 FireFox GitHub HTML ImageMagick IP-networking Kate KDE Kodi L2TP/IPsec LibreELEC Linux LXDE Linux Mint NAS Netrunner NFS OpenSUSE Orvibo S20 P2P phpMyAdmin Python Raspberry Pi RAID remote-access rsync SABnzbd+ screen scripts shell Shuttle SickBeard SickRage SQL SQLite SSD SSH Transmission Ubuntu UPS Usenet VirtualBox VPN web-servers WiFi WordPress XFCE XFS XML

wh1sp3r
Starting out
Posts: 10
Joined: Fri Feb 17, 2017 5:41 pm
Model: TS-ECx80U

Re: TS-EC2480U stopped responding while expanding

Postby wh1sp3r » Fri Feb 17, 2017 6:04 pm

Thanks!

Unfortunately no monitor and no serial ports, this is all I got:

https://download.qnap.com/Origin/images ... ware01.png

...Anything else?

wh1sp3r
Starting out
Posts: 10
Joined: Fri Feb 17, 2017 5:41 pm
Model: TS-ECx80U

Re: TS-EC2480U stopped responding while expanding

Postby wh1sp3r » Fri Feb 17, 2017 6:05 pm

Wait... there is an HDMI port!

I'll try right now, thanks again!

wh1sp3r
Starting out
Posts: 10
Joined: Fri Feb 17, 2017 5:41 pm
Model: TS-ECx80U

Re: TS-EC2480U stopped responding while expanding

Postby wh1sp3r » Fri Feb 17, 2017 6:28 pm

Nothing... I tried with an HDMI monitor and a USB keyboard (I tried all the ports) but no video signal seems to be coming from the NAS... All I see is a black screen.

Other ideas?

User avatar
OneCD
Experience counts
Posts: 2320
Joined: Sun Aug 21, 2016 10:48 am
Model: TS-x69 Pro
Location: over there, behind that sofa...

Re: TS-EC2480U stopped responding while expanding

Postby OneCD » Fri Feb 17, 2017 6:33 pm

Yes. :geek:

You may not be presented with a login for the default HDMI output.

But as the base OS is derived from Ubuntu, you can type CTRL+ALT+F1 through to F7 to get other console logins.
Stuff I like: Apache automation BASH blocklists BTFS BusyBox CLI CouchPotato CRON CSS DD-WRT Debian DNS DNSMasq Entware-3x EXT2/3/4 FireFox GitHub HTML ImageMagick IP-networking Kate KDE Kodi L2TP/IPsec LibreELEC Linux LXDE Linux Mint NAS Netrunner NFS OpenSUSE Orvibo S20 P2P phpMyAdmin Python Raspberry Pi RAID remote-access rsync SABnzbd+ screen scripts shell Shuttle SickBeard SickRage SQL SQLite SSD SSH Transmission Ubuntu UPS Usenet VirtualBox VPN web-servers WiFi WordPress XFCE XFS XML

wh1sp3r
Starting out
Posts: 10
Joined: Fri Feb 17, 2017 5:41 pm
Model: TS-ECx80U

Re: TS-EC2480U stopped responding while expanding

Postby wh1sp3r » Fri Feb 17, 2017 7:05 pm

Still nothing, I got no signal on the monitor: the led on the monitor stays red, meaning there is no video signal coming from the NAS, apparently...

User avatar
OneCD
Experience counts
Posts: 2320
Joined: Sun Aug 21, 2016 10:48 am
Model: TS-x69 Pro
Location: over there, behind that sofa...

Re: TS-EC2480U stopped responding while expanding

Postby OneCD » Fri Feb 17, 2017 8:27 pm

I'm out of ideas. :(
Stuff I like: Apache automation BASH blocklists BTFS BusyBox CLI CouchPotato CRON CSS DD-WRT Debian DNS DNSMasq Entware-3x EXT2/3/4 FireFox GitHub HTML ImageMagick IP-networking Kate KDE Kodi L2TP/IPsec LibreELEC Linux LXDE Linux Mint NAS Netrunner NFS OpenSUSE Orvibo S20 P2P phpMyAdmin Python Raspberry Pi RAID remote-access rsync SABnzbd+ screen scripts shell Shuttle SickBeard SickRage SQL SQLite SSD SSH Transmission Ubuntu UPS Usenet VirtualBox VPN web-servers WiFi WordPress XFCE XFS XML

wh1sp3r
Starting out
Posts: 10
Joined: Fri Feb 17, 2017 5:41 pm
Model: TS-ECx80U

Re: TS-EC2480U stopped responding while expanding

Postby wh1sp3r » Sat Feb 18, 2017 12:35 am

Guys, I'm still stuck: what would you do in this situation?

Risk a restart or wait but risking waiting forever?

User avatar
Spider99
Experience counts
Posts: 1777
Joined: Fri Oct 21, 2011 11:14 pm
Model: TS-253A/453A/653A/853A
Location: Warrington, England

Re: TS-EC2480U stopped responding while expanding

Postby Spider99 » Sat Feb 18, 2017 1:37 am

Why no ssh access? Without that you are caught in a bind as you have no way of knowing if its stuck or still chugging along

As you have backups i would risk a restart - i.e. press the power button for a couple of sec to get it to shutdown - my take awhile

IIRC on restart it will try and re sync the array but you should be able to get in and see what errors might be in the logs etc and turn on ssh if its off :)

Good Luck
Tim

TS-853A(16GB): - 4.2.2 - 16/12 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
=========================================================================
Online Manuals - Tutorials & FAQ - Posting guidelines

wh1sp3r
Starting out
Posts: 10
Joined: Fri Feb 17, 2017 5:41 pm
Model: TS-ECx80U

Re: TS-EC2480U stopped responding while expanding

Postby wh1sp3r » Sat Feb 18, 2017 6:33 pm

Thanks for the help.
Just to provide a bigger picture:
SSH was ON and working until the NAS got stuck, now the only thing that is responding is ping...

Another additional information: the QTS OS resides on the same pool that is espanding (pool 1), so if the restart doesn't work as expected (i.e. if the pool 1 is compromised), tha NAS OS would not be able to start anymore, and besides data, we would have to restore also all the settings, users quota and so many settings used in our corporate environment that it could take days and days just to sort it out.

As we are still waiting for the QNAP support to reply, we are still stuck on what to do... recovering data from backups is our last resort, I would strongly avoid that!

Thanks for any other advice

User avatar
Spider99
Experience counts
Posts: 1777
Joined: Fri Oct 21, 2011 11:14 pm
Model: TS-253A/453A/653A/853A
Location: Warrington, England

Re: TS-EC2480U stopped responding while expanding

Postby Spider99 » Sat Feb 18, 2017 10:34 pm

Well if you cant get in via the normal methods Qnap support will not be able to help much until you can get a connection to the nas

You can wait but it might be a waste of time - are the disks being accessed at all can you hear the disks working?

My suspicion is you have a drive failure or two on the raid5 and the nas has lost its system drives and has become unresponsive
Tim

TS-853A(16GB): - 4.2.2 - 16/12 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
=========================================================================
Online Manuals - Tutorials & FAQ - Posting guidelines

User avatar
MrVideo
Experience counts
Posts: 3288
Joined: Fri May 03, 2013 2:26 pm
Model: TS-x69L

Re: TS-EC2480U stopped responding while expanding

Postby MrVideo » Sun Feb 19, 2017 12:21 am

wh1sp3r wrote:Another additional information: the QTS OS resides on the same pool that is espanding (pool 1)

Actually it does not. The OS is placed in a separate RAID1 partition.
QTS 4.1.n/4.2.n MANUAL
Submit QNAP Support Ticket - QNAP Tutorials, FAQs, Downloads, Wiki - Product Support Status
When you ask a question, please include the following
(Thanks to Toxic17 for the links)
=====================================================================================
Model: TS-869L -- RAM: 3G -- FW: QTS 4.1.4 Build 20150522 (used for data storage)
WD60EFRX-68L0BN1(x1)/68MYMN1(x7) Red HDDs -- RAID6: 8x6TB -- Cold spare: 1x6TB
entware-ng
=====================================================================================
Model: TS-451A -- RAM: 2G -- FW: QTS 4.2.3 Build 20170213 (used as a video server)
WL3000GSA6472(x3) White label NAS HDDs -- RAID5: 3x3TB
entware-ng -- Kodi 17 Beta 5 (default is Kodi 16)

wh1sp3r
Starting out
Posts: 10
Joined: Fri Feb 17, 2017 5:41 pm
Model: TS-ECx80U

Re: TS-EC2480U stopped responding while expanding

Postby wh1sp3r » Mon Feb 20, 2017 4:38 pm

MrVideo wrote:Actually it does not. The OS is placed in a separate RAID1 partition.


Thanks a lot for the information, since I was not here when the NAS was initialized I might not have the big picture about its OS:
The separate RAID1 partition must be indicated before the initialization or you are talking about another separate internal memory which has nothing to do with the 24 HDD slots?
In this case, if I reboot the NAS the OS should stay clean and operate correclty, am I right?

User avatar
Spider99
Experience counts
Posts: 1777
Joined: Fri Oct 21, 2011 11:14 pm
Model: TS-253A/453A/653A/853A
Location: Warrington, England

Re: TS-EC2480U stopped responding while expanding

Postby Spider99 » Mon Feb 20, 2017 4:57 pm

the raid1 is across all disks in the nas - its a mirror which in your case, if the nas is full, 22 spare partitions available
there are also ones for swap and apps - also raid 1's
and your data volume uses the fourth partition as raid5/6/or 10 etc

there is also a DOM which bootstraps the nas until the firmware can be loaded - although i think some nas do not have this IIRC

Are you on a recent firmware version and have you upgraded recently i.e. say just before the raid expansion?
Tim

TS-853A(16GB): - 4.2.2 - 16/12 - Static volume - Raid5 - 8 x 4TB HGST Deskstar NAS
Windows Server + StableBit Drivepool and Scanner ~115 TB Backup Server
TS-412 & TS-459 Pro II: Retired
Clients: 3 x Windows 10 Pro(64bit)
=========================================================================
Online Manuals - Tutorials & FAQ - Posting guidelines


Return to “System & Disk Volume Management”

Who is online

Users browsing this forum: No registered users and 3 guests