TS-EC879u-RP with FW 3.5.1.1002 Unstable

[ProTeck]

I have an 879 unit that has been running fine for the past few weeks and all of the sudden it rebooted(shut down) by itself with no reasons for it do so. I use this device in production and this is unacceptable! System RAID is now resynchronizing.

Here is what is in the Event logs of the device.

2011-11-16 07:38:14 System 127.0.0.1 localhost The system was not shut down properly last time.
-> Seriously vague and no details.
2011-11-16 07:38:15 System 127.0.0.1 localhost System started.

On another note, I have had 809 systems in the past with stability issues. Random reboots, Hard Drives being ejected, etc.
Seems like QNAP has some serious issues with stability in their product line.

I have already put in a ticket with QNAP support and hope to get a real answer on this.

I haven't seen any of these issues before on any other storage company's devices.

Again, this is unacceptable.

[ProTeck]

After discussing with QNAP support, they believe it's a Bad drive that caused 879 to reboot for no reason. They determined this after conducting an ssh session on it and pulled log file out of the CLI.

$64,000.00 question is(game show reference), Why isn't this type of data or other types not being logged in the event logs?

I think we might have to hire a dedicated Linux engineer to baby sit the units and monitor it from the CLI. (Sarcasm)

Please improve the Event logging and event notification components in the interface.

[ProTeck]

BTW, we are using Seagate ST2000NM0011 Constellation Enterprise hard drives on this unit.

[ProTeck]

Ok,

On Saturday night 12/03/2011 The 879 shut down unexpectedly and for no reason at all. Below is data from the log files:

2011-12-03 20:12:10 System 127.0.0.1 localhost Lan 2 link is Up.
2011-12-03 20:10:22 System 127.0.0.1 localhost [RAID5 Disk Volume: Drive 8 2 3 4 5 6 7] The file system is not clean. It is suggested that you run "check disk". 2011-12-03 20:10:06 System 127.0.0.1 localhost System started.
2011-12-03 20:10:05 System 127.0.0.1 localhost The system was not shut down properly last time.
2011-12-03 20:00:01 System 127.0.0.1 localhost [Remote Replication] Backup-VMs4 started.

Seems to me that something hung up(don't know what) and caused the system to reboot. This seems like an issue with the Firmware and not a hardware issue. Since this issue last Saturday, we have upgraded the firmware to 3.5.2. We are now very nervous to use this in production now. It seems to run fine without problems for a few weeks and then it crashes.

Anyone else having this issue?

[ProTeck]

Also, SCAN Disk and Check now found no issues on all the drives.

[ProTeck]

Anyone from QNAP have any idea on this?

[ProTeck]

Low and behold, our EC879 rebooted unexepectedly again last night Dec. 8th,2011

2011-12-09 05:40:05 System 127.0.0.1 localhost [RAID5 Disk Volume: Drive 8 2 3 4 5 6 7 Hot Spare Disk: 1] Resyncing done.
2011-12-08 22:50:17 System 127.0.0.1 localhost [RAID5 Disk Volume: Drive 8 2 3 4 5 6 7 Hot Spare Disk: 1] Start resyncing.
2011-12-08 22:48:08 System 127.0.0.1 localhost [RAID5 Disk Volume: Drive 8 2 3 4 5 6 7] The file system is not clean. It is suggested that you run "check disk".
2011-12-08 22:47:52 System 127.0.0.1 localhost System started.
2011-12-08 22:47:51 System 127.0.0.1 localhost The system was not shut down properly last time.

This is now occurring every 1 to 2 weeks. I stress, that we cannot have this. Is there a problem(s) with the Firmware or the system itself that QNAP does not want publish?
We already swapped out a hard drive that QNAP Support originally thought was the problem(see initial post).

We already posted a ticket with Support earlier this morning and are desperately awaiting their feedback.

System Config:

Firmware 3.5.2
Raid array 5 + Hot spare with ext 4
Seagate Constellation ES 2TB 6GB SATA drives
NFS for VMWare
iSCSI for LUNS (iSCSI process is bound to NIC 2)
Nightly RSYNC replication jobs.

We have not experienced theses issues with any of our 809u-RP systems.

Also, Is firmware 3.4.x compatible with the EC879?

[ProTeck]

McFly!!!

[patriots]

McFly??? What a reference... Were you running that QNAP at 88 Miles per hour with 1.21 Gigawatts of power??? Is that what caused your QNAP to crash, or was Biff crashing it?

[ProTeck]

Yeah... Where's Doc Brown when you need him to fix it.

[patriots]

That's It!!! Your 879 was probably stuck by a bolt of lightning!

[kevinliao]

Hi ProTeck, do your problem get fixed? Do you get any useful feedback from the customer service?

[ProTeck]

Hello,

Right now, the EC879 has not reboot since the last post stating it. Although, we have removed most of the functions/jobs that it was previously doing, thus reducing the load of the system by almost 90%. Since there is no load on it, it hasn't rebooted. We are going to place more of a load on it this week to see what happens.

[gm85]

I'm having the same issue with 3.5.2 Build 1126T on the 879u-RP. It has happened 3 times, with the latest happing on Saturday.

It has happened both in the middle of the night, and during the weekend (fortunately when activity on our servers/NAS is low). I am keeping my fingers crossed it does not happen during the work day.

We have 2 Mirrored Volumes, each used with iSCSI for our VMWare Servers.
Our third Mirrored Volume is used as a network share for Workstations (Windows Networking)
We also have an External USB Drive shared via NFS (mounted to a linux system for RSync Backups).

No other services are activated on the server.

Since the QNAP Logs don't show anything, I have enabled Syslog (OS-level Syslog... not Qnaps's web syslog) and am directing the output to a file stored on one of the volumes. Hopefully that will provide some insight to the random restarts. I can attach the instructions on how to do this if anyone else is interested. Hopefully it'll shed some light on the issue.

Although it's not a good issue to have, I'm glad to see I'm not alone. Hopefully it ends up being simply a software issue.

[QNAPJason]

Hi gm85,
we would like to find the root cause of your issue as soon as possible.
May I know if you can provide the system logs and kernel messages for our support team?
If you can provide remote session right after it occurs, it can also help us identifying the issue.

May I know:
1. your HDD models
2. vmware server version
3. how many concurrent client?

At last, can you perform bad block scan on your HDDs?