TS-659 unable to rescue data from failing RAID 5

[grijzeduif]

Hi everyone,
I was running a TS-659 for 4 years without any interruptions, but after a failing RAID 5 configuration, 1 disk faulty, 1 disk in warning state I decided to replace as a start all the disks with 2 bigger and modern WD REDs (WD4001FFSX)
and to decide about fault tolerace a little later.
What happend till now:
firmware 4.1.0, 6 disks Seagate ST31500341AS CC1H

13/08/2014 RAID-5 error, drive 5 faulty, drive 2 Warning
16/08/2014 copied data to local pc, and replaced raid 5 with single disk, WD4001FFSX and configured the box with shares etc. and copied data to NAS again
18/08/2014 noticed that I forgot to copy my picture folder from the faulty RAID config, so I put the disks back in, loaded a backup file from 2010 with (I presumed) the correct RAID config
volume management: drive 1,3,4 and 6 Good, drive 2 and 5 in warning
RAID management: RAID 5 no size, Not active and Recover not working.
So at the moment I cant access my data, from which I thought that should still be on the disks.
Can you help me with recovering the RAID 5 and my pictures (I know RAID 5 isn't a backup but nevertheless... ) thxs Philip

[pwilson]

Hi everyone,
I was running a TS-659 for 4 years without any interruptions, but after a failing RAID 5 configuration, 1 disk faulty, 1 disk in warning state I decided to replace as a start all the disks with 2 bigger and modern WD REDs (WD4001FFSX)
and to decide about fault tolerace a little later.
What happend till now:
firmware 4.1.0, 6 disks Seagate ST31500341AS CC1H

13/08/2014 RAID-5 error, drive 5 faulty, drive 2 Warning
16/08/2014 copied data to local pc, and replaced raid 5 with single disk, WD4001FFSX and configured the box with shares etc. and copied data to NAS again
18/08/2014 noticed that I forgot to copy my picture folder from the faulty RAID config, so I put the disks back in, loaded a backup file from 2010 with (I presumed) the correct RAID config
volume management: drive 1,3,4 and 6 Good, drive 2 and 5 in warning
RAID management: RAID 5 no size, Not active and Recover not working.
So at the moment I cant access my data, from which I thought that should still be on the disks.
Can you help me with recovering the RAID 5 and my pictures (I know RAID 5 isn't a backup but nevertheless... ) thxs Philip

A RAID5 array with two duff drives is no longer viable. RAID5 can only tolerate a single drive failure. Why didn't you simply hot-swap out HDD5 on 2014.08.13, and then HDD2 on 2014.08.14?

Rebuild your NAS on the new WD Reds, and restore from a backup made prior to 2014.08.13. Was it really 2010 when you last did a backup? Wow.
I would have expected you to backup your "Pictures" first, especially without other Backups in place. Movies/TV Shows/Music files can typically be re-created, but if your pictures were "Photos" then you may never get them back.

[grijzeduif]

Hi Patrick, thxs for the quick reply.
The second and fifth are in warning state, that doesn't imply defective I thought? The fifth was in faulty before the "rebuild". With the backup I ment the backup of the QNAP config, not the actual data.
Why didn't I swap. I don't have disk on a shelf at home I thought I had the rebuild process under control, but finally forgot that stupid picture folder.
But at the moment I have 2 drives in warning, but not defective I presume. SMART states "The hard disk was detected not to be in the best status. Your hard disk should be operating properly"
Is there hope anywhere?

[pwilson]

Hi Patrick, thxs for the quick reply.
The second and fifth are in warning state, that doesn't imply defective I thought? The fifth was in faulty before the "rebuild". With the backup I ment the backup of the QNAP config, not the actual data.
Why didn't I swap. I don't have disk on a shelf at home I thought I had the rebuild process under control, but finally forgot that stupid picture folder.
But at the moment I have 2 drives in warning, but not defective I presume. SMART states "The hard disk was detected not to be in the best status. Your hard disk should be operating properly"
Is there hope anywhere?

With the original drives installed in the NAS, please login to your NAS via SSH, and run:

Code: Select all

#!/bin/sh
rm -f /tmp/nasreport
touch /tmp/nasreport
chmod +x /tmp/nasreport
cat <<EOF >>/tmp/nasreport
#!/bin/sh
#
# NAS Report by Patrick Wilson
# see: http://forum.qnap.com/viewtopic.php?f=185&t=82260#p366188
#
# 
echo "*********************"
echo "** QNAP NAS Report **"
echo "*********************"
echo " "
echo "NAS Model:      \$(getsysinfo model)"
echo "Firmware:       \$(getcfg system version) Build \$(getcfg system 'Build Number')"
echo "System Name:    \$(/bin/hostname)"
echo "Workgroup:      \$(getcfg system workgroup)"
echo "Base Directory: \$(dirname \$(getcfg -f /etc/config/smb.conf Public path))"
echo "NAS IP address: \$(ifconfig \$(getcfg network 'Default GW Device') | grep addr: | awk '{ print \$2 }' | cut -d: -f2)"
echo " " 
echo "Default Gateway Device: \$(getcfg network 'Default GW Device')" 
echo " "
ifconfig \$(getcfg network 'Default GW Device') | grep -v HWaddr
echo " "
echo -n "DNS Nameserver(s):" 
cat /etc/resolv.conf | grep nameserver | cut -d' ' -f2
echo " "
echo " "
echo "HDD Information:"
echo " "
if [ -x /sbin/hdparm ]; then
   for i in {a..d}; do echo /dev/sd\$i; echo -n; hdparm -i /dev/sd\$i | grep "Model"; done
else 
   echo "   /sbin/hdparm is not present" 
fi
echo " "
echo "Disk Space:"
echo " "
df -h | grep -v qpkg
echo " "
echo "Mount Status:" 
echo " "
mount | grep -v qpkg
echo " " 
echo "RAID Status:" 
echo " " 
cat /proc/mdstat
echo " " 
#echo "QNAP Media Scanner / Transcoder processes running: "
#echo " " 
#/bin/ps | grep medialibrary | grep -v grep
#echo " " 
#echo -n "MediaLibrary Configuration file: " 
#ls -alF /etc/config/medialibrary.conf
#echo " " 
#echo "/etc/config/medialibrary.conf:"
#cat /etc/config/medialibrary.conf
echo " "
echo "Memory Information:" 
echo " "
cat /proc/meminfo | grep Mem
echo " "
echo "NASReport completed on \$(date +'%Y-%m-%d %T') ($0)" 
EOF
sleep 2
clear
/tmp/nasreport
#done

Please cut&paste the output of the resulting NASReport back to this message thread.

[grijzeduif]

I'm not an expert on linux to begin, but have a brief experience with WINSCP. Tried to run your command:
=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2014.08.19 17:17:37 =~=~=~=~=~=~=~=~=~=~=~=
login as: admin
admin@10.0.0.160's password:
[~] # !/bin/sh
-sh: !/bin/sh: event not found
[~] # cd !
-sh: cd: !: No such file or directory
[~] # cd bin\sh
-sh: cd: binsh: No such file or directory
[~] # cd bin\shcd bin
-sh: cd: bin: No such file or directory
[~] # cd..
-sh: cd..: command not found
[~] # cd ..
[/] # cd bin\sh
-sh: cd: binsh: No such file or directory
[/] # cd bin\shls
bin/ etc/ lib/ linuxrc@ mnt/ php.ini@ root/ sbin/ sys/ usr/
dev/ home/ lib64/ lost+found/ opt/ proc/ rpc/ share/ tmp/ var/
[/] # cd bin
[/bin] # cd sh
-sh: cd: sh: Not a directory
[/bin] # ls
[@ chgrp@ df@ egrep@ login@ netstat@ sh* uname@
addgroup@ chmod@ dmesg@ false@ ls@ pidof@ sha1sum@ uniq@
adduser@ chown@ e2fsck* fgrep@ md5sum@ ping@ sleep@ usleep@
ash@ cmp* e2fsck_64* grep@ mkdir@ ping6@ sync@ utelnetd*
awk@ cp* e2fsck_64_qnap* gunzip@ mknod* ps@ tar* vi@
bash@ cut@ e3* gzip@ mktemp@ pwd@ touch@ vim@
busybox* date* e3pi@ hostname@ more@ renice@ tr@ wget-s*
bzcat@ dd@ e3vi@ ip* mount* rm@ true@ zcat@
bzip2* delgroup@ e3ws@ kill@ mv@ rmdir@ tty@
cat@ deluser@ echo@ ln@ naslogin* sed@ umount*
[/bin] # sh
[/bin] # rm -f /tmp/nasreport
[/bin] # touch /tmp/nasreport
[/bin] # chmod +x /tmp/nasreport
[/bin] # cat <<EOF >>/tmp/nasreport
> #!/bin/sh
> #
> # NAS Report by Patrick Wilson
> # see: http://forum.qnap.com/viewtopic.php?f=1 ... 60#p366188
echo " "
> #
> #
echo " "
> echo "*********************"
#echo "QNAP Media Scanner / Transcod> echo "** QNAP NAS Report **"
/tmp/nasreport
#>

I seem to lose commands, and the expected file nasreport in the temp folder look empty.
Where do I go wrong?

30 minutes later I had some output:
#!/bin/sh
#
# NAS Report by Patrick Wilson
# see: http://forum.qnap.com/viewtopic.php?f=1 ... 60#p366188
#
#
echo "*********************"
echo "** QNAP NAS Report **"

and ther it stopped.

[pwilson]

I'm not an expert on linux to begin, but have a brief experience with WINSCP. Tried to run your command:
=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2014.08.19 17:17:37 =~=~=~=~=~=~=~=~=~=~=~=
login as: admin
admin@10.0.0.160's password:
[~] # !/bin/sh
-sh: !/bin/sh: event not found
[~] # cd !
-sh: cd: !: No such file or directory
[~] # cd bin\sh
-sh: cd: binsh: No such file or directory
[~] # cd bin\shcd bin
-sh: cd: bin: No such file or directory
[~] # cd..
-sh: cd..: command not found
[~] # cd ..
[/] # cd bin\sh
-sh: cd: binsh: No such file or directory
[/] # cd bin\shls
bin/ etc/ lib/ linuxrc@ mnt/ php.ini@ root/ sbin/ sys/ usr/
dev/ home/ lib64/ lost+found/ opt/ proc/ rpc/ share/ tmp/ var/
[/] # cd bin
[/bin] # cd sh
-sh: cd: sh: Not a directory
[/bin] # ls
[@ chgrp@ df@ egrep@ login@ netstat@ sh* uname@
addgroup@ chmod@ dmesg@ false@ ls@ pidof@ sha1sum@ uniq@
adduser@ chown@ e2fsck* fgrep@ md5sum@ ping@ sleep@ usleep@
ash@ cmp* e2fsck_64* grep@ mkdir@ ping6@ sync@ utelnetd*
awk@ cp* e2fsck_64_qnap* gunzip@ mknod* ps@ tar* vi@
bash@ cut@ e3* gzip@ mktemp@ pwd@ touch@ vim@
busybox* date* e3pi@ hostname@ more@ renice@ tr@ wget-s*
bzcat@ dd@ e3vi@ ip* mount* rm@ true@ zcat@
bzip2* delgroup@ e3ws@ kill@ mv@ rmdir@ tty@
cat@ deluser@ echo@ ln@ naslogin* sed@ umount*
[/bin] # sh
[/bin] # rm -f /tmp/nasreport
[/bin] # touch /tmp/nasreport
[/bin] # chmod +x /tmp/nasreport
[/bin] # cat <<EOF >>/tmp/nasreport
> #!/bin/sh
> #
> # NAS Report by Patrick Wilson
> # see: http://forum.qnap.com/viewtopic.php?f=1 ... 60#p366188
echo " "
> #
> #
echo " "
> echo "*********************"
#echo "QNAP Media Scanner / Transcod> echo "** QNAP NAS Report **"
/tmp/nasreport
#>

I seem to lose commands, and the expected file nasreport in the temp folder look empty.
Where do I go wrong?

I did not ask for the Putty.log, I asked for a cut&paste of the output of the commands themselves.
That said, this output is a mess, and doesn't look good at all.

How recent is your most recent complete NAS data backup?

[grijzeduif]

I ran the report again:
login as: admin
admin@10.0.0.160's password:
[~] # #!/bin/sh
[~] # rm -f /tmp/nasreport
touch /tmp/nasreport
chmod +x /tmp/nasreport
cat <<EOF >>/tmp/nasreport
#!/bin/sh
#
[~] # touch /tmp/nasreport
# NAS Report by Patrick Wilson
# see: http://forum.qnap.com/viewtopic.php?f=1 ... 60#p366188
#
#
echo "*********************"
echo "** QNAP NAS Report **"
echo "*********************"
[~] # chmod +x /tmp/nasreport
echo " "
if [ -x /sbin/hdparm ]; then
for i in {a..d}; do echo /dev/sd\$i; echo -n; hdparm -i /dev/sd\$i | grep "Model"; done
else
echo " /sbin/hdparm is not present"
fi
echo " "
echo "Disk Space:"
echo " "
[~] # cat <<EOF >>/tmp/nasreport
#echo " "
#echo "/etc/config/medialibrary.conf:"
> #!/bin/sh
cat /> #
echo "NASReport complet>

Being used to Windows might be a handicap ; most recent backup none, I relied on the QNAP RAID-5.
I don't see any other output other than what is listed above.

Storage Manager states: Logical Volumes: RAID Volume not active. Would a check file system help?

[grijzeduif]

I ran the script commands manually as far I could get:
TS-659
4.1.0
TS-659
thuis

BusyBox v1.01 (2014.05.19-18:40+0000) multi-call binary

eth0 Link encap:Ethernet HWaddr 00:08:9B:BE:04:B2
inet addr:10.0.0.160 Bcast:10.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:158278 errors:0 dropped:1361 overruns:0 frame:0
TX packets:36020 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:18756873 (17.8 MiB) TX bytes:15198194 (14.4 MiB)
Interrupt:18 Memory:feae0000-feb00000

eth1 Link encap:Ethernet HWaddr 00:08:9B:BE:04:B3
UP BROADCAST NOTRAILERS MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:17 Memory:fe9e0000-fea00000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:162406 errors:0 dropped:0 overruns:0 frame:0
TX packets:162406 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:53760048 (51.2 MiB) TX bytes:53760048 (51.2 MiB)

Disk Space:

Filesystem Size Used Available Use% Mounted on
/dev/ram0 139.5M 123.1M 16.3M 88% /
tmpfs 64.0M 212.0k 63.8M 0% /tmp
/dev/sda4 371.0M 250.0M 121.0M 67% /mnt/ext
/dev/md9 509.5M 114.6M 394.8M 22% /mnt/HDA_ROOT
tmpfs 32.0M 0 32.0M 0% /.eaccelerator.tmp


Mount Status:
/proc on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
sysfs on /sys type sysfs (rw)
tmpfs on /tmp type tmpfs (rw,size=64M)
none on /proc/bus/usb type usbfs (rw)
/dev/sda4 on /mnt/ext type ext3 (rw)
/dev/md9 on /mnt/HDA_ROOT type ext3 (rw,data=ordered)
tmpfs on /.eaccelerator.tmp type tmpfs (rw,size=32M)

RAID Status:

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md6 : active raid1 sdf2[2](S) sdd2[3](S) sdc2[4](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]

md13 : active raid1 sde4[5] sda4[0] sdf4[4] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [6/6] [UUUUUU]
bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sda1[0] sdf1[5] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [6/5] [UUUU_U]
bitmap: 64/65 pages [256KB], 4KB chunk

Memory Information:

MemTotal: 1012688 kB
MemFree: 330108 kB

Hope this output can help you a little more.

[pwilson]

I ran the script commands manually as far I could get:
TS-659
4.1.0
TS-659
thuis

BusyBox v1.01 (2014.05.19-18:40+0000) multi-call binary

eth0 Link encap:Ethernet HWaddr 00:08:9B:BE:04:B2
inet addr:10.0.0.160 Bcast:10.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:158278 errors:0 dropped:1361 overruns:0 frame:0
TX packets:36020 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:18756873 (17.8 MiB) TX bytes:15198194 (14.4 MiB)
Interrupt:18 Memory:feae0000-feb00000

eth1 Link encap:Ethernet HWaddr 00:08:9B:BE:04:B3
UP BROADCAST NOTRAILERS MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:17 Memory:fe9e0000-fea00000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:162406 errors:0 dropped:0 overruns:0 frame:0
TX packets:162406 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:53760048 (51.2 MiB) TX bytes:53760048 (51.2 MiB)

Disk Space:

Filesystem Size Used Available Use% Mounted on
/dev/ram0 139.5M 123.1M 16.3M 88% /
tmpfs 64.0M 212.0k 63.8M 0% /tmp
/dev/sda4 371.0M 250.0M 121.0M 67% /mnt/ext
/dev/md9 509.5M 114.6M 394.8M 22% /mnt/HDA_ROOT
tmpfs 32.0M 0 32.0M 0% /.eaccelerator.tmp


Mount Status:
/proc on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
sysfs on /sys type sysfs (rw)
tmpfs on /tmp type tmpfs (rw,size=64M)
none on /proc/bus/usb type usbfs (rw)
/dev/sda4 on /mnt/ext type ext3 (rw)
/dev/md9 on /mnt/HDA_ROOT type ext3 (rw,data=ordered)
tmpfs on /.eaccelerator.tmp type tmpfs (rw,size=32M)

RAID Status:

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath]
md6 : active raid1 sdf2[2](S) sdd2[3](S) sdc2[4](S) sdb2[1] sda2[0]
530048 blocks [2/2] [UU]

md13 : active raid1 sde4[5] sda4[0] sdf4[4] sdd4[3] sdc4[2] sdb4[1]
458880 blocks [6/6] [UUUUUU]
bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sda1[0] sdf1[5] sdd1[3] sdc1[2] sdb1[1]
530048 blocks [6/5] [UUUU_U]
bitmap: 64/65 pages [256KB], 4KB chunk

Memory Information:

MemTotal: 1012688 kB
MemFree: 330108 kB

Hope this output can help you a little more.

Looks like you are still attempting to use failed HDD5. Did you hot-swap it for another drive?
How recent is that backup?

[grijzeduif]

I haven't replaced it yet, I put all 6 original RAID 5 disks in, in a attempt to recreate the situation that I still was able to copy data from NAS to local pc drive.
So still running with the failing disk-5, and the disk-2 in warning state. But without the RAID 5 Logical Volume I cant do anything.

[pwilson]

I haven't replaced it yet, I put all 6 original RAID 5 disks in, in a attempt to recreate the situation that I still was able to copy data from NAS to local pc drive.
So still running with the failing disk-5, and the disk-2 in warning state. But without the RAID 5 Logical Volume I cant do anything.

Stop wasting my time. Good luck.

[grijzeduif]

Sorry Mr.Wilson for making you angry. I don't understand why.
Case=damaged RAID 5 (disk 2 warning, disk 5 damaged), and no spare disk available. Had the idea to create a new volume, with 2x WD Reds in RAID 1 instead. Copied almost all data to local disk. Wanted to rebuild with 2 new 4TB Red's, and noticed that not all data had been copied from the original RAID 5 volume. I put the old RAID 5 disks in again, and hoped to get the situation as before (damaged RAID 5 (disk5) and a disk in warning) and then to be able to copy the forgotten folder.But my RAID 5 volume can't be created anymore.
I don't see why you think of it as wasting your time, as I am very happy that somebody is trying to help me out with a technic (RAID as well Linux) I don't understand.
If you tell me that it is not possible to rebuild the RAID config, than so be it. It was my own stupid failure after all.
Thanks anyway for your help so far. Kind regards

[grijzeduif]

After being a little disappointed by the reaction of moderator P.Wilson about my lack of knowledge I presume (that's what the forum is for imho , helping people who don't understand the technology as good as they would like to) I went to the QNAP helpdesk who fixed my problem the same day . Nevertheless I am glad that people like P.Wilson help people as good as possible, but please don't get frustrated about the difference in knowledge, and state so without an explanation . Kind regard.

[joerideschepper]

What did the helpdesk do to fix your problem?

[grijzeduif]

Hi Joeri,
as you can see in the post I don't have a lot of linux/unix knowledge. I have a putty logfile showing the commands QNAP issued.
In the end my missing files could be found in lost+found/ in a series of numbered folders.
If you don't have the knowledge as I have I recommend opening a case at QNAP. Excellent service.
If you are interested in the logfile, send an email to my personal account as I am not able to attach the log in this post