[HOWTO] Expanding RAID5 from the command line (TS 509)

[olaf]

I've also got the TS-639 Pro and was having a memory error when trying to add a 6th 2 TB disk to a RAID5.

I watched the process closely, and resize2fs was actually using up all of the RAM and the space in the default 500MB swap partition. The process started at around 300MB, then jumped to over 600MB for a while, then really took off (and failed).

I was able to get the process to succeed by adding a couple of swap files to to the system swap. It looks like you may be able to get by if you fill up the remaining space in /mnt/ext and /mnt/HDA_ROOT with two large swap files - I think resize2fs only used another 500-700 MB. Since I like to tempt fate, and was curious to see if I could actually make it work, I just hooked up my 160GB iPod to the QNAP and set up a few 2GB swap files on that. iPod-swap worked like a charm, but to be safe I would strongly suggest trying to make use of the available space on the system's native devices first.

Just remember to remove the swap files from the system when you're done.

I am not very familiar with Linux, so could you please describe step by step instructions to increase the swap file on an external drive via SSH connection.
Thank you in advance

[patrickgraf]

Hi PK_TS509,

I am sad to report that I had to use your post to solve an Add Disk failure on my new TS-410. Happy to report that it fixed my problem... well nearly fixed it.

The array is now 4 disks and 4TB as expected, but in the Web Admin interface it still shows as 3 Disks with the 4th Disk Unmounted. This is obviously wrong, as I can certainly see the full disk space and the lights on the NAS flicker as I would expect when disk access happens i.e. all 4 disk lights flash as I am used to seeing with RAID5 arrays.

So, I am wondering if you know how to fix the Web Admin Interface to tell the truth?

You seem to know what is going on under the hood. I assume using Linux skills. Unfortunately, I'm a very techie Windows person with only limited Linux skills. I will be asking QNAP for support, but thought you might be able to help solve this problem faster than they do.

Thanks for the post and hopefully your help.

Regards Phil

Heyhey
had the same problem after expanding...
I'm new to this whole QNAP NAS THING but here something that worked for me.

I added a 5th disk to my 4disk RAID 5 following the first post. it worked, but also showed 4+1 disk in the web interface.

Solution:

1.In /etc/storage.conf delete the single disk record completely. (in my case the disk /dev/sde3)
2.Then add the disk number to the “raid disks=" line.(in my case I only added ",5" )
After that it should show the right config in your webinterface.


[VOLUME 1]
device name = /dev/sde3
raid level = -2
raid disks = 5
spare raid disks =
status = -3
record_time = Thu Feb 11 10:58:33 2010

filesystem = 103
[VOLUME 2]
device name = /dev/md0
raid level = 5
raid disks = 1,2,3,4
spare raid disks =
status = 0
record_time = Thu Feb 11 10:58:33 2010

filesystem = 103
[Global]
Available Disk = 5
[VOLUME 5]
status = -3
[~] #

Greetings from switzerland ( sorry about my english )

[stoopid]

Thank you!

For the benefit of others, I report my story.

Upon attempting to add a hard drive to my RAID6 (to make a 6 disk RAID from a 5 DISK raid), it got near the end of the process and reported that the expansion had failed. In the Web UI, the size of the array was reported as ~4.2TB. But, mdadm reported it to be ~5.6TB.

I went through these steps and it fixed my issue and now the Web UI reports a size of ~5.6TB.

Disabling journaling in ext4 was the same as ext3.

On a TS-659.

[MichaelTiemann]

I was also successful using resize2fs to expand from a 4-drive RAID6 to a 6-drive RAID6 on a TS-659 (with 2TB drives). I did not have out of memory problems, probably because the x59 series has a full 1GB of RAM. The resize2fs took about 1 hour.

[grayscale]

This HOWTO has saved me a lot of grief. Thank you OP!

I tried Expanding the volume via the web interface and it came back with this log:

2011-01-04 04:25:10 System 127.0.0.1 localhost [RAID5 Disk Volume: Drive 1 2 3 4 5] Expanding Raid Device failed.
2011-01-04 04:23:11 System 127.0.0.1 localhost [RAID5 Disk Volume: Drive 1 2 3 4 5] Rebuilding completed.
2011-01-03 17:47:59 System 127.0.0.1 localhost [RAID5 Disk Volume: Drive 1 2 3 4 5] Start rebuilding.
2011-01-03 17:47:35 System 127.0.0.1 localhost [RAID5 Disk Volume: Drive 1 2 3 4] Start to expand Raid Device: Add Drive 5.
2011-01-03 17:46:46 System 127.0.0.1 localhost Drive 5 plugged in

Despite the Expand task reporting as failed, it still committed the 5th disk to the RAID array. I ran through your HOWTO and now it's reporting accurately.

I have a very noob-ish question to follow up. So I have a 5 disk RAID5 array that when data is being accessed, disks 1-4 blink in sync while disk 5 blinks on an alternating pattern against disks 1-4. My question is, "Are the flashing lights that indicate read/write operations strictly cosmetic or is there something of concern with the 5th disk that was not flashing in sync with disks 1-4?"

Again, I can't begin to thank you enough for the concise directions in regard to this!

[yomeyo]

I've also got the TS-639 Pro and was having a memory error when trying to add a 6th 2 TB disk to a RAID5.

I watched the process closely, and resize2fs was actually using up all of the RAM and the space in the default 500MB swap partition. The process started at around 300MB, then jumped to over 600MB for a while, then really took off (and failed).

I was able to get the process to succeed by adding a couple of swap files to to the system swap. It looks like you may be able to get by if you fill up the remaining space in /mnt/ext and /mnt/HDA_ROOT with two large swap files - I think resize2fs only used another 500-700 MB. Since I like to tempt fate, and was curious to see if I could actually make it work, I just hooked up my 160GB iPod to the QNAP and set up a few 2GB swap files on that. iPod-swap worked like a charm, but to be safe I would strongly suggest trying to make use of the available space on the system's native devices first.

Just remember to remove the swap files from the system when you're done.

This thread helped me alot when solving my problem.

I had the same problem of resize2fs crashing on my TS-859 Pro. dmesg shows that there is not enough memory, although there is 1GB RAM and 512MB swap available. All services were stopped using:

Code: Select all

/etc/init.d/services.sh stop.


I solved it by putting additional swap files on the partitions that had enough free space:

Code: Select all

# Check what partitions have enough free space to create new files to be used as swapspace.
df
# Change the current working directory to one of the partitions that have free space available.
cd /mnt/HDA_ROOT
# Use the dd tool to create an "empty" file with 450MB unallocated space, this was the amount that would fit in my case.
dd if=/dev/zero of=swap1 bs=1M count=450
# Format the file to a swapfile.
mkswap swap1
# Use the file as extra swap space.
swapon swap1

# The same for another partition that had enough free space (130MB).
cd /mnt/ext
dd if=/dev/zero of=swap2 bs=1M count=130
mkswap swap2
swapon swap2

# Check to see if the additional space is added.
free


Now I had > 1GB swap space, which apparently IS enough for resize2fs to complete the expansion

Remember to de-activate and delete the swapfiles afterwards. In my case:

Code: Select all

cd /mnt/HDA_ROOT
swapoff swap1
rm swap1
cd /mnt/ext
swapoff swap2
rm swap2


After the expansion I rebooted, and the filesystem had the appropriate amount of space.

[petur]

Thanks for posting this very informative addition! - Great first post, welcome on the forum

[yomeyo]

Thanks for posting this very informative addition! - Great first post, welcome on the forum

Thank you. I was reading along for a while I've expanded the info in my previous post to make it more clear.

[nogames]

I get this error

Code: Select all

e2fsck 1.41.4 (27-Jan-2009)
e2fsck: Operation not permitted while trying to open /dev/md0
You must have r/w access to the filesystem or be root


My raid is in read only mode, because of a failed drive, I am trying to replace a disk, but my qnap will not add disk, it just says Failed to add drive

Any idea?

[petur]

contact QNAP online support using MSN or skype, and let them look at (and fix) it

[love1997]

Hi,

I would like to add-on to the original steps provided in the initial start of this thread.

For QNAP NAS TS-410 Turbo – 3.4.2 Build 0331T

Under Step 6 – [/etc/init.d] # umount /dev/md0

If you encounter error ( umount : /share/MD0_DATA : device is busy ), you can

1) Use command - /etc/init,d/services.sh stop
2) If you know the process PID, then use command – kill -9 PID_num

Under Step 8 – Turn ext3 or ext4 journaling off

[/etc/init.d] # tune2fs –O ^has_journal /dev/md0
If you have encountered error message such as Wrong or Missing command, it is most probably located in /sbin instead of /etc/init.d

[ /sbin ] # tune2fs –O ^has_journal /dev/md0

Under Step 9 – Extend the RAID

Resize2fs command may not be in /etc/init.d folder, it’s in /sbin
The command is [ /sbin ] # /sbin/resize2fs /dev/md0

For any steps that are not listed above, there is no change and it is working fine.

Regards,
Ed

[LHong]

contact QNAP online support using MSN or skype, and let them look at (and fix) it

I have similar problem, during Expanding RAID5 process from GUI we lost the power. After reboot, the system was hung. So I installed the Original Hard-drive back and roboot the system.
Now we got 5 Disks are unmounted!
I have TS-509 Firmware version 3.4.3-0520T
1> contact QNAP online support using Skype ? What is QNAP online support ID's Skype so that I could get help?
2> It has more than 4TB data that I really want to recover all or part of it as possible! I followed the CMD line steps and ran
[~] # e2fsck -f /dev/md0

Following is the log messages:T

Code: Select all

[~] # e2fsck -f /dev/md0
e2fsck 1.41.4 (27-Jan-2009)
Superblock has an invalid journal (inode 8).
answer=1
*** ext3 journal has been deleted - filesystem is now ext2 only ***

Resize inode not valid.  answer=1
Pass 1: Checking inodes, blocks, and sizes
Inode 149979210 has illegal block(s).  answer=1
Illegal block #12 (3489414239) in inode 149979210.  answer=1
CLEARED.
Illegal block #14 (3194623276) in inode 149979210.  answer=1
CLEARED.
Illegal block #15 (2238558787) in inode 149979210.  answer=1
CLEARED.
Illegal block #16 (3567736189) in inode 149979210.  answer=1
CLEARED.
Illegal block #18 (2742172055) in inode 149979210.  answer=1
CLEARED.
Illegal block #19 (1635741518) in inode 149979210.  answer=1
CLEARED.
Illegal block #21 (4033228849) in inode 149979210.  answer=1
CLEARED.
Illegal block #22 (3348989571) in inode 149979210.  answer=1
CLEARED.
Illegal block #23 (1830172368) in inode 149979210.  answer=1
CLEARED.
Illegal block #24 (4230817166) in inode 149979210.  answer=1
CLEARED.
Illegal block #25 (2223907978) in inode 149979210.  answer=1
CLEARED.
Too many illegal blocks in inode 149979210.
answer=1
Inode 175141045, i_size is 3009359, should be 3014656.  answer=1
Inode 175141045, i_blocks is 5888, should be 5896.  answer=1
Recreate journalanswer=1
Creating journal (32768 blocks):  Done.

*** journal has been re-created - filesystem is now ext3 again ***
Restarting e2fsck from the beginning...
Pass 1: Checking inodes, blocks, and sizes
Extended attribute block 1173985119 has reference count 8, should be 7.  answer=                                              1

Running additional passes to resolve blocks claimed by more than one inode...
Pass 1B: Rescanning for multiply-claimed blocks
Multiply-claimed block(s) in inode 8: 799367372 799367419 799367421 799367422 79                                              9367423 799367424
Multiply-claimed block(s) in inode 399671316: 799367372
Multiply-claimed block(s) in inode 399671317: 799367419 799367421 799367422 7993                                              67423 799367424                                                                            Pa                                 ss 1C: Scanning directories for inodes with multiply-claimed blocks
Pass 1D: Reconciling multiply-claimed blocks
(There are 2 inodes containing multiply-claimed blocks.)

File <The journal inode> (inode #8, mod time Sat Jul 23 22:51:46 2011)
  has 6 multiply-claimed block(s), shared with 2 file(s):
        ??? (inode #399671317, mod time Tue Jul 19 18:44:30 2011)
        ??? (inode #399671316, mod time Sun Jul 17 22:18:50 2011)
answer=1
File ??? (inode #399671316, mod time Sun Jul 17 22:18:50 2011)
  has 1 multiply-claimed block(s), shared with 1 file(s):
        <The journal inode> (inode #8, mod time Sat Jul 23 22:51:46 2011)
Multiply-claimed blocks already reassigned or cloned.

File ??? (inode #399671317, mod time Tue Jul 19 18:44:30 2011)
  has 5 multiply-claimed block(s), shared with 1 file(s):
        <The journal inode> (inode #8, mod time Sat Jul 23 22:51:46 2011)
Multiply-claimed blocks already reassigned or cloned.

Pass 2: Checking directory structure
Error allocating icount structure: Memory allocation failed
e2fsck: aborted
[~] #


How do we remount the drives?
Please advise!

[k995]

THX worked

Added a hard drive to the raid and it showed in the raid but the extra space wasnt there.

Followed the instructions and everything OK now .

Only had a "device is busy" with the " umount : /share/MD0_DATA"

Solved it by rebooting and running sh services.sh stop .

Thx again.

[paolomarino]

*G*R*E*A*T* Guide

Thanks. Worked like a charm on a 509 upgrading from 5*1TB to 5*2TB

I think the problem of the failing of the standard web expanding procedure is the lack of memory in 509
So I used this guide to manual expand the raid with the integration of creating a bigger swap file on eSataDisk1 and all went fine.

[SoMoney]

QNAP 419P was a little different:
Volume did not re mount for me at 1st! I was getting that memory error and decided to reboot once to see if it would reclaim some memory.. When the QNAP came back the /etc/mtab didnt have /dev/md0 in it anymore..
Luckily I saved ALL OUTPUT in NOTEPAD!

[~] # /etc/init.d/services.sh stop
[~] # /etc/init.d/Qthttpd.sh stop #<<<<---- BUG Cant unmount md0 if Qthttpd is still using it!
[~] # cat /etc/mtab #<<<<----- SAVE SAVE SAVE THIS OUTPUT! I NEEDED THE MD0 INFO TO RESTORE.
[~] # tune2fs -O ^has_journal /dev/md0
[~] # e2fsck -n /dev/md0;resize2fs /dev/md0 #<<<<<----- I added swap space but still got memory errors! When I did a back2back e2fsk & resize it worked! Dont know why.
[~] # tune2fs -j /dev/md0
[~] # echo "/dev/md0 /share/MD0_DATA ext4 rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,delalloc,noacl 0 0" >> /etc/mtab
[~] # mount /dev/md0
[~] # reboot
FIXED for my ext4 raid