[HOWTO] Expanding RAID5 from the command line (TS 509)

Questions about SNMP, Power, System, Logs, disk, & RAID.
PK_TS509
First post
Posts: 1
Joined: Mon Oct 27, 2008 1:20 am

[HOWTO] Expanding RAID5 from the command line (TS 509)

Post by PK_TS509 »

Hi all,

I'm seem to be having the same issues expanding my TS 509 as described in this post:

http://forum.qnap.com/viewtopic.php?f=25&t=9083

Now the solution does look good but seems to be lacking some steps.

So this is what I think the process should be, please feel free to correct and expand any steps listed below. The only thing that isn't working is the RAID status in the Web Admin, as this still reports only 3 drives in the RAID - but does report the free disk space correctly.

** DISCLAIMER ** This worked for me, it might not work for you - backup first!!

-----------------------------------------------------------------------------------------------------------------------------
1.) ssh onto your TS box.

Use your favorite SSH client to log onto the TS*** and become the root user.

2.) Check RAID size.

Code: Select all

[/etc/init.d] # df /dev/md0
Filesystem                Size      Used Available Use% Mounted on
/dev/md0                  2.7T      2.7T      4.7G 100% /share/MD0_DATA
Not a lot of space left :)

3.) Check to see if all disks are present with the mdadm command.

Code: Select all

[/etc/init.d] # mdadm -D /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Tue Oct 21 22:34:19 2008
     Raid Level : raid5
     Array Size : 4390708800 (4187.31 GiB 4496.09 GB)
  Used Dev Size : 1463569600 (1395.77 GiB 1498.70 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sat Feb 28 19:59:42 2009
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : b95bf024:8c1517d6:af35b4a1:c06378bd
         Events : 0.2441216

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3
       2       8       35        2      active sync   /dev/sdc3
       3       8       51        3      active sync   /dev/sdd3
For me the /dev/sdd line above indicates that I have my new 4th drive visible to the system.

4.) Check what type of filesystem the RAID is by checking mtab.

Code: Select all

[/etc/init.d] # cat /etc/mtab 
/proc /proc proc rw 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0
sysfs /sys sysfs rw 0 0
tmpfs /tmp tmpfs rw,size=32M 0 0
usbfs /proc/bus/usb usbfs rw 0 0
/dev/sda4 /mnt/ext ext3 rw 0 0
/dev/md9 /mnt/HDA_ROOT ext3 rw 0 0
/dev/md0 /share/MD0_DATA ext3 rw,usrjquota=aquota.user,jqfmt=vfsv0,data=writeback 0 0
The last line shows that the current RAID is ext3.

5.) Turn off all network mounts.

From the Web Admin page go to the Network Settings and disable the following services:
  • • Microsoft Networking
    • Apple Networking
    • NFS Service
    • Web File Manager
    • FTP Service
    • Multimedia Station
    • iTunes Service
    • Download Station
    • Web Server
    • MySQL Server
6.) Unmounting the volume.

From the command line run the following:

Code: Select all

[/etc/init.d] # umount /dev/md0
If you see this message after running the above command stop, this means that you have not disabled all of the network services.

Code: Select all

umount: /share/MD0_DATA: device is busy
7.) File system check.

Run the e2fsck command to check your filesystem for errors, this may take a while:

Code: Select all

[/etc/init.d] # e2fsck -f /dev/md0
e2fsck 1.40.4 (31-Dec-2007)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/md0: 51686/365903872 files (11.0% non-contiguous), 730560098/731784800 blocks
8.) Turn ext3 journaling off

As step 4 showed that the RAID was using ext3 we need to turn off the journaling prior to expansion.

Code: Select all

[/etc/init.d] # tune2fs -O ^has_journal /dev/md0
9.) Extend the RAID, this will take a while.

Code: Select all

[/etc/init.d] # resize2fs /dev/md0
resize2fs 1.40.4 (31-Dec-2007)
Resizing the filesystem on /dev/md0 to 1097677200 (4k) blocks.
The filesystem on /dev/md0 is now 1097677200 blocks long.
10.) Recheck the filesystem - using the -n option so no changes are made.

Code: Select all

[/etc/init.d] # e2fsck -n /dev/md0
e2fsck 1.40.4 (31-Dec-2007)
/dev/md0: clean, 51686/365903872 files, 730560098/731784800 blocks
This looks good as the number of blocks reported above has increased from what we saw in Step 7 above.

11.) Turn the journaling back on.

Code: Select all

[/etc/init.d] # tune2fs -j /dev/md0
tune2fs 1.40.4 (31-Dec-2007)
Creating journal inode: done
This filesystem will be automatically checked every -1 mounts or
0 days, whichever comes first.  Use tune2fs -c or -i to override.
12.) Reboot the system.

Just to make sure everything is in order do a reboot to bring everything up cleanly:

Code: Select all

[/etc/init.d] # reboot
13.) Check RAID size.

SSH back on and check your RAID has been expanded.

Code: Select all

[/etc/init.d] # df /dev/md0
Filesystem                Size      Used Available Use% Mounted on
/dev/md0                  4.0T      2.7T      1.3T  67% /share/MD0_DATA
14.) Re enable Network Services.

Turn the Network services you want to use back on with the Web Admin interface.

-----------------------------------------------------------------------------------------------------------------------------
Last edited by PK_TS509 on Sun Mar 29, 2009 12:28 am, edited 1 time in total.
SpikeNbg
Starting out
Posts: 18
Joined: Fri Sep 12, 2008 5:10 pm

Re: [TS 509] Expanding RAID5 from the command line

Post by SpikeNbg »

Hello PK_TS509!

Thank you very very much for this excellent description! :D

Your solution worked perfekt for me!

(And i must say, the support wasn't very helpful over Skype..)

Bye Spike
User avatar
petur
Moderator
Posts: 4606
Joined: Sun Mar 30, 2008 5:42 pm
Location: Gent, Belgium
Contact:

Re: [TS 509] Expanding RAID5 from the command line

Post by petur »

Can you make your topic line include the text [HOWTO], I'll ask a mod to make this post sticky.
Praat je liever over QNAP in het Nederlands?
Liever een community bij jou in de buurt?

Kom naar QNAPclub België/Nederland
jonninemonic
New here
Posts: 7
Joined: Tue Aug 18, 2009 2:05 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by jonninemonic »

Hi!

I have a TS-639 and want to go from 5 to 6 disks (2 TB each) - a total of 10 TB. When executing the resize2fs command i get the message

Code: Select all

[~] # resize2fs /dev/md0
resize2fs 1.41.4 (27-Jan-2009)
Resizing the filesystem on /dev/md0 to 2439932000 (4k) blocks.
resize2fs: Memory allocation failed while trying to resize /dev/md0
[~] #
Any suggestions?

/Jonni
fr0zen
New here
Posts: 5
Joined: Wed Apr 15, 2009 5:46 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by fr0zen »

actually this is ridiculous! how come there is no patch from QNap on this?

i am having the same problem and i dont want to do all this on a live production NAS.
fr0zen
New here
Posts: 5
Joined: Wed Apr 15, 2009 5:46 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by fr0zen »

well it didnt work for me :(

it said the block sizes are the same or something, nothing to do...which is BS, i just put in 2 x 2TB drives and the available and total space is the exact same.

QNAP Support = **!!
fr0zen
New here
Posts: 5
Joined: Wed Apr 15, 2009 5:46 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by fr0zen »

The following shows up at the error for me. I have already put in 2 new drives, 1x 1.5TB and 1x 2TB, and the total data size available is still the same

Code: Select all

[~] # resize2fs /dev/md0
resize2fs 1.41.4 (27-Jan-2009)
The filesystem is already 913756960 blocks long.  Nothing to do!
Atei
First post
Posts: 1
Joined: Thu Sep 10, 2009 3:30 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by Atei »

Awsome description works perfect with my qnap, ext4 this is also working exactly like ext3
Markeh
New here
Posts: 2
Joined: Thu Jun 26, 2008 10:43 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by Markeh »

Hi,

I have a TS 409 that I want to put 4x 1TB disks in; it now has 4x 500GB disks. Will the procedure below work on this model also?
If there is an easier way, I would really appreciate it. The poor quality of the QNAP software makes me crazy. I've already had one major system failure. I cannot afford another.

Tnx,
markeh
UnkyPhil
New here
Posts: 2
Joined: Mon Dec 28, 2009 5:35 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by UnkyPhil »

Hi PK_TS509,

I am sad to report that I had to use your post to solve an Add Disk failure on my new TS-410. Happy to report that it fixed my problem... well nearly fixed it.

The array is now 4 disks and 4TB as expected, but in the Web Admin interface it still shows as 3 Disks with the 4th Disk Unmounted. This is obviously wrong, as I can certainly see the full disk space and the lights on the NAS flicker as I would expect when disk access happens i.e. all 4 disk lights flash as I am used to seeing with RAID5 arrays.

So, I am wondering if you know how to fix the Web Admin Interface to tell the truth?

You seem to know what is going on under the hood. I assume using Linux skills. Unfortunately, I'm a very techie Windows person with only limited Linux skills. I will be asking QNAP for support, but thought you might be able to help solve this problem faster than they do.

Thanks for the post and hopefully your help.

Regards Phil
abmagfab
Getting the hang of things
Posts: 77
Joined: Thu Dec 10, 2009 12:34 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by abmagfab »

For stopping services, I'd stongly recommend using /etrc/init.d/services.sh stop, instead of using the web UI to turn off everything. If you do it like the sticky says, for example with Windows, you'll remove it from the domain and have to re-add it - not a good idea.

Also make sure to temporarily turn off bitmap during the process if you have it on.
Last edited by abmagfab on Sat Jan 16, 2010 1:37 pm, edited 1 time in total.
abmagfab
Getting the hang of things
Posts: 77
Joined: Thu Dec 10, 2009 12:34 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by abmagfab »

And another helpful hint - if you're having a problem even adding a new drive to the array, you'll need to manually grow the array.

Just:
1) Telnet/SSH in
2) Check the array using mdadm -D /dev/md0
2a) It should show the current array, with the new drive as a "spare"
3) Do an mdadm -E /dev/sda3 (or whatever) for each of the drives, including the new one - they should all look the same, with all drives listed for each mdadm you run
4) Do an mdadm --grow /dev/md0 --raid-devices=<total number of drives with new drive>

Then you can cat /proc/mdstat to see the progress.

If you want to speed it up, you can do something like:
echo 40000 > /proc/sys/dev/raid/speed_limit_min

The default min is 5000, but the above should keep it at ~25000 for most of the rebuild, then around 35-40000 when it gets to the free space.

5) Wait about 24 hours for a 2TB additional drive
6) Follow the steps in the first post to expand the array to use the new drive. This should only take about 10 minutes
7) You likely will now have a web UI with information that doesn't quite match reality. Edit the files indicated towards the end of this thread to correct that:
http://forum.qnap.com/viewtopic.php?f=25&t=24455

I have no idea why the web UI stops working after a while for adding/expanding, but for some reason it does. It worked for me for a couple, but now I have to do the --grow and then the expand steps at the front of this thread. Weird, huh?

Hope this helps!
Klepo
First post
Posts: 1
Joined: Fri Jan 29, 2010 3:01 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by Klepo »

I've also got the TS-639 Pro and was having a memory error when trying to add a 6th 2 TB disk to a RAID5.

I watched the process closely, and resize2fs was actually using up all of the RAM and the space in the default 500MB swap partition. The process started at around 300MB, then jumped to over 600MB for a while, then really took off (and failed).

I was able to get the process to succeed by adding a couple of swap files to to the system swap. It looks like you may be able to get by if you fill up the remaining space in /mnt/ext and /mnt/HDA_ROOT with two large swap files - I think resize2fs only used another 500-700 MB. Since I like to tempt fate, and was curious to see if I could actually make it work, I just hooked up my 160GB iPod to the QNAP and set up a few 2GB swap files on that. iPod-swap worked like a charm, but to be safe I would strongly suggest trying to make use of the available space on the system's native devices first.

Just remember to remove the swap files from the system when you're done.
abmagfab
Getting the hang of things
Posts: 77
Joined: Thu Dec 10, 2009 12:34 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by abmagfab »

Would you mind posting the command line for how you did this? Just to have it captured for anyone that needs to do this in the future.

Thanks!
olaf
New here
Posts: 2
Joined: Mon Aug 17, 2009 6:17 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by olaf »

Klepo wrote:I've also got the TS-639 Pro and was having a memory error when trying to add a 6th 2 TB disk to a RAID5.

I watched the process closely, and resize2fs was actually using up all of the RAM and the space in the default 500MB swap partition. The process started at around 300MB, then jumped to over 600MB for a while, then really took off (and failed).

I was able to get the process to succeed by adding a couple of swap files to to the system swap. It looks like you may be able to get by if you fill up the remaining space in /mnt/ext and /mnt/HDA_ROOT with two large swap files - I think resize2fs only used another 500-700 MB. Since I like to tempt fate, and was curious to see if I could actually make it work, I just hooked up my 160GB iPod to the QNAP and set up a few 2GB swap files on that. iPod-swap worked like a charm, but to be safe I would strongly suggest trying to make use of the available space on the system's native devices first.

Just remember to remove the swap files from the system when you're done.
I am not very familiar with Linux, so could you please describe step by step instructions to increase the swap file on an external drive via SSH connection.
Thank you in advance
Post Reply

Return to “System & Disk Volume Management”