[HOWTO] Expanding RAID5 from the command line (TS 509)

Questions about SNMP, Power, System, Logs, disk, & RAID.

[HOWTO] Expanding RAID5 from the command line (TS 509)

Postby PK_TS509 » Sun Mar 01, 2009 5:41 am

Hi all,

I'm seem to be having the same issues expanding my TS 509 as described in this post:

viewtopic.php?f=25&t=9083

Now the solution does look good but seems to be lacking some steps.

So this is what I think the process should be, please feel free to correct and expand any steps listed below. The only thing that isn't working is the RAID status in the Web Admin, as this still reports only 3 drives in the RAID - but does report the free disk space correctly.

** DISCLAIMER ** This worked for me, it might not work for you - backup first!!

-----------------------------------------------------------------------------------------------------------------------------
1.) ssh onto your TS box.

Use your favorite SSH client to log onto the TS*** and become the root user.

2.) Check RAID size.

Code: Select all
[/etc/init.d] # df /dev/md0
Filesystem                Size      Used Available Use% Mounted on
/dev/md0                  2.7T      2.7T      4.7G 100% /share/MD0_DATA


Not a lot of space left :)

3.) Check to see if all disks are present with the mdadm command.

Code: Select all
[/etc/init.d] # mdadm -D /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Tue Oct 21 22:34:19 2008
     Raid Level : raid5
     Array Size : 4390708800 (4187.31 GiB 4496.09 GB)
  Used Dev Size : 1463569600 (1395.77 GiB 1498.70 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sat Feb 28 19:59:42 2009
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : b95bf024:8c1517d6:af35b4a1:c06378bd
         Events : 0.2441216

    Number   Major   Minor   RaidDevice State
       0       8        3        0      active sync   /dev/sda3
       1       8       19        1      active sync   /dev/sdb3
       2       8       35        2      active sync   /dev/sdc3
       3       8       51        3      active sync   /dev/sdd3


For me the /dev/sdd line above indicates that I have my new 4th drive visible to the system.

4.) Check what type of filesystem the RAID is by checking mtab.

Code: Select all
[/etc/init.d] # cat /etc/mtab
/proc /proc proc rw 0 0
none /dev/pts devpts rw,gid=5,mode=620 0 0
sysfs /sys sysfs rw 0 0
tmpfs /tmp tmpfs rw,size=32M 0 0
usbfs /proc/bus/usb usbfs rw 0 0
/dev/sda4 /mnt/ext ext3 rw 0 0
/dev/md9 /mnt/HDA_ROOT ext3 rw 0 0
/dev/md0 /share/MD0_DATA ext3 rw,usrjquota=aquota.user,jqfmt=vfsv0,data=writeback 0 0


The last line shows that the current RAID is ext3.

5.) Turn off all network mounts.

From the Web Admin page go to the Network Settings and disable the following services:

    • Microsoft Networking
    • Apple Networking
    • NFS Service
    • Web File Manager
    • FTP Service
    • Multimedia Station
    • iTunes Service
    • Download Station
    • Web Server
    • MySQL Server

6.) Unmounting the volume.

From the command line run the following:

Code: Select all
[/etc/init.d] # umount /dev/md0


If you see this message after running the above command stop, this means that you have not disabled all of the network services.

Code: Select all
umount: /share/MD0_DATA: device is busy


7.) File system check.

Run the e2fsck command to check your filesystem for errors, this may take a while:

Code: Select all
[/etc/init.d] # e2fsck -f /dev/md0
e2fsck 1.40.4 (31-Dec-2007)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/md0: 51686/365903872 files (11.0% non-contiguous), 730560098/731784800 blocks


8.) Turn ext3 journaling off

As step 4 showed that the RAID was using ext3 we need to turn off the journaling prior to expansion.

Code: Select all
[/etc/init.d] # tune2fs -O ^has_journal /dev/md0


9.) Extend the RAID, this will take a while.

Code: Select all
[/etc/init.d] # resize2fs /dev/md0
resize2fs 1.40.4 (31-Dec-2007)
Resizing the filesystem on /dev/md0 to 1097677200 (4k) blocks.
The filesystem on /dev/md0 is now 1097677200 blocks long.


10.) Recheck the filesystem - using the -n option so no changes are made.

Code: Select all
[/etc/init.d] # e2fsck -n /dev/md0
e2fsck 1.40.4 (31-Dec-2007)
/dev/md0: clean, 51686/365903872 files, 730560098/731784800 blocks


This looks good as the number of blocks reported above has increased from what we saw in Step 7 above.

11.) Turn the journaling back on.

Code: Select all
[/etc/init.d] # tune2fs -j /dev/md0
tune2fs 1.40.4 (31-Dec-2007)
Creating journal inode: done
This filesystem will be automatically checked every -1 mounts or
0 days, whichever comes first.  Use tune2fs -c or -i to override.


12.) Reboot the system.

Just to make sure everything is in order do a reboot to bring everything up cleanly:

Code: Select all
[/etc/init.d] # reboot


13.) Check RAID size.

SSH back on and check your RAID has been expanded.

Code: Select all
[/etc/init.d] # df /dev/md0
Filesystem                Size      Used Available Use% Mounted on
/dev/md0                  4.0T      2.7T      1.3T  67% /share/MD0_DATA


14.) Re enable Network Services.

Turn the Network services you want to use back on with the Web Admin interface.

-----------------------------------------------------------------------------------------------------------------------------
Last edited by PK_TS509 on Sun Mar 29, 2009 12:28 am, edited 1 time in total.
PK_TS509
First post
 
Posts: 1
Joined: Mon Oct 27, 2008 1:20 am
NAS Model: TS-419P

Re: [TS 509] Expanding RAID5 from the command line

Postby SpikeNbg » Tue Mar 24, 2009 7:51 am

Hello PK_TS509!

Thank you very very much for this excellent description! :D

Your solution worked perfekt for me!

(And i must say, the support wasn't very helpful over Skype..)

Bye Spike
SpikeNbg
New here
 
Posts: 9
Joined: Fri Sep 12, 2008 5:10 pm
NAS Model: TS-419P

Re: [TS 509] Expanding RAID5 from the command line

Postby petur » Tue Mar 24, 2009 4:54 pm

Can you make your topic line include the text [HOWTO], I'll ask a mod to make this post sticky.
User avatar
petur
Moderator
 
Posts: 4492
Joined: Sun Mar 30, 2008 5:42 pm
Location: Gent, Belgium
NAS Model: TS-x69 Pro

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Postby jonninemonic » Mon Aug 31, 2009 4:37 pm

Hi!

I have a TS-639 and want to go from 5 to 6 disks (2 TB each) - a total of 10 TB. When executing the resize2fs command i get the message
Code: Select all
[~] # resize2fs /dev/md0
resize2fs 1.41.4 (27-Jan-2009)
Resizing the filesystem on /dev/md0 to 2439932000 (4k) blocks.
resize2fs: Memory allocation failed while trying to resize /dev/md0
[~] #


Any suggestions?

/Jonni
jonninemonic
New here
 
Posts: 7
Joined: Tue Aug 18, 2009 2:05 pm
NAS Model: TS-239 Pro

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Postby fr0zen » Mon Nov 02, 2009 8:22 am

actually this is ridiculous! how come there is no patch from QNap on this?

i am having the same problem and i dont want to do all this on a live production NAS.
fr0zen
New here
 
Posts: 5
Joined: Wed Apr 15, 2009 5:46 am
NAS Model: TS-239 Pro

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Postby mistrall » Wed Nov 04, 2009 10:39 pm

Thank you for the instructions! it helped a lot.
It's a pity that qnap software is still so bug'y.
mistrall
First post
 
Posts: 1
Joined: Sun Jul 12, 2009 4:49 am
NAS Model: TS-119/219

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Postby fr0zen » Sun Nov 08, 2009 5:12 am

well it didnt work for me :(

it said the block sizes are the same or something, nothing to do...which is BS, i just put in 2 x 2TB drives and the available and total space is the exact same.

QNAP Support = **!!
fr0zen
New here
 
Posts: 5
Joined: Wed Apr 15, 2009 5:46 am
NAS Model: TS-239 Pro

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Postby fr0zen » Sun Nov 08, 2009 9:32 pm

The following shows up at the error for me. I have already put in 2 new drives, 1x 1.5TB and 1x 2TB, and the total data size available is still the same

Code: Select all
[~] # resize2fs /dev/md0
resize2fs 1.41.4 (27-Jan-2009)
The filesystem is already 913756960 blocks long.  Nothing to do!
fr0zen
New here
 
Posts: 5
Joined: Wed Apr 15, 2009 5:46 am
NAS Model: TS-239 Pro

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Postby Atei » Fri Nov 27, 2009 5:46 am

Awsome description works perfect with my qnap, ext4 this is also working exactly like ext3
Atei
First post
 
Posts: 1
Joined: Thu Sep 10, 2009 3:30 am
NAS Model: TS-419U

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Postby Markeh » Tue Dec 08, 2009 6:15 am

Hi,

I have a TS 409 that I want to put 4x 1TB disks in; it now has 4x 500GB disks. Will the procedure below work on this model also?
If there is an easier way, I would really appreciate it. The poor quality of the QNAP software makes me crazy. I've already had one major system failure. I cannot afford another.

Tnx,
markeh
Markeh
New here
 
Posts: 2
Joined: Thu Jun 26, 2008 10:43 pm
NAS Model: TS-109/209 Pro II

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Postby UnkyPhil » Mon Dec 28, 2009 6:01 pm

Hi PK_TS509,

I am sad to report that I had to use your post to solve an Add Disk failure on my new TS-410. Happy to report that it fixed my problem... well nearly fixed it.

The array is now 4 disks and 4TB as expected, but in the Web Admin interface it still shows as 3 Disks with the 4th Disk Unmounted. This is obviously wrong, as I can certainly see the full disk space and the lights on the NAS flicker as I would expect when disk access happens i.e. all 4 disk lights flash as I am used to seeing with RAID5 arrays.

So, I am wondering if you know how to fix the Web Admin Interface to tell the truth?

You seem to know what is going on under the hood. I assume using Linux skills. Unfortunately, I'm a very techie Windows person with only limited Linux skills. I will be asking QNAP for support, but thought you might be able to help solve this problem faster than they do.

Thanks for the post and hopefully your help.

Regards Phil
UnkyPhil
New here
 
Posts: 2
Joined: Mon Dec 28, 2009 5:35 pm
NAS Model: TS-410

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Postby abmagfab » Sun Jan 10, 2010 6:55 am

For stopping services, I'd stongly recommend using /etrc/init.d/services.sh stop, instead of using the web UI to turn off everything. If you do it like the sticky says, for example with Windows, you'll remove it from the domain and have to re-add it - not a good idea.

Also make sure to temporarily turn off bitmap during the process if you have it on.
Last edited by abmagfab on Sat Jan 16, 2010 1:37 pm, edited 1 time in total.
abmagfab
Getting the hang of things
 
Posts: 77
Joined: Thu Dec 10, 2009 12:34 am
NAS Model: TS-639U

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Postby abmagfab » Fri Jan 15, 2010 12:37 pm

And another helpful hint - if you're having a problem even adding a new drive to the array, you'll need to manually grow the array.

Just:
1) Telnet/SSH in
2) Check the array using mdadm -D /dev/md0
2a) It should show the current array, with the new drive as a "spare"
3) Do an mdadm -E /dev/sda3 (or whatever) for each of the drives, including the new one - they should all look the same, with all drives listed for each mdadm you run
4) Do an mdadm --grow /dev/md0 --raid-devices=<total number of drives with new drive>

Then you can cat /proc/mdstat to see the progress.

If you want to speed it up, you can do something like:
echo 40000 > /proc/sys/dev/raid/speed_limit_min

The default min is 5000, but the above should keep it at ~25000 for most of the rebuild, then around 35-40000 when it gets to the free space.

5) Wait about 24 hours for a 2TB additional drive
6) Follow the steps in the first post to expand the array to use the new drive. This should only take about 10 minutes
7) You likely will now have a web UI with information that doesn't quite match reality. Edit the files indicated towards the end of this thread to correct that:
viewtopic.php?f=25&t=24455

I have no idea why the web UI stops working after a while for adding/expanding, but for some reason it does. It worked for me for a couple, but now I have to do the --grow and then the expand steps at the front of this thread. Weird, huh?

Hope this helps!
abmagfab
Getting the hang of things
 
Posts: 77
Joined: Thu Dec 10, 2009 12:34 am
NAS Model: TS-639U

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Postby Klepo » Mon Feb 01, 2010 7:49 am

I've also got the TS-639 Pro and was having a memory error when trying to add a 6th 2 TB disk to a RAID5.

I watched the process closely, and resize2fs was actually using up all of the RAM and the space in the default 500MB swap partition. The process started at around 300MB, then jumped to over 600MB for a while, then really took off (and failed).

I was able to get the process to succeed by adding a couple of swap files to to the system swap. It looks like you may be able to get by if you fill up the remaining space in /mnt/ext and /mnt/HDA_ROOT with two large swap files - I think resize2fs only used another 500-700 MB. Since I like to tempt fate, and was curious to see if I could actually make it work, I just hooked up my 160GB iPod to the QNAP and set up a few 2GB swap files on that. iPod-swap worked like a charm, but to be safe I would strongly suggest trying to make use of the available space on the system's native devices first.

Just remember to remove the swap files from the system when you're done.
Klepo
First post
 
Posts: 1
Joined: Fri Jan 29, 2010 3:01 pm
NAS Model: TS-509 Pro

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Postby abmagfab » Mon Feb 01, 2010 8:48 am

Would you mind posting the command line for how you did this? Just to have it captured for anyone that needs to do this in the future.

Thanks!
abmagfab
Getting the hang of things
 
Posts: 77
Joined: Thu Dec 10, 2009 12:34 am
NAS Model: TS-639U

Next

Return to System & Disk Volume Management

Who is online

Users browsing this forum: jup and 12 guests