[HOWTO] Expanding RAID5 from the command line (TS 509)

Questions about SNMP, Power, System, Logs, disk, & RAID.
soar
First post
Posts: 1
Joined: Sun Feb 10, 2013 9:02 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by soar »

Thanks for your guide, I experience the same issue. Adding a fourth hdd to a raid 5 array. Expansion failed.

Followed your guide, got stuck with memory problem during resize. Created an additional swap space of 350M and then it worked as described.

Thanks a lot, saved my day!!


QNAP please fix this problem asap, since this is one of your top marketing topics that expansion/migration works like a charm. BUT IT DOES NOT RIGHT NOW.
rodi22
New here
Posts: 2
Joined: Thu Apr 08, 2010 7:40 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by rodi22 »

Had the same issues (4x2TB disks replaced by 4x4TB, failed via web) and solved it by following the guides here.

However some additional comments referring to page 1 of this thread.
step 6) if you cannot unmount /dev/md0:
find out which processes are blocking the unmount by:
# lsof +f -- /dev/md0

kill those processes with:
# kill -9 <PID>

recheck with
# lsof +f -- /dev/md0

If nothing is being listed, now you should be able to unmount /dev/md0.

step 9)

I had the problem that the resize described in step 9 failed (as described on page 1 of this thread)
"resize2fs: Memory allocation failed while trying to resize /dev/md0"

Solution: add more swap space (input from this thread: http://forum.qnap.com/viewtopic.php?t=1510&start=10)
a) plugin a USB stick (i had one, which was formatted with fat32 on a windows system) (it should be mounted automatically on the QNAP)
b) enter "# mount" to find out the device name of the stick (assume it is "/share/external/sdi1")
c) "# cd /share/external/sdi1"
d) create an empty file (1GB big) :
"# dd if=/dev/zero of=swapfile bs=1024 count=1024k"
e) prepare the file for swapping:
"# mkswap -c swapfile"
f) tell the system to use the file for swapping
"# swapon /share/external/sdi1/swapfile"
g) check and see if the system has accepted the file as a swap destination:
"# cat /proc/swaps"

now continue (or retry) with step 9): "# resize2fs /dev/md0" (with the addtitional swap on a USB stick it worked for me.

follow the other steps 10), 11),.... (from page 1 of this thread)
leistiko
New here
Posts: 4
Joined: Mon Feb 07, 2011 4:41 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by leistiko »

Excellent guide. All steps working except the crucial one. When doing the resize, I get a "The filesystem is already ... blocks long. Nothing to do!" I just upgraded from 8x2TB to 8x3TB and wanted to try, if the resize2fs_64 is already capable of going over that 16TB limit. After having replaced the individual discs using the GUI functions, the sd[x]3 partitions the raid resides on are already 2999GB according to parted. Maybe I missed something, maybe its the resize2fs tools. Does anyone else has this problem?
musashi77
Starting out
Posts: 17
Joined: Mon Jan 28, 2008 8:02 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by musashi77 »

I gotta that I, extremely disaoppointed that I need to go to this level to extend my RAID and wish I never started cause I think Ive gone past the point of no return.

Anyway I am having troubles unmounting the RAID on TS-409.
When I put in the command I get this.
[~] # umount /dev/md0
umount: /dev/md0: not mounted


However when I go to check it I get the following
[~] # e2fsck -f /dev/md0
e2fsck 1.40-WIP (02-Oct-2006)
/dev/md0 is mounted.

WARNING!!! Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.

Do you really want to continue (y/n)?


I attempted to stop the services with the "services.sh stop" and "/etc/init.d/Qthttpd.sh stop"
However no luck still and when I run the following command to try and identify the problem I get the follow:
[~] # lsof +f -- /dev/md0
-sh: lsof: command not found


Any ideas?
musashi77
Starting out
Posts: 17
Joined: Mon Jan 28, 2008 8:02 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by musashi77 »

Little help please...
Running step four just sits there, doesn't give my anything except the version number and date. Not even a new line to put a command in.

# tune2fs -O ^has_journal /dev/md0
tune2fs 1.40-WIP (02-Oct-2006)
philhu
Know my way around
Posts: 157
Joined: Fri May 03, 2013 12:17 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by philhu »

Trying to resize a 3tb and 2x4tb disks, I replaced the 3tb one so I had 3x4tb.

I was seeing the gui resize fail. So I watched it in ssh console. It seemed ok, but failed in resize2fs, so I tried the items in this thread. It is failing as such:

Code: Select all

[/etc/rcS_init.d] # /etc/init.d/services.sh stop
Stop qpkg service: Squid is disabled, disable Squid rotate in crontab.
/etc/rcK.d/QK102Squid: line 71: /share/HDA_DATA/.qpkg/Squid/opt/etc/init.d/S80squid: No such file or directory
Disable Optware/ipkg
Shutting down SABnzbdplus... 
.
Stop service: cloud3p.sh vpn_openvpn.sh vpn_pptp.sh ldap_server.sh antivirus.sh iso_mount.sh qsyncman.sh rsyslog.sh snmp nvrd.sh lunportman.sh iscsitrgt.sh smb.sh nfs crond.sh Qthttpd.sh twonkymedia.sh init_iTune.sh ImRd.sh StartMediaService.sh bt_scheduler.sh btd.sh ftp.sh atalk.sh mysqld.sh recycled.sh .

[/etc/rcS_init.d] # umount /dev/md0             

[/etc/rcS_init.d] # e2fsck -f /dev/md0
e2fsck 1.41.4 (27-Jan-2009)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/md0: 143356/366092288 files (17.5% non-contiguous), 468829977/1464348768 blocks

[/etc/rcS_init.d] # resize2fs /dev/md0
resize2fs 1.41.4 (27-Jan-2009)
Resizing the filesystem on /dev/md0 to 1952724768 (4k) blocks.
resize2fs: Memory allocation failed while trying to resize /dev/md0
As you see, it is failing on memory allocation. I have a TS-412, so I assume am hitting the 256m only in the system.

Any ideas how to proceed???
philhu
Know my way around
Posts: 157
Joined: Fri May 03, 2013 12:17 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by philhu »

I fixed it.....I found other docs for adding swap files....

The TS-412 is obviously too low in memory for large operations like this.

So I took a 4G UCB Stick, put it in usb3, made a swapfile with dd, mkswap, added the swapfile
and then used the manual instructions at the top of this thread to do all the steps.

worked fine.

I think I am going to leave the little 4gb swapfile in place as rthe ts-412 is a bit low on memory for special operations...That might just make it all a bit more stable.
AmoLago
New here
Posts: 4
Joined: Mon Jul 16, 2012 4:26 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by AmoLago »

Thanks guys, with the combination of the first post, the useful /etc/init.d/services.sh stop and the post regarding introducing an extra swap file, I have successfully (finally) been able get my extra 2TB in to the RAID 5. At least I wont have to do it again now for a couple of years! ;)
ChillyWilly
Starting out
Posts: 21
Joined: Fri Jun 12, 2009 4:02 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by ChillyWilly »

For people who have not done the raid expansion in the web interface, you need to grow the raid first, before you can resize2fs.

Code: Select all

mdadm --grow /dev/md0 --size max
And you can the -p option so you get some output on the progress

Code: Select all

[~] # resize2fs -p /dev/md0
resize2fs 1.41.4 (27-Jan-2009)
Resizing the filesystem on /dev/md0 to 2928697600 (4k) blocks.
Begin pass 1 (max = 59616)
Extending the inode table     X---------------------------------------
And I don't know if this is what normally happens, but my raid is expanding at the same time as the resize2fs.

Code: Select all

[~] # cat /proc/mdstat 
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] 
md0 : active raid5 sde3[4] sdd3[3] sdc3[2] sdb3[1] sda3[0]
      11714790400 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
      [========>............]  resync = 40.7% (1194108124/2928697600) finish=551.7min speed=52399K/sec
      
md5 : active raid1 sde2[2](S) sdd2[0] sdc2[3](S) sdb2[4](S) sda2[1]
      530048 blocks [2/2] [UU]
      
md13 : active raid1 sde4[4] sdd4[3] sdc4[2] sdb4[1] sda4[0]
      458880 blocks [5/5] [UUUUU]
      bitmap: 0/57 pages [0KB], 4KB chunk

md9 : active raid1 sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
      530048 blocks [5/5] [UUUUU]
      bitmap: 0/65 pages [0KB], 4KB chunk

unused devices: <none>
User avatar
karlegas
Know my way around
Posts: 124
Joined: Thu Apr 26, 2012 2:38 pm
Contact:

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by karlegas »

musashi77 wrote:I gotta that I, extremely disaoppointed that I need to go to this level to extend my RAID and wish I never started cause I think Ive gone past the point of no return.

Anyway I am having troubles unmounting the RAID on TS-409.
When I put in the command I get this.
[~] # umount /dev/md0
umount: /dev/md0: not mounted


However when I go to check it I get the following
[~] # e2fsck -f /dev/md0
e2fsck 1.40-WIP (02-Oct-2006)
/dev/md0 is mounted.

WARNING!!! Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.

Do you really want to continue (y/n)?


I attempted to stop the services with the "services.sh stop" and "/etc/init.d/Qthttpd.sh stop"
However no luck still and when I run the following command to try and identify the problem I get the follow:
[~] # lsof +f -- /dev/md0
-sh: lsof: command not found


Any ideas?
I apply this suggestions below and also disable the HD Station add -ons

RAID expansion failure! (TS-409)
Postby Asmodision » Thu Oct 30, 2008 7:31 am

FIXED!

After almost a week I managed to resize the filesystem. It was tricky I might add.
First of all. Don't install QPKG :) it just make your life misserable if you are planning to upgrade your disks.

Quick how to!

1: Disable QPKG and all other QPKG package you might have installed
2: Remove every single QPKG package
3: Reboot
4: Expand RAID system..takes forever and will most likly fail :lol: if it doesn't you don't need to read more.
5: Reboot again
6: Unmount everything that is connected to the raid via SSH console.
7: Runs "e2fsck -f /dev/md0"
8: Resize using "resize2fs /dev/md0" If it complains that it can't do it you need to remove journal from then ext3 fs. Then resize and after that add the journal again.
9: Reboot

BTW You have to remove /share/MD0_DATA/optware catalog if you wan't to install QPKG again.
Favorite Apps: Deluge - Plex, Twonky and TVmobili streaming
Models: TVS-463 QTS 4.3.4 / TS-221 QTS 4.3.3
Website: http://www.karlegas.com Blog
Website: http://www.naseros.com Tutorials and News in Spanish for QNAP NAS
manu-31
Starting out
Posts: 19
Joined: Fri Dec 02, 2011 4:35 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by manu-31 »

Don't need to remove qpkg, ... Just shut all the processes down !
A simple and efficient way is to (temporarily) remove /etc/config/qpkg.conf
before a reboot (this file is used to run the qpkg services on startup, a strange
choice rather than a /etc/qpkg/init.d structure btw !). Then you can
umount /dev/md0
Do not forget to restore /etc/config/qpkg.conf then reboot to reactivate
qpkg services, ...
TS 212 - 2x2T RAID1 (QTS 4.3.3.1864)
TS 420 - 2x4T RAID1 + 2x4T RAID1 (QTS 4.3.3.1432)
TS 431 - 3x2T RAID5 + 1x8T (QTS 4.3.6.1907)
TS 509 Pro - 5x8T RAID (openmediavault 5.6.13)
hmp
First post
Posts: 1
Joined: Thu Nov 24, 2011 5:18 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by hmp »

Hello PK_TS509!

Thank you so much for your comprehensive description how to solve that problem!!! I was about to loose my trust in QNAP...

I had the same problem with a TS-639PRO running firmware 4.0.2, desperately trying to exchange the old 1.5TB hdds to 6 new 3TB WD units (WD AV-GP 3000GB). I followed the process under the GUI and exchanged the drives one by one (took 8-9 hrs to rebuild each time). But the final step - size expansion - always failed... :evil:

So I followed your path step by step - and it just worked out perfectly!! Shame on you QNAP that the online migration obviously still fails from the GUI after all that time that this issue is known...!

I just had to add two more steps:

1. Before being able to unmount the volume: Even after having disabled all services accesible from the GUI (and checking like 3 dozen times, no service showing active in the system service overview) I still got the error message "device is busy"...
This trick helped: Just execute " # /etc/init.d/services.sh stop " from the shell

2. The e2fsck always failed and I learned from this post http://forum.qnap.com/viewtopic.php?p=216117 that it was due to the swap file that was too small. The solution described by RottUlf worked out nicely.

Result: 13.746GB new space :D

Cheers
replicator
New here
Posts: 2
Joined: Sun Mar 15, 2009 8:27 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by replicator »

Hi All,
I'm a long time lurker here, hoping someone might have a suggestion with an array resizing problem on my TS-639 Pro.
Firmware is 4.0.5 (Jan 17 2014)

I had 3x 4TB WD reds (WD40EFRX) and 2x seagate 3TB Barracudas (ST3000DM01) in Raid 5, making a total of 10.7 TB.
The seagates have now been swapped out with 2 more 4TB reds (5 total), and I am trying to expand the array to 14.9 TB

Using the the Storage Manager in the GUI, I first disabled the bitmap, then began the expansion process.
It began at 50%, and failed at 75%.

I then tried it with the CLI, following the steps in this thread.

All the services were stopped using /etc/inet.d/services.sh stop

unmounted /dev/md0

turned off the journal

ran e2fsck -f /dev/md0, completed ok.

ran resize2fs -f /dev/md0, Pass 1 completes ok then gives the memory allocation error after it finishes extending the inode table.

[~] # e2fsck -f /dev/md0
e2fsck 1.41.4 (27-Jan-2009)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/md0: 1269101/732176384 files (1.8% non-contiguous), 2783648887/2928697600 blocks


[~] # resize2fs -p /dev/md0
resize2fs 1.41.4 (27-Jan-2009)
Resizing the filesystem on /dev/md0 to 3905449600 (4k) blocks.
Begin pass 1 (max = 29808)
Extending the inode table XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
resize2fs: Memory allocation failed while trying to resize /dev/md0

The Qnap has 2GB memory, and most of this, plus 38MB swap space appears to be used by resize2fs.

[~] # free
total used free shared buffers
Mem: 2070032 2025216 44816 0 568708
Swap: 530044 38844 491200
Total: 2600076 2064060 536016

I also created an additional 3GB swapfile on an external Hard drive, but receive the same error.
The size has previously been expanded from 5.5 TB to 10.7 TB without problem.

I'm wondering if I'm getting the error as a result of trying to expand an array that is already getting close to the 16GB limit ?
As a last resort, I will rebuild the array and restore from backups, but 10 TB of data takes a loooong time to restore at USB2 speeds :-)
replicator
New here
Posts: 2
Joined: Sun Mar 15, 2009 8:27 am

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by replicator »

I tried all sorts of things to fix the memory allocation problem - multiple USB sticks with extra swapfiles, disconnecting from the network and using the console only -I even put the original 1GB Dimm back, in case the extra memory was causing an issue during resizing.

I eventually solved it by downloading the latest e2fsprogs with ipkg:

# /opt/sbin/resize2fs -p /dev/md0
resize2fs 1.41.14 (22-Dec-2010)
Filesystem at /dev/md0 is mounted on /share/MD0_DATA; on-line resizing required
old desc_blocks = 699, new_desc_blocks = 932
Performing an on-line resize of /dev/md0 to 3905449600 (4k) blocks.
The filesystem on /dev/md0 is now 3905449600 blocks long.

An extra 4TB available, all good !
The new resize2fs didn't use any swapfile at all during the resizing process, although most of the RAM was used up.

I guess e2fsprogs is another thing Qnap should consider updating in a future release...
chrispm1968
Starting out
Posts: 13
Joined: Sun Aug 12, 2012 11:10 pm

Re: [HOWTO] Expanding RAID5 from the command line (TS 509)

Post by chrispm1968 »

Out of interest, how long did the expansion take?
Post Reply

Return to “System & Disk Volume Management”