[HOWTO] TS-439 ext3->ext4 filesystem upgrade

Introduce yourself to us and other members here, or share your own product reviews, suggestions, and tips and tricks of using QNAP products.

[HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby adamvh » Wed Jul 08, 2009 8:50 am

Current firmware version: 3.1.0 Build 0627T

Stop services running. Eg. smb/apache Run:
Code: Select all
unmount /dev/md0


You need to use the tune2fs and fsck tools in the filesystem, and that filesystem needs to be unmounted. Run:
Code: Select all
tune2fs -O extents,uninit_bg,dir_index /dev/md0


After running this command you MUST run fsck. If you don't do it, Ext4 WILL NOT MOUNT your filesystem. This fsck run is needed to return the filesystem to a consistent state. It WILL tell you that it finds checksum errors in the group descriptors - it's expected, and it's exactly what it needs to be rebuilt to be able to mount it as Ext4, so don't get surprised by them. Since each time it finds one of those errors it asks you what to do, always say YES. If you don't want to be asked, add the "-p" parameter to the fsck command, it means "automatic repair":

Code: Select all
fsck -pDf /dev/md0


There's another thing that must be mentioned. All your existing files will continue using the old indirect mapping to map all the blocks of data. The online defrag tool will be able to migrate each one of those files to a extent format (using a ioctl that tells the filesystem to rewrite the file with the extent format; you can use it safely while you're using the filesystem normally)

Code: Select all
mount -t ext4 /dev/md0 /share/MD0_DATA
adamvh
New here
 
Posts: 9
Joined: Mon Mar 30, 2009 10:32 am
NAS Model: NMP-1000

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby petur » Wed Jul 08, 2009 3:39 pm

Thanks for this great post, I think I'll try this one of these days :)
User avatar
petur
Moderator
 
Posts: 4489
Joined: Sun Mar 30, 2008 5:42 pm
Location: Gent, Belgium
NAS Model: TS-x69 Pro

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby drfruit » Wed Jul 08, 2009 4:26 pm

Looks interesting, but maybe someone should add a little explanation of the type or level of users this is aimed at, and what they would need to know and use or have installed in order to even contemplate using this guide?
I am sure the Linux experts know what to do, but most users will have no clue, including me.

Luckily I have already formatted my 439 as EXT4, even though it took a lot of time to backup and restore all my files. ;-)
TS-109 Pro (SAMSUNG HD154UI 1,5TB EXT3)
TS-439 Pro (4x Hitachi 2TB RAID5)
Mede8er 1000X3D
User avatar
drfruit
Easy as a breeze
 
Posts: 375
Joined: Mon Sep 10, 2007 4:27 pm
NAS Model: TS-239 Pro

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby bobertuk » Wed Jul 08, 2009 7:06 pm

I notice that there is an EXT4 defrag tool '4defrag' available for Linux systems - it appears to be part of Linux from kernel 2.6.28 but QNAP kernel is only at 2.6.24. Is there such a defrag tool suitable/available that can be used with current QNAP NAS models?

Bob
1 x Touch - LMS 7.7.1 SQLite - SSOTS v4.14
1 x Radio
1 x Boom
Lavry DA-10 DAC
QNAP TS-509 3.5.2 build 1124T
Starfish Pre-amp : Based on NAIM
Heavily modified NAIM NAP 250 Power-amp
Behringer DEQ2496
Linn Isobarik DMS
bobertuk
Easy as a breeze
 
Posts: 273
Joined: Fri Mar 27, 2009 12:12 am
NAS Model: TS-439U-SP/-RP

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby niwa2 » Wed Jul 08, 2009 7:34 pm

it would be nice if someone who knows what he is doing could compile ext4defrag for our nas systems. this would not only be useful for converting the filesystem but also for keeping up the performance.
TS-639 Pro | Version 4.0.1 | 5x 3TB WDC WD30EFRX RAID5 + 1x 320GB WDC WD3200AAKS
LianLi EX-50 with 3x 2TB SAMSUNG HD203WI + 2x 2TB SAMSUNG HD204UI in RAID5 as Backup drive
LianLi EX-503 with 5x 2TB SAMSUNG HD204UI in RAID5 as Backup drive
TS-109 + TS-109 Pro II | Version 3.3.2 Build 0918
HDI Dune BD Prime 3.0, Mede8er MED600X3D, SqueezeBox Duet, Netgear GS608 v2 8 Port GBit Switch
niwa2
Experience counts
 
Posts: 1728
Joined: Sat Oct 13, 2007 2:21 am
Location: Germany
NAS Model: TS-509 Pro

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby adamvh » Wed Jul 15, 2009 3:14 pm

After a little more research running EXT4 on early kernels can be very unstable.
Depends if QNAP dev team applied specific kernel patches.

[/] # uname -a
Linux QNAP 2.6.24 #1 SMP Sat Jun 27 02:52:09 CST 2009 i686 unknown
[/] #


It should be noted that the stock 2.6.26 ext4 has problems with delayed allocation and with filesystems with non-extent based files. So until Debian starts shipping a 2.6.27 based kernel or a 2.6.26 kernel with at least the 2.6.26-ext4-7 patchset, you should mount ext4dev filesystems using -o nodelalloc and only use freshly created filesystems using "mke2fs -t ext4dev". (Without these fixes, if you try to use an ext3 filesystem which was converted using "tune2fs -E test_fs -o extents /dev/DEV", you will probably hit a kernel BUG the moment you try to delete or truncate an old non-extent based file.)
adamvh
New here
 
Posts: 9
Joined: Mon Mar 30, 2009 10:32 am
NAS Model: NMP-1000

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby edwardl » Sun Jul 26, 2009 8:59 am

Hello,

I use TS-239 with 2 harddisks formated as EXT3 and RAID 0. I am considering to convert it to EXT4 by using the method mentioned above. Would anyone used this method before please advise if there is any problem faced after conversion? Is EXT4 better than EXT3? Is EXT4 more stable under QNAP NAS?

Many thanks!!
Edward
edwardl
Getting the hang of things
 
Posts: 62
Joined: Fri Jun 06, 2008 11:51 pm
NAS Model: TS-239 Pro

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby Moogle Stiltzkin » Wed Jul 29, 2009 10:08 am

adamvh wrote:After a little more research running EXT4 on early kernels can be very unstable.
Depends if QNAP dev team applied specific kernel patches.

[/] # uname -a
Linux QNAP 2.6.24 #1 SMP Sat Jun 27 02:52:09 CST 2009 i686 unknown
[/] #


It should be noted that the stock 2.6.26 ext4 has problems with delayed allocation and with filesystems with non-extent based files. So until Debian starts shipping a 2.6.27 based kernel or a 2.6.26 kernel with at least the 2.6.26-ext4-7 patchset, you should mount ext4dev filesystems using -o nodelalloc and only use freshly created filesystems using "mke2fs -t ext4dev". (Without these fixes, if you try to use an ext3 filesystem which was converted using "tune2fs -E test_fs -o extents /dev/DEV", you will probably hit a kernel BUG the moment you try to delete or truncate an old non-extent based file.)


To be quite honest i didn't really quite get that except your saying EXT 4 may be buggy ?

But what if i did a full format, then formated as EXT4 ? would that work ?


Whats the benefits of EXT4 over EXT3 by the way ? :?
NAS
[Main Server] QNAP TS-659 Pro w. 4x 2TB Samsung EcoGreen F3 (HD203WI) EXT4 Raid5
[Backup] QNAP TS-509 Pro w. 5x 1TB Western Digital RE3 (WD1002FBYS) EXT4 Raid5

Other Devices
ESR-9850 Router | 10mbps download/upload FTTH Internet Connection | Windows 7 Ultimate Water Cooled PC with Intel i7 920 c0/c1 |


Desktop hard drives recommended for raid

How to setup Plex Media Server with QNAP
User avatar
Moogle Stiltzkin
Experience counts
 
Posts: 1518
Joined: Thu Dec 04, 2008 12:21 am
Location: Malaysia
NAS Model: TS-659 Pro II

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby GunRunner » Mon Sep 28, 2009 7:40 pm

Moogle Stiltzkin wrote:Whats the benefits of EXT4 over EXT3 by the way ? :?



Is there an answer to this question anywhere on this forum please? With regards to the QNAP NASes that have this EXT4 I mean? Would QNAP staff please respond?

Edited. I have just carried out a search of the forums, and I find there are more answers than I can shake a stick at. Many are contradictory. So if starting from scratch with a new machine (TS-439 Pro), does one start with EXT4 or EXT3? The reasoning behind your answer would most appreciated.

Thanks

Kind regards
Last edited by GunRunner on Tue Sep 29, 2009 1:51 am, edited 1 time in total.
GunRunner
Starting out
 
Posts: 22
Joined: Sun Aug 09, 2009 12:14 am
NAS Model: TS-239 Pro

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby niwa2 » Tue Sep 29, 2009 1:50 am

GunRunner wrote:
Moogle Stiltzkin wrote:Whats the benefits of EXT4 over EXT3 by the way ? :?



Is there an answer to this question anywhere on this forum please? With regards to the QNAP NASes that have this EXT4 I mean? Would QNAP staff please respond?

Thanks

Kind regards


yes this has been answered in this forum.

basicaly ext4 is a little faster than ext3, it supports bigger volumes than ext3 does and it is supposed to avoid fragmentation better than ext3.
that is all i an recall.

at the moment the only official way to get ext4 on a QNAP TS is to fully format your drives with ext4.
TS-639 Pro | Version 4.0.1 | 5x 3TB WDC WD30EFRX RAID5 + 1x 320GB WDC WD3200AAKS
LianLi EX-50 with 3x 2TB SAMSUNG HD203WI + 2x 2TB SAMSUNG HD204UI in RAID5 as Backup drive
LianLi EX-503 with 5x 2TB SAMSUNG HD204UI in RAID5 as Backup drive
TS-109 + TS-109 Pro II | Version 3.3.2 Build 0918
HDI Dune BD Prime 3.0, Mede8er MED600X3D, SqueezeBox Duet, Netgear GS608 v2 8 Port GBit Switch
niwa2
Experience counts
 
Posts: 1728
Joined: Sat Oct 13, 2007 2:21 am
Location: Germany
NAS Model: TS-509 Pro

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby GunRunner » Tue Sep 29, 2009 1:55 am

niwa2 wrote:
GunRunner wrote:
Moogle Stiltzkin wrote:Whats the benefits of EXT4 over EXT3 by the way ? :?



Is there an answer to this question anywhere on this forum please? With regards to the QNAP NASes that have this EXT4 I mean? Would QNAP staff please respond?

Thanks

Kind regards


yes this has been answered in this forum.

basicaly ext4 is a little faster than ext3, it supports bigger volumes than ext3 does and it is supposed to avoid fragmentation better than ext3.
that is all i an recall.

at the moment the only official way to get ext4 on a QNAP TS is to fully format your drives with ext4.


Thanks for the reply. As you can see I was editing my post when you answered, so you will see some text differences. As I have nothing on my TS-439 yet, then I will format it to EXT4 and start to use it.

Thanks again.
GunRunner
Starting out
 
Posts: 22
Joined: Sun Aug 09, 2009 12:14 am
NAS Model: TS-239 Pro

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby gmontem » Mon Oct 11, 2010 3:18 am

adamvh wrote:Current firmware version: 3.1.0 Build 0627T

Stop services running. Eg. smb/apache Run:
Code: Select all
unmount /dev/md0


You need to use the tune2fs and fsck tools in the filesystem, and that filesystem needs to be unmounted. Run:
Code: Select all
tune2fs -O extents,uninit_bg,dir_index /dev/md0


After running this command you MUST run fsck. If you don't do it, Ext4 WILL NOT MOUNT your filesystem. This fsck run is needed to return the filesystem to a consistent state. It WILL tell you that it finds checksum errors in the group descriptors - it's expected, and it's exactly what it needs to be rebuilt to be able to mount it as Ext4, so don't get surprised by them. Since each time it finds one of those errors it asks you what to do, always say YES. If you don't want to be asked, add the "-p" parameter to the fsck command, it means "automatic repair":

Code: Select all
fsck -pDf /dev/md0


There's another thing that must be mentioned. All your existing files will continue using the old indirect mapping to map all the blocks of data. The online defrag tool will be able to migrate each one of those files to a extent format (using a ioctl that tells the filesystem to rewrite the file with the extent format; you can use it safely while you're using the filesystem normally)

Code: Select all
mount -t ext4 /dev/md0 /share/MD0_DATA


I tried this last night with my ext3-formatted QNAP TS-509, firmware 3.3.3 Build 0928T. The conversion seemed to have gone well.

The call to fsck (e2fsck in my system) took a very long time (my 5 drive RAID 5 was nearly filled) so I left that operation overnight. When I got up the e2fsck was done. After remounting it, I rebooted the machine.

Code: Select all
# uname -a
Linux QNAP509PRO 2.6.33.2 #1 SMP Tue Sep 28 04:00:46 CST 2010 i686 unknown


After rebooting:

Code: Select all
# mount
/proc on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
sysfs on /sys type sysfs (rw)
tmpfs on /tmp type tmpfs (rw,size=32M)
none on /proc/bus/usb type usbfs (rw)
/dev/sda4 on /mnt/ext type ext3 (rw)
/dev/md9 on /mnt/HDA_ROOT type ext3 (rw)
/dev/md0 on /share/MD0_DATA type ext4 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,delalloc)
User avatar
gmontem
New here
 
Posts: 3
Joined: Thu Aug 14, 2008 8:11 pm
NAS Model: TS-439U-SP/-RP

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby lunzet » Tue Oct 26, 2010 7:12 am

Thanks for these Instructions which worked for me perfectly as well on my TS-419P with the latest firmware on.

In regards to the restructuring of the already existing data I have found the following: http://ck.kolivas.org/apps/defrag/
As I am not a programmer I cannot tell if that will really work "kind of safe" but as I have not been able to find any other solution for the file rewrite
I will test it.

Thanks
TS-459pro+ , 4 x 1,5TB SAMSUNG HD154UI 1AG0, Firmware: 3.4.4 build 0718T
lunzet
Starting out
 
Posts: 14
Joined: Tue Oct 26, 2010 4:34 am
NAS Model: TS-459 Pro+

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby RichieB » Thu Feb 03, 2011 2:29 am

I just upgraded my 4TB NAS (4x1.5TB in RAID 5) from ext3 to ext4 using the above procedure. It took 2.5 hours, but it went very well. Thanks.
RichieB
Getting the hang of things
 
Posts: 80
Joined: Fri Sep 18, 2009 5:48 am
NAS Model: TS-459 Pro+

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Postby jojje2 » Sun May 22, 2011 1:32 am

All your existing files will continue using the old indirect mapping to map all the blocks of data. The online defrag tool will be able to migrate each one of those files to a extent format


And here is a statically linked binary of e4defrag (since I've seen some people ask one)
MD5: 16b6f1a0698f943f5ae78952e4812320
SHA1: fa218d113f1e001867b1add0600d81d6f2af8528

I also followed the instructions in the initial post and had great success in migrating a 5TB raid5 based ext3 volume to ext4.
Thanks Adam, for the very useful guide !

PS. one thing made me a bit concerned / confused; namely that there was no mentioning of updating /etc/fstab or other configuration files so that the NAS would mount the volume correctly after the migration. A quick look at the QNAP volume management interface still reported the volume as ext3 after the volume was migrated to ext4. However after rebooting the NAS, the volume was correctly identified as ext4 "automagically" and mounted with all pertinent options. In other words, no need to touch /etc/fstab when following the procedure since the NAS is "clever" enough to see that the volume is ext4 and mount it appropriately upon reboot.

EDIT: after a bit of experimentation it seems the online defrag can't be run on ext3 -> ext4 migrated filesystems as it triggers an error complaining that the EXT4_IOC_MOVE_EXT ioctl operation is not supported. If an ext4 filesystem is created from scratch on the NAS however, the online defrag works just fine. So I'll have to buy some TB-disks for backupping the NAS it seems, and then wipe the NAS array :(
Was hoping to avoid this with the described recipe in this thread...
jojje2
New here
 
Posts: 3
Joined: Sat May 14, 2011 4:57 am
NAS Model: TS-239 Pro

Next

Return to Users' Corner

Who is online

Users browsing this forum: No registered users and 9 guests