[HOWTO] TS-439 ext3->ext4 filesystem upgrade

[adamvh]

Current firmware version: 3.1.0 Build 0627T

Stop services running. Eg. smb/apache Run:

Code: Select all

unmount /dev/md0

You need to use the tune2fs and fsck tools in the filesystem, and that filesystem needs to be unmounted. Run:

Code: Select all

tune2fs -O extents,uninit_bg,dir_index /dev/md0

After running this command you MUST run fsck. If you don't do it, Ext4 WILL NOT MOUNT your filesystem. This fsck run is needed to return the filesystem to a consistent state. It WILL tell you that it finds checksum errors in the group descriptors - it's expected, and it's exactly what it needs to be rebuilt to be able to mount it as Ext4, so don't get surprised by them. Since each time it finds one of those errors it asks you what to do, always say YES. If you don't want to be asked, add the "-p" parameter to the fsck command, it means "automatic repair":

Code: Select all

fsck -pDf /dev/md0

There's another thing that must be mentioned. All your existing files will continue using the old indirect mapping to map all the blocks of data. The online defrag tool will be able to migrate each one of those files to a extent format (using a ioctl that tells the filesystem to rewrite the file with the extent format; you can use it safely while you're using the filesystem normally)

Code: Select all

mount -t ext4 /dev/md0 /share/MD0_DATA

[petur]

Thanks for this great post, I think I'll try this one of these days

[drfruit]

Looks interesting, but maybe someone should add a little explanation of the type or level of users this is aimed at, and what they would need to know and use or have installed in order to even contemplate using this guide?
I am sure the Linux experts know what to do, but most users will have no clue, including me.

Luckily I have already formatted my 439 as EXT4, even though it took a lot of time to backup and restore all my files.

[bobertuk]

I notice that there is an EXT4 defrag tool '4defrag' available for Linux systems - it appears to be part of Linux from kernel 2.6.28 but QNAP kernel is only at 2.6.24. Is there such a defrag tool suitable/available that can be used with current QNAP NAS models?

Bob

[niwa2]

it would be nice if someone who knows what he is doing could compile ext4defrag for our nas systems. this would not only be useful for converting the filesystem but also for keeping up the performance.

[adamvh]

After a little more research running EXT4 on early kernels can be very unstable.
Depends if QNAP dev team applied specific kernel patches.

[/] # uname -a
Linux QNAP 2.6.24 #1 SMP Sat Jun 27 02:52:09 CST 2009 i686 unknown
[/] #

It should be noted that the stock 2.6.26 ext4 has problems with delayed allocation and with filesystems with non-extent based files. So until Debian starts shipping a 2.6.27 based kernel or a 2.6.26 kernel with at least the 2.6.26-ext4-7 patchset, you should mount ext4dev filesystems using -o nodelalloc and only use freshly created filesystems using "mke2fs -t ext4dev". (Without these fixes, if you try to use an ext3 filesystem which was converted using "tune2fs -E test_fs -o extents /dev/DEV", you will probably hit a kernel BUG the moment you try to delete or truncate an old non-extent based file.)

[edwardl]

Hello,

I use TS-239 with 2 harddisks formated as EXT3 and RAID 0. I am considering to convert it to EXT4 by using the method mentioned above. Would anyone used this method before please advise if there is any problem faced after conversion? Is EXT4 better than EXT3? Is EXT4 more stable under QNAP NAS?

Many thanks!!
Edward

[Moogle Stiltzkin]

After a little more research running EXT4 on early kernels can be very unstable.
Depends if QNAP dev team applied specific kernel patches.

[/] # uname -a
Linux QNAP 2.6.24 #1 SMP Sat Jun 27 02:52:09 CST 2009 i686 unknown
[/] #

It should be noted that the stock 2.6.26 ext4 has problems with delayed allocation and with filesystems with non-extent based files. So until Debian starts shipping a 2.6.27 based kernel or a 2.6.26 kernel with at least the 2.6.26-ext4-7 patchset, you should mount ext4dev filesystems using -o nodelalloc and only use freshly created filesystems using "mke2fs -t ext4dev". (Without these fixes, if you try to use an ext3 filesystem which was converted using "tune2fs -E test_fs -o extents /dev/DEV", you will probably hit a kernel BUG the moment you try to delete or truncate an old non-extent based file.)

To be quite honest i didn't really quite get that except your saying EXT 4 may be buggy ?

But what if i did a full format, then formated as EXT4 ? would that work ?


Whats the benefits of EXT4 over EXT3 by the way ?

[GunRunner]

Whats the benefits of EXT4 over EXT3 by the way ?

Is there an answer to this question anywhere on this forum please? With regards to the QNAP NASes that have this EXT4 I mean? Would QNAP staff please respond?

Edited. I have just carried out a search of the forums, and I find there are more answers than I can shake a stick at. Many are contradictory. So if starting from scratch with a new machine (TS-439 Pro), does one start with EXT4 or EXT3? The reasoning behind your answer would most appreciated.

Thanks

Kind regards

[niwa2]

Whats the benefits of EXT4 over EXT3 by the way ?

Is there an answer to this question anywhere on this forum please? With regards to the QNAP NASes that have this EXT4 I mean? Would QNAP staff please respond?

Thanks

Kind regards

yes this has been answered in this forum.

basicaly ext4 is a little faster than ext3, it supports bigger volumes than ext3 does and it is supposed to avoid fragmentation better than ext3.
that is all i an recall.

at the moment the only official way to get ext4 on a QNAP TS is to fully format your drives with ext4.

[GunRunner]

Whats the benefits of EXT4 over EXT3 by the way ?

Is there an answer to this question anywhere on this forum please? With regards to the QNAP NASes that have this EXT4 I mean? Would QNAP staff please respond?

Thanks

Kind regards

yes this has been answered in this forum.

basicaly ext4 is a little faster than ext3, it supports bigger volumes than ext3 does and it is supposed to avoid fragmentation better than ext3.
that is all i an recall.

at the moment the only official way to get ext4 on a QNAP TS is to fully format your drives with ext4.

Thanks for the reply. As you can see I was editing my post when you answered, so you will see some text differences. As I have nothing on my TS-439 yet, then I will format it to EXT4 and start to use it.

Thanks again.

[gmontem]

Current firmware version: 3.1.0 Build 0627T

Stop services running. Eg. smb/apache Run:

Code: Select all

unmount /dev/md0

You need to use the tune2fs and fsck tools in the filesystem, and that filesystem needs to be unmounted. Run:

Code: Select all

tune2fs -O extents,uninit_bg,dir_index /dev/md0

After running this command you MUST run fsck. If you don't do it, Ext4 WILL NOT MOUNT your filesystem. This fsck run is needed to return the filesystem to a consistent state. It WILL tell you that it finds checksum errors in the group descriptors - it's expected, and it's exactly what it needs to be rebuilt to be able to mount it as Ext4, so don't get surprised by them. Since each time it finds one of those errors it asks you what to do, always say YES. If you don't want to be asked, add the "-p" parameter to the fsck command, it means "automatic repair":

Code: Select all

fsck -pDf /dev/md0

There's another thing that must be mentioned. All your existing files will continue using the old indirect mapping to map all the blocks of data. The online defrag tool will be able to migrate each one of those files to a extent format (using a ioctl that tells the filesystem to rewrite the file with the extent format; you can use it safely while you're using the filesystem normally)

Code: Select all

mount -t ext4 /dev/md0 /share/MD0_DATA

I tried this last night with my ext3-formatted QNAP TS-509, firmware 3.3.3 Build 0928T. The conversion seemed to have gone well.

The call to fsck (e2fsck in my system) took a very long time (my 5 drive RAID 5 was nearly filled) so I left that operation overnight. When I got up the e2fsck was done. After remounting it, I rebooted the machine.

Code: Select all

# uname -a
Linux QNAP509PRO 2.6.33.2 #1 SMP Tue Sep 28 04:00:46 CST 2010 i686 unknown

After rebooting:

Code: Select all

# mount
/proc on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
sysfs on /sys type sysfs (rw)
tmpfs on /tmp type tmpfs (rw,size=32M)
none on /proc/bus/usb type usbfs (rw)
/dev/sda4 on /mnt/ext type ext3 (rw)
/dev/md9 on /mnt/HDA_ROOT type ext3 (rw)
/dev/md0 on /share/MD0_DATA type ext4 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,delalloc)

[lunzet]

Thanks for these Instructions which worked for me perfectly as well on my TS-419P with the latest firmware on.

In regards to the restructuring of the already existing data I have found the following: http://ck.kolivas.org/apps/defrag/
As I am not a programmer I cannot tell if that will really work "kind of safe" but as I have not been able to find any other solution for the file rewrite
I will test it.

Thanks

[RichieB]

I just upgraded my 4TB NAS (4x1.5TB in RAID 5) from ext3 to ext4 using the above procedure. It took 2.5 hours, but it went very well. Thanks.

[jojje2]

All your existing files will continue using the old indirect mapping to map all the blocks of data. The online defrag tool will be able to migrate each one of those files to a extent format

And here is a statically linked binary of e4defrag (since I've seen some people ask one)
MD5: 16b6f1a0698f943f5ae78952e4812320
SHA1: fa218d113f1e001867b1add0600d81d6f2af8528

I also followed the instructions in the initial post and had great success in migrating a 5TB raid5 based ext3 volume to ext4.
Thanks Adam, for the very useful guide !

PS. one thing made me a bit concerned / confused; namely that there was no mentioning of updating /etc/fstab or other configuration files so that the NAS would mount the volume correctly after the migration. A quick look at the QNAP volume management interface still reported the volume as ext3 after the volume was migrated to ext4. However after rebooting the NAS, the volume was correctly identified as ext4 "automagically" and mounted with all pertinent options. In other words, no need to touch /etc/fstab when following the procedure since the NAS is "clever" enough to see that the volume is ext4 and mount it appropriately upon reboot.

EDIT: after a bit of experimentation it seems the online defrag can't be run on ext3 -> ext4 migrated filesystems as it triggers an error complaining that the EXT4_IOC_MOVE_EXT ioctl operation is not supported. If an ext4 filesystem is created from scratch on the NAS however, the online defrag works just fine. So I'll have to buy some TB-disks for backupping the NAS it seems, and then wipe the NAS array
Was hoping to avoid this with the described recipe in this thread...