[HOWTO] TS-439 ext3->ext4 filesystem upgrade

Introduce yourself to us and other members here, or share your own product reviews, suggestions, and tips and tricks of using QNAP products.
adamvh
New here
Posts: 9
Joined: Mon Mar 30, 2009 10:32 am

[HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by adamvh »

Current firmware version: 3.1.0 Build 0627T

Stop services running. Eg. smb/apache Run:

Code: Select all

unmount /dev/md0
You need to use the tune2fs and fsck tools in the filesystem, and that filesystem needs to be unmounted. Run:

Code: Select all

tune2fs -O extents,uninit_bg,dir_index /dev/md0
After running this command you MUST run fsck. If you don't do it, Ext4 WILL NOT MOUNT your filesystem. This fsck run is needed to return the filesystem to a consistent state. It WILL tell you that it finds checksum errors in the group descriptors - it's expected, and it's exactly what it needs to be rebuilt to be able to mount it as Ext4, so don't get surprised by them. Since each time it finds one of those errors it asks you what to do, always say YES. If you don't want to be asked, add the "-p" parameter to the fsck command, it means "automatic repair":

Code: Select all

fsck -pDf /dev/md0
There's another thing that must be mentioned. All your existing files will continue using the old indirect mapping to map all the blocks of data. The online defrag tool will be able to migrate each one of those files to a extent format (using a ioctl that tells the filesystem to rewrite the file with the extent format; you can use it safely while you're using the filesystem normally)

Code: Select all

mount -t ext4 /dev/md0 /share/MD0_DATA
User avatar
petur
Moderator
Posts: 4606
Joined: Sun Mar 30, 2008 5:42 pm
Location: Gent, Belgium
Contact:

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by petur »

Thanks for this great post, I think I'll try this one of these days :)
Praat je liever over QNAP in het Nederlands?
Liever een community bij jou in de buurt?

Kom naar QNAPclub België/Nederland
drfruit
Easy as a breeze
Posts: 420
Joined: Mon Sep 10, 2007 4:27 pm

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by drfruit »

Looks interesting, but maybe someone should add a little explanation of the type or level of users this is aimed at, and what they would need to know and use or have installed in order to even contemplate using this guide?
I am sure the Linux experts know what to do, but most users will have no clue, including me.

Luckily I have already formatted my 439 as EXT4, even though it took a lot of time to backup and restore all my files. ;-)
bobertuk
Easy as a breeze
Posts: 273
Joined: Fri Mar 27, 2009 12:12 am

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by bobertuk »

I notice that there is an EXT4 defrag tool '4defrag' available for Linux systems - it appears to be part of Linux from kernel 2.6.28 but QNAP kernel is only at 2.6.24. Is there such a defrag tool suitable/available that can be used with current QNAP NAS models?

Bob
1 x Touch - LMS 7.7.1 SQLite - SSOTS v4.14
1 x Radio
1 x Boom
Lavry DA-10 DAC
QNAP TS-509 3.5.2 build 1124T
Starfish Pre-amp : Based on NAIM
Heavily modified NAIM NAP 250 Power-amp
Behringer DEQ2496
Linn Isobarik DMS
niwa2
Experience counts
Posts: 1786
Joined: Sat Oct 13, 2007 2:21 am
Location: Germany

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by niwa2 »

it would be nice if someone who knows what he is doing could compile ext4defrag for our nas systems. this would not only be useful for converting the filesystem but also for keeping up the performance.
TVS-1282 | Version 4.4.3 | 3x 12TB WDC WD121KRYZ RAID5
TS-639 Pro | Version 4.2.6 | 5x 3TB WDC WD30EFRX RAID5 as Backup
LianLi EX-50 with 3x 2TB SAMSUNG HD203WI + 2x 2TB SAMSUNG HD204UI in RAID5 as Backup drive
LianLi EX-503 with 5x 2TB SAMSUNG HD204UI in RAID5 as Backup drive
adamvh
New here
Posts: 9
Joined: Mon Mar 30, 2009 10:32 am

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by adamvh »

After a little more research running EXT4 on early kernels can be very unstable.
Depends if QNAP dev team applied specific kernel patches.
[/] # uname -a
Linux QNAP 2.6.24 #1 SMP Sat Jun 27 02:52:09 CST 2009 i686 unknown
[/] #
It should be noted that the stock 2.6.26 ext4 has problems with delayed allocation and with filesystems with non-extent based files. So until Debian starts shipping a 2.6.27 based kernel or a 2.6.26 kernel with at least the 2.6.26-ext4-7 patchset, you should mount ext4dev filesystems using -o nodelalloc and only use freshly created filesystems using "mke2fs -t ext4dev". (Without these fixes, if you try to use an ext3 filesystem which was converted using "tune2fs -E test_fs -o extents /dev/DEV", you will probably hit a kernel BUG the moment you try to delete or truncate an old non-extent based file.)
edwardl
Getting the hang of things
Posts: 63
Joined: Fri Jun 06, 2008 11:51 pm

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by edwardl »

Hello,

I use TS-239 with 2 harddisks formated as EXT3 and RAID 0. I am considering to convert it to EXT4 by using the method mentioned above. Would anyone used this method before please advise if there is any problem faced after conversion? Is EXT4 better than EXT3? Is EXT4 more stable under QNAP NAS?

Many thanks!!
Edward
User avatar
Moogle Stiltzkin
Guru
Posts: 11448
Joined: Thu Dec 04, 2008 12:21 am
Location: Around the world....
Contact:

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by Moogle Stiltzkin »

adamvh wrote:After a little more research running EXT4 on early kernels can be very unstable.
Depends if QNAP dev team applied specific kernel patches.
[/] # uname -a
Linux QNAP 2.6.24 #1 SMP Sat Jun 27 02:52:09 CST 2009 i686 unknown
[/] #
It should be noted that the stock 2.6.26 ext4 has problems with delayed allocation and with filesystems with non-extent based files. So until Debian starts shipping a 2.6.27 based kernel or a 2.6.26 kernel with at least the 2.6.26-ext4-7 patchset, you should mount ext4dev filesystems using -o nodelalloc and only use freshly created filesystems using "mke2fs -t ext4dev". (Without these fixes, if you try to use an ext3 filesystem which was converted using "tune2fs -E test_fs -o extents /dev/DEV", you will probably hit a kernel BUG the moment you try to delete or truncate an old non-extent based file.)
To be quite honest i didn't really quite get that except your saying EXT 4 may be buggy ?

But what if i did a full format, then formated as EXT4 ? would that work ?


Whats the benefits of EXT4 over EXT3 by the way ? :?
NAS
[Main Server] QNAP TS-877 (QTS) w. 4tb [ 3x HGST Deskstar NAS & 1x WD RED NAS ] EXT4 Raid5 & 2 x m.2 SATA Samsung 850 Evo raid1 +16gb ddr4 Crucial+ QWA-AC2600 wireless+QXP PCIE
[Backup] QNAP TS-653A (Truenas Core) w. 4x 2TB Samsung F3 (HD203WI) RaidZ1 ZFS + 8gb ddr3 Crucial
[^] QNAP TL-D400S 2x 4TB WD Red Nas (WD40EFRX) 2x 4TB Seagate Ironwolf, Raid5
[^] QNAP TS-509 Pro w. 4x 1TB WD RE3 (WD1002FBYS) EXT4 Raid5
[^] QNAP TS-253D (Truenas Scale)
[Mobile NAS] TBS-453DX w. 2x Crucial MX500 500gb EXT4 raid1

Network
Qotom Pfsense|100mbps FTTH | Win11, Ryzen 5600X Desktop (1x2tb Crucial P50 Plus M.2 SSD, 1x 8tb seagate Ironwolf,1x 4tb HGST Ultrastar 7K4000)


Resources
[Review] Moogle's QNAP experience
[Review] Moogle's TS-877 review
https://www.patreon.com/mooglestiltzkin
GunRunner
Starting out
Posts: 22
Joined: Sun Aug 09, 2009 12:14 am

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by GunRunner »

Moogle Stiltzkin wrote: Whats the benefits of EXT4 over EXT3 by the way ? :?

Is there an answer to this question anywhere on this forum please? With regards to the QNAP NASes that have this EXT4 I mean? Would QNAP staff please respond?

Edited. I have just carried out a search of the forums, and I find there are more answers than I can shake a stick at. Many are contradictory. So if starting from scratch with a new machine (TS-439 Pro), does one start with EXT4 or EXT3? The reasoning behind your answer would most appreciated.

Thanks

Kind regards
Last edited by GunRunner on Tue Sep 29, 2009 1:51 am, edited 1 time in total.
niwa2
Experience counts
Posts: 1786
Joined: Sat Oct 13, 2007 2:21 am
Location: Germany

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by niwa2 »

GunRunner wrote:
Moogle Stiltzkin wrote: Whats the benefits of EXT4 over EXT3 by the way ? :?

Is there an answer to this question anywhere on this forum please? With regards to the QNAP NASes that have this EXT4 I mean? Would QNAP staff please respond?

Thanks

Kind regards
yes this has been answered in this forum.

basicaly ext4 is a little faster than ext3, it supports bigger volumes than ext3 does and it is supposed to avoid fragmentation better than ext3.
that is all i an recall.

at the moment the only official way to get ext4 on a QNAP TS is to fully format your drives with ext4.
TVS-1282 | Version 4.4.3 | 3x 12TB WDC WD121KRYZ RAID5
TS-639 Pro | Version 4.2.6 | 5x 3TB WDC WD30EFRX RAID5 as Backup
LianLi EX-50 with 3x 2TB SAMSUNG HD203WI + 2x 2TB SAMSUNG HD204UI in RAID5 as Backup drive
LianLi EX-503 with 5x 2TB SAMSUNG HD204UI in RAID5 as Backup drive
GunRunner
Starting out
Posts: 22
Joined: Sun Aug 09, 2009 12:14 am

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by GunRunner »

niwa2 wrote:
GunRunner wrote:
Moogle Stiltzkin wrote: Whats the benefits of EXT4 over EXT3 by the way ? :?

Is there an answer to this question anywhere on this forum please? With regards to the QNAP NASes that have this EXT4 I mean? Would QNAP staff please respond?

Thanks

Kind regards
yes this has been answered in this forum.

basicaly ext4 is a little faster than ext3, it supports bigger volumes than ext3 does and it is supposed to avoid fragmentation better than ext3.
that is all i an recall.

at the moment the only official way to get ext4 on a QNAP TS is to fully format your drives with ext4.
Thanks for the reply. As you can see I was editing my post when you answered, so you will see some text differences. As I have nothing on my TS-439 yet, then I will format it to EXT4 and start to use it.

Thanks again.
User avatar
gmontem
New here
Posts: 7
Joined: Thu Aug 14, 2008 8:11 pm

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by gmontem »

adamvh wrote:Current firmware version: 3.1.0 Build 0627T

Stop services running. Eg. smb/apache Run:

Code: Select all

unmount /dev/md0
You need to use the tune2fs and fsck tools in the filesystem, and that filesystem needs to be unmounted. Run:

Code: Select all

tune2fs -O extents,uninit_bg,dir_index /dev/md0
After running this command you MUST run fsck. If you don't do it, Ext4 WILL NOT MOUNT your filesystem. This fsck run is needed to return the filesystem to a consistent state. It WILL tell you that it finds checksum errors in the group descriptors - it's expected, and it's exactly what it needs to be rebuilt to be able to mount it as Ext4, so don't get surprised by them. Since each time it finds one of those errors it asks you what to do, always say YES. If you don't want to be asked, add the "-p" parameter to the fsck command, it means "automatic repair":

Code: Select all

fsck -pDf /dev/md0
There's another thing that must be mentioned. All your existing files will continue using the old indirect mapping to map all the blocks of data. The online defrag tool will be able to migrate each one of those files to a extent format (using a ioctl that tells the filesystem to rewrite the file with the extent format; you can use it safely while you're using the filesystem normally)

Code: Select all

mount -t ext4 /dev/md0 /share/MD0_DATA
I tried this last night with my ext3-formatted QNAP TS-509, firmware 3.3.3 Build 0928T. The conversion seemed to have gone well.

The call to fsck (e2fsck in my system) took a very long time (my 5 drive RAID 5 was nearly filled) so I left that operation overnight. When I got up the e2fsck was done. After remounting it, I rebooted the machine.

Code: Select all

# uname -a
Linux QNAP509PRO 2.6.33.2 #1 SMP Tue Sep 28 04:00:46 CST 2010 i686 unknown
After rebooting:

Code: Select all

# mount
/proc on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
sysfs on /sys type sysfs (rw)
tmpfs on /tmp type tmpfs (rw,size=32M)
none on /proc/bus/usb type usbfs (rw)
/dev/sda4 on /mnt/ext type ext3 (rw)
/dev/md9 on /mnt/HDA_ROOT type ext3 (rw)
/dev/md0 on /share/MD0_DATA type ext4 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,delalloc)
lunzet
Starting out
Posts: 22
Joined: Tue Oct 26, 2010 4:34 am

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by lunzet »

Thanks for these Instructions which worked for me perfectly as well on my TS-419P with the latest firmware on.

In regards to the restructuring of the already existing data I have found the following: http://ck.kolivas.org/apps/defrag/
As I am not a programmer I cannot tell if that will really work "kind of safe" but as I have not been able to find any other solution for the file rewrite
I will test it.

Thanks
RichieB
Getting the hang of things
Posts: 87
Joined: Fri Sep 18, 2009 5:48 am

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by RichieB »

I just upgraded my 4TB NAS (4x1.5TB in RAID 5) from ext3 to ext4 using the above procedure. It took 2.5 hours, but it went very well. Thanks.
jojje2
New here
Posts: 4
Joined: Sat May 14, 2011 4:57 am

Re: [HOWTO] TS-439 ext3->ext4 filesystem upgrade

Post by jojje2 »

All your existing files will continue using the old indirect mapping to map all the blocks of data. The online defrag tool will be able to migrate each one of those files to a extent format
And here is a statically linked binary of e4defrag (since I've seen some people ask one)
MD5: 16b6f1a0698f943f5ae78952e4812320
SHA1: fa218d113f1e001867b1add0600d81d6f2af8528

I also followed the instructions in the initial post and had great success in migrating a 5TB raid5 based ext3 volume to ext4.
Thanks Adam, for the very useful guide !

PS. one thing made me a bit concerned / confused; namely that there was no mentioning of updating /etc/fstab or other configuration files so that the NAS would mount the volume correctly after the migration. A quick look at the QNAP volume management interface still reported the volume as ext3 after the volume was migrated to ext4. However after rebooting the NAS, the volume was correctly identified as ext4 "automagically" and mounted with all pertinent options. In other words, no need to touch /etc/fstab when following the procedure since the NAS is "clever" enough to see that the volume is ext4 and mount it appropriately upon reboot.

EDIT: after a bit of experimentation it seems the online defrag can't be run on ext3 -> ext4 migrated filesystems as it triggers an error complaining that the EXT4_IOC_MOVE_EXT ioctl operation is not supported. If an ext4 filesystem is created from scratch on the NAS however, the online defrag works just fine. So I'll have to buy some TB-disks for backupping the NAS it seems, and then wipe the NAS array :(
Was hoping to avoid this with the described recipe in this thread...
Post Reply

Return to “Users' Corner”