Migrate settings, QPKGs, NOT data to new disks on TS-451

Questions about SNMP, Power, System, Logs, disk, & RAID.
Post Reply
occamsrazor
Know my way around
Posts: 209
Joined: Tue Mar 30, 2010 8:30 pm

Migrate settings, QPKGs, NOT data to new disks on TS-451

Post by occamsrazor » Tue Aug 20, 2019 8:21 pm

Hi, I currently have a TS-451 with 4x4TB WD Red in Raid-5. I now have a new set of 4x10TB WD Red which I plan to swap in and remove the 4x4TB.

First question:

I'm wondering what is the best way to migrate all the settings, QPKGs, etc. BUT because all the main data is just backups of other machines it would seem cleaner to start afresh with a clean volume with no data. But I don't want to have to redo all the settings. I posted similar question on Reddit and was advised to do a disk-by-disk storage expansion but that doesn't seem a clean way to me.

So what exactly does System > Control Panel > Backup/Restore save, and what does it not?

Second question:

Regarding the disk pool/volume setup... I currently just have one big storage pool with one big thick volume of a similar size. Is there a better way to do this? Most of my data is:

- Time Machine backups from various Macs.
- Music and Movie backups cloned across network from Macs
- A few other archived files like software installers

...all of which are just organized in subfolders under the /Multimedia folder.

This has worked OK for me for a number of years, but am wondering if there is a better storage organisation strategy.... Thanks
TS-451 [4 x 10TB WD Reds in Raid-5]
TS-239 Pro II [2 x 3TB in Raid-0]
pfSense router and Ubiquiti Unifi switches
Mac Minis, MacBook Pro, iPhones

P3R
Guru
Posts: 10999
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Migrate settings, QPKGs, NOT data to new disks on TS-451

Post by P3R » Wed Aug 21, 2019 4:31 am

occamsrazor wrote:
Tue Aug 20, 2019 8:21 pm
I posted similar question on Reddit and was advised to do a disk-by-disk storage expansion...
That was a good answer.
So what exactly does System > Control Panel > Backup/Restore save, and what does it not?
It backup only the basic NAS configuration. It's completely unaware of any apps (QPKGs)!
Regarding the disk pool/volume setup... I currently just have one big storage pool with one big thick volume of a similar size. Is there a better way to do this?
In most cases one large volume holding everything is the best as it mean less administration and a minimum of storage overhead.

Thick volumes are my recommendation as they aren't limiting any features and at the same time are easy to understand. Very experienced administrators know themselves when they need thin volumes but if you need to ask, you don't need them (and you can convert later anyway).

There can be reasons to use multiple volumes in special cases but it's often a waste of time and resources on a Qnap. People with a background in administering other systems are often overdoing their Qnaps with multiple volumes because they're used to doing it that way but that doesn't mean it's a good idea.

What's important for the future is the inode setting but hopefully you have at least the default setting of 16384. That will allow future volume expansion up to 63.99 TB.

Also now that you expand your storage pool significantly, my advice is to not expand the volume to the maximum immediately. Expand the volume successively when needed with some TB at a time. That way you have the possibility to add more volumes, snapshots or anything else in the future should you one day feel the need.
This has worked OK for me for a number of years....
That tell me a single thick volume is very good for you. Don't try to fix something that isn't broken!
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

occamsrazor
Know my way around
Posts: 209
Joined: Tue Mar 30, 2010 8:30 pm

Re: Migrate settings, QPKGs, NOT data to new disks on TS-451

Post by occamsrazor » Wed Aug 21, 2019 5:15 am

P3R wrote:
Wed Aug 21, 2019 4:31 am
It backup only the basic NAS configuration. It's completely unaware of any apps (QPKGs)!
Hmm, that's not very smart really. I see why I was recommended the disk-by-disk expansion in this case. I don't know, it just seems an over-complicated way to do things and from what I've read is going to take a really long time for 4x4TB. Also does it result in as clean and reliable a volume as if I initialize it from scratch? I'd be willing to start from a restored basic settings if it's likely to be more reliable long-term.
P3R wrote:
Wed Aug 21, 2019 4:31 am
In most cases one large volume holding everything is the best as it mean less administration and a minimum of storage overhead.
Thick volumes are my recommendation as they aren't limiting any features and at the same time are easy to understand. Very experienced administrators know themselves when they need thin volumes but if you need to ask, you don't need them (and you can convert later anyway).
Thanks, that is very helpful...
P3R wrote:
Wed Aug 21, 2019 4:31 am
What's important for the future is the inode setting but hopefully you have at least the default setting of 16384. That will allow future volume expansion up to 63.99 TB.
Thanks. It's quite likely I will upgrade to a 6/8-bay in the near future so I do want to plan for future possibilities. Is that a setting of the pool or the volume, and where do I find this setting?
EDITED: OK I found it for the existing volume and it is set to 16K.
P3R wrote:
Wed Aug 21, 2019 4:31 am
Also now that you expand your storage pool significantly, my advice is to not expand the volume to the maximum immediately. Expand the volume successively when needed with some TB at a time. That way you have the possibility to add more volumes, snapshots or anything else in the future should you one day feel the need.
Ah I see... so when I replace the disks one by one it would expand the storage pool to fill the 30TB (4x10TB Raid5), but the existing thick volume would stay the same size for now? Then I can manually increase it as I need?
One thing I'd like to do is use snapshots which I don't at the moment because my volume is so full. But I find the way these work really utterly confusing in terms of how to set them up with guaranteed snapshot space etc... Any recommendations?

Thanks for your help... I have attached some screenshots showing current storage setup.
You do not have the required permissions to view the files attached to this post.
TS-451 [4 x 10TB WD Reds in Raid-5]
TS-239 Pro II [2 x 3TB in Raid-0]
pfSense router and Ubiquiti Unifi switches
Mac Minis, MacBook Pro, iPhones

P3R
Guru
Posts: 10999
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Migrate settings, QPKGs, NOT data to new disks on TS-451

Post by P3R » Wed Aug 21, 2019 6:15 am

occamsrazor wrote:
Wed Aug 21, 2019 5:15 am
Hmm, that's not very smart really. I see why I was recommended the disk-by-disk expansion in this case. I don't know, it just seems an over-complicated way to do things and from what I've read is going to take a really long time for 4x4TB.
Yes it'll probably take a week or so.

A reinstall, personal fine tune and restore of all data will probably in total take as long but it'll require much more hands on work and active time from you.
Also does it result in as clean and reliable a volume as if I initialize it from scratch?
Impossible to say for sure but most likely yes if you don't have problems with it today.
Thanks. It's quite likely I will upgrade to a 6/8-bay in the near future so I do want to plan for future possibilities.
It's recommended to switch to RAID 6 when wanting to add more disks than 4 and In my opinion, that would be a much better time to redo the configuration from scratch.
Ah I see... so when I replace the disks one by one it would expand the storage pool to fill the 30TB (4x10TB Raid5), but the existing thick volume would stay the same size for now? Then I can manually increase it as I need?
Correct.
One thing I'd like to do is use snapshots which I don't at the moment because my volume is so full. But I find the way these work really utterly confusing in terms of how to set them up with guaranteed snapshot space etc... Any recommendations?
No, I see very little value in local snapshots so I've only enabled it in a small scale for fun as I have some free space but I really don't have any experience or configuration recommendations to share.

I use backups on external systems to protect my data.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

occamsrazor
Know my way around
Posts: 209
Joined: Tue Mar 30, 2010 8:30 pm

Re: Migrate settings, QPKGs, NOT data to new disks on TS-451

Post by occamsrazor » Wed Aug 21, 2019 4:09 pm

P3R wrote:
Wed Aug 21, 2019 6:15 am
Yes it'll probably take a week or so.
A reinstall, personal fine tune and restore of all data will probably in total take as long but it'll require much more hands on work and active time from you.
Wow... I guess no big deal as long as it works OK. I need to have a closer look and see what exactly I'd need to restore manually if I did a config backup/restore. Assuming it backs up all the settings in the control panel... then it probably wouldn't be too big a deal.
P3R wrote:
Wed Aug 21, 2019 6:15 am
It's recommended to switch to RAID 6 when wanting to add more disks than 4 and In my opinion, that would be a much better time to redo the configuration from scratch.
It's likely to be 4-drive Raid5 for quite a while, as the 4x10tb was a significant expense for me, so I'll stick with that for now.
Regarding the inode setting of 16k... if I did do a clean install would you advise to increase that? It seems unlikely I'd go higher than a 60tb storage pool in the future, but am not sure if there are other advantages/disadvantages.
P3R wrote:
Wed Aug 21, 2019 6:15 am
No, I see very little value in local snapshots so I've only enabled it in a small scale for fun as I have some free space but I really don't have any experience or configuration recommendations to share.
I played with it briefly but then stopped as I couldn't really spare the space. That said, I recently had the (direct-attached) drive with all my movies on my Mac die. No problem I thought, I had a cloned backup on my QNAP so ordered another drive for the Mac intending to copy it all back. The night before the new drive was delivered I somehow (!!) managed to manually delete the backup folder on the QNAP... resulting in permanent loss of all my data. In this scenario if I'd had local snapshots enabled I could have simply restored that whole deleted folder.... You've always got to plan for user stupidity :-)
TS-451 [4 x 10TB WD Reds in Raid-5]
TS-239 Pro II [2 x 3TB in Raid-0]
pfSense router and Ubiquiti Unifi switches
Mac Minis, MacBook Pro, iPhones

occamsrazor
Know my way around
Posts: 209
Joined: Tue Mar 30, 2010 8:30 pm

Re: Migrate settings, QPKGs, NOT data to new disks on TS-451

Post by occamsrazor » Thu Aug 22, 2019 1:15 am

Thanks for all the advice... I've taken onboard everything said and reviewed what apps I'd really need to re-install and configure and it's not actually so much. I can definitely see the arguments for a disk-by-disk expansion... but am also thinking I will use this as an opportunity to maybe set things up a bit better than I did before.
So planning to do a full reinitialization and manually set everything up from scratch, having taken a config backup and screenshotted every settings screen.
Will do one storage pool of the whole 4-disk Raid-5 with inode 16384, then create a thick volume in that.
One question... if I really mess things up for some reason, I should be able to just put back the original 4-disk set and it should boot exactly the same as before I pulled them... correct?
TS-451 [4 x 10TB WD Reds in Raid-5]
TS-239 Pro II [2 x 3TB in Raid-0]
pfSense router and Ubiquiti Unifi switches
Mac Minis, MacBook Pro, iPhones

P3R
Guru
Posts: 10999
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Migrate settings, QPKGs, NOT data to new disks on TS-451

Post by P3R » Thu Aug 22, 2019 3:32 am

occamsrazor wrote:
Thu Aug 22, 2019 1:15 am
So planning to do a full reinitialization and manually set everything up from scratch, having taken a config backup and screenshotted every settings screen.
Excellent! You don't follow my recommendation strictly but I like it even better that you've thought it over well and is taking an informed decision. Doing it like that will also give you more exposure to the setup and configuration of the system, which is always a useful experience for the future.
Will do one storage pool of the whole 4-disk Raid-5 with inode 16384, then create a thick volume in that.
The inode setting is on the volume (it affects the file system) not the pool. :wink:
One question... if I really mess things up for some reason, I should be able to just put back the original 4-disk set and it should boot exactly the same as before I pulled them... correct?
I don't think you need to worry but if you've done a normal controlled shutdown and then replaced all disks, you should be able to put them all back in and return to the current configuration.

Good luck!
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

occamsrazor
Know my way around
Posts: 209
Joined: Tue Mar 30, 2010 8:30 pm

Re: Migrate settings, QPKGs, NOT data to new disks on TS-451

Post by occamsrazor » Thu Aug 22, 2019 4:09 pm

Thanks... I've now installed all the new drives and am setting up QTS.
One odd thing - the first 2 bays are reporting "current" SATA speed as 3gbps and "maximum" as 6gbps, whereas the second 2 bays are reporting 6gbps for each. I read a thread or two:
viewtopic.php?t=117467
viewtopic.php?t=127890
...that suggested this could be fixed by swapping the drives around and back again... will try that later.
Currently am doing a bad blocks scan on all the drives before I initialize them into a Raid-5 storage pool, going to take quite a while... does this order of doing that make sense, or better to scan for bad blocks after creating the pool?
Assuming all goes to plan I will reserve 20% of the pool for guaranteed snapshot space and create a thick volume filling all the rest.
TS-451 [4 x 10TB WD Reds in Raid-5]
TS-239 Pro II [2 x 3TB in Raid-0]
pfSense router and Ubiquiti Unifi switches
Mac Minis, MacBook Pro, iPhones

P3R
Guru
Posts: 10999
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: Migrate settings, QPKGs, NOT data to new disks on TS-451

Post by P3R » Thu Aug 22, 2019 4:27 pm

occamsrazor wrote:
Thu Aug 22, 2019 4:09 pm
Currently am doing a bad blocks scan on all the drives before I initialize them into a Raid-5 storage pool, going to take quite a while... does this order of doing that make sense, or better to scan for bad blocks after creating the pool?
Personally I would probably do it the other way around but neither approach is wrong and the way you do it is more logical.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!

occamsrazor
Know my way around
Posts: 209
Joined: Tue Mar 30, 2010 8:30 pm

Re: Migrate settings, QPKGs, NOT data to new disks on TS-451

Post by occamsrazor » Wed Aug 28, 2019 6:47 pm

Hi, Just wanted to say thanks again for all the help.... I got it all set up, one large storage pool, one thick volume (32kb bytes-per-inode in case I were to go above 64TB in future), 20% guaranteed storage space, etc. All good. Been cloning my files onto it for a couple of days now....

The only issue was I experienced was same as this: viewtopic.php?f=25&t=133717&p=622407&hilit=3gbps+6gbps

Opened a ticket and they basically said it wasn't anything to worry about. Not so sure I'm confident with that, but as my unit is out of warranty anyway and I haven't seen any issues I will live with it.
TS-451 [4 x 10TB WD Reds in Raid-5]
TS-239 Pro II [2 x 3TB in Raid-0]
pfSense router and Ubiquiti Unifi switches
Mac Minis, MacBook Pro, iPhones

Post Reply

Return to “System & Disk Volume Management”