Big data projects could use advice.

[ver2ual]

I run a small film production company. For various reasons, I just pulled an editing project from an editor and picked up my TVS-672XT from them- first time I've seen this or any other QNAP. I had it drop-shipped (new) direct to the editor with 6 x 10TB Seagate IronWolf Pro Drives (new) and they set it up and since uploaded all project footage. Now that I have the QNAP I'm on a steep learning curve. Once I solved the Mac+sleep= no TB3 connection, I discovered in Control Panel a 45.4TB "Storage Pool 2", one 43.6TB "Data Volume 2", with 34.5TB "Used" and 1.3TB of "Free Space". Array is RAID 5. Despite having the QNAP since May 2021, the editors never upgraded the firmware- or anything else. I followed steps learned in this forum and manually updated firmware to 4.5.4.2012. QTS. I connect via TB3 to M1 Macbook Pro OS Monterey. No externals on QNAP. No SSD cache.

For starters, the editors advised the QNAP is safe in RAID 5 and no backup required. I learned in this forum that back-ups are critical. I immediately bought 2 x 20TB IronWolf Pro drives and backed-up the 34.5TB to a TB3 G-RAID drive in RAID 0 and stored it in a fireproof safe.The footage is a huge investment. I'm nervous with other footage waiting to be uploaded and other large data projects coming in. Questions: (opinions welcome)

1- Is the QNAP storage pool/volume/free space allocated correctly? Is a large single volume the best solution for mutli big-data projects? What's with the 1.3TB free space?

2- There are no Snapshots. I get an error saying there is not enough room in the volume and asks to make room. While I research what a Snapshot is, should I agree to make room on the volume and do a Snapshot? How much room does that take?

3- We have a 2nd project with big data coming up (+/-35TB). Should I remove the 6 x 10TB drives from the QNAP and put them in a safe, and use new drives for Project 2? Will the data be preserved on the 6 x1 0TB Project 1 drives if I put them back in the QNAP in same order? Not sure best practices here. Alternately, I have 5 x 20TB IronWolf Pro drives. Should I replace the current 6 x 10TB drives one by one with 20TB and just leave both projects on the same QNAP and backup new data as it comes? Is it possible to have 4/5 x 20TB drives replace the 6 x 10TB drives, or will I need all 6 x 20 TB drives to rebuild (then expand) the volume in RAID 5 using the "Replace Disk One by One" option?

4- I cannot download or connect to the internet through the QNAP control panel. I have 50+ notifications such as "Failed to Update Virus Definitions", etc. There appears to be zero apps (other than default) or malware installed. I read here to not connect the QNAP directly to the internet- so how do I install apps? Any critical app suggestions?

5- TB3 transfer speeds are +/-850mbs write, 1050mbs read. I cannot for the life of me figure out the TB3 bridge settings to connect 10gbe. Do I need a device for this? Would the speeds be faster with TB3 10gbe bridge? Any recommendations for improving speed for 4K+ video editing regarding future setups, volumes, drives, apps, speed, etc?

Apologies and thanks for perhaps basic knowledge questions.

[Bob Zelin]

replies below -


I run a small film production company.

REPLY - never searched for any info on this forum , or Reddit, or Creative Cow, or LiftGammaGain, or Google ?


For various reasons, I just pulled an editing project from an editor and picked up my TVS-672XT from them- first time I've seen this or any other QNAP. I had it drop-shipped (new) direct to the editor with 6 x 10TB Seagate IronWolf Pro Drives (new) and they set it up and since uploaded all project footage.

REPLY - well - how do you know if it's set up correctly for video editing ?


Now that I have the QNAP I'm on a steep learning curve. Once I solved the Mac+sleep= no TB3 connection,

REPLY - no sleep on the Mac, and no sleep on the QNAP, and no TB connection - you use 10G. Are you using a Mac ? You need a thunderbolt 3 to 10G adapter, or a Mac Studio (or Mac Mini) with a 10G port.



I discovered in Control Panel a 45.4TB "Storage Pool 2", one 43.6TB "Data Volume 2", with 34.5TB "Used" and 1.3TB of "Free Space". Array is RAID 5. Despite having the QNAP since May 2021, the editors never upgraded the firmware- or anything else. I followed steps learned in this forum and manually updated firmware to 4.5.4.2012. QTS. I connect via TB3 to M1 Macbook Pro OS Monterey. No externals on QNAP. No SSD cache.

REPLY - you should be running QTS 5.0.0, not 4.5.4. Stop using TB3 connection. And of course, no SSD cache.




For starters, the editors advised the QNAP is safe in RAID 5 and no backup required.

REPLY - the editor is an idiot. You always need a backup. RAID 5 is not backup. If one drive fails, you lose everything, if the rebuild is not quick enough, and a second drive fails.


I learned in this forum that back-ups are critical. I immediately bought 2 x 20TB IronWolf Pro drives and backed-up the 34.5TB to a TB3 G-RAID drive in RAID 0 and stored it in a fireproof safe.The footage is a huge investment. I'm nervous with other footage waiting to be uploaded and other large data projects coming in. Questions: (opinions welcome)

1- Is the QNAP storage pool/volume/free space allocated correctly? Is a large single volume the best solution for mutli big-data projects? What's with the 1.3TB free space?


REPLY - for video editing, you do a single static volume on a TVS-672XT or TVS-872XT. That is how you get your speeds. You use 10G ethernet, not thunderbolt 3.



2- There are no Snapshots. I get an error saying there is not enough room in the volume and asks to make room. While I research what a Snapshot is, should I agree to make room on the volume and do a Snapshot? How much room does that take?

REPLY - it's a long answer, but at this point, forget Snapshots.



3- We have a 2nd project with big data coming up (+/-35TB). Should I remove the 6 x 10TB drives from the QNAP and put them in a safe, and use new drives for Project 2? Will the data be preserved on the 6 x1 0TB Project 1 drives if I put them back in the QNAP in same order? Not sure best practices here. Alternately, I have 5 x 20TB IronWolf Pro drives. Should I replace the current 6 x 10TB drives one by one with 20TB and just leave both projects on the same QNAP and backup new data as it comes? Is it possible to have 4/5 x 20TB drives replace the 6 x 10TB drives, or will I need all 6 x 20 TB drives to rebuild (then expand) the volume in RAID 5 using the "Replace Disk One by One" option?

REPLY - OMG. This is so insane. You don't remove the drives. Yopu have a "Big Project" - buy another QNAP. Buy a TVS-h1288X this time. You are spending a LOT of money for 20 TB Ironwolf Pro drives (you could save a TON of money by buying Seagate EXOS 18 TB drives). You put in ALL the drives - not 5 drives - SIX drives.





4- I cannot download or connect to the internet through the QNAP control panel. I have 50+ notifications such as "Failed to Update Virus Definitions", etc. There appears to be zero apps (other than default) or malware installed. I read here to not connect the QNAP directly to the internet- so how do I install apps? Any critical app suggestions?


REPLY - you download or connect thru the internet with MyQnapCloud, CloudLink, or QVPN.


5- TB3 transfer speeds are +/-850mbs write, 1050mbs read. I cannot for the life of me figure out the TB3 bridge settings to connect 10gbe. Do I need a device for this? Would the speeds be faster with TB3 10gbe bridge? Any recommendations for improving speed for 4K+ video editing regarding future setups, volumes, drives, apps, speed, etc?

REPLY - you will never have reliable connections until you get a Thunderbolt 3 to 10G adapter for your MacBook Pro (good luck finding one these days from Sonnet, OWC or QNAP) - of course, Mac's can now be ordered with 10G ports. (thanks Apple) - they bought all the Marvell 10G chips.

bobzelin@icloud.com
(who is this idiot Bob Zelin ?)




Apologies and thanks for perhaps basic knowledge questions.

[dosborne]

REPLY - you should be running QTS 5.0.0, not 4.5.4.

That may be your opinion, but most (or at least, many) people still consider the 5.0.0 firmware stream to be Beta quality (at best). QNAP is still releasing security fixes for 4.5.4 and for a critical production environment, at least in my opinion, 4.5.4 is significantly more stable and just as secure. There are still significant issues with 5.0.0 that has caused many people to roll back to get around things that don't work.

[ver2ual]

replies below -

REPLY - never searched for any info on this forum , or Reddit, or Creative Cow, or LiftGammaGain, or Google ?

Thanks for your answers. I run a small company. I don't edit, I hired (past tense) editors. I don't manage data- I hired (past tense) someone to do that. I've been dumped a QNAP with 35TB of data on it. My questions on this forum are damage control to protect the footage until I can hire another post production team. My questions were also about scalability since I have Project #2 footage coming in already but have no post team in place for that either. (it was potentially the same as Project #1). I have 4 projects on the horizon. It somehow doesn't make sense to have 4 QNAPs. Seems stupid. Seems more logical to backup data and store QNAP archival footage/drives somewhere else in fireproof safe- but if that's not an option then it's not an option.

If it sounds like I don't know what I'm doing regarding managing footage on this scale, it's becasue I don't know what I'm doing. I have my expertise. I hire (preferably competent) people to be experts in their own field. Meanwhile, I've got to learn quickly about QNAP in order to store and protect my assets till I find other folks to do that.

Thanks

[ritters]

RAID is always a good idea, for example a RAID 6 with dual redundancy - but a RAID is never to be seen as making backups obsolete. You always need backups!

In your case with the importance of your data, you should have at least two independent backups, better three. One could be another RAID, one external discs and one in the cloud. It is important to have the backups physically independent, even in different houses, just in case. The media should not be connected to anything if not needed, so you are safe from malware, power surges and so on.

Having something like a paranoia thinking about data safety is a sign of just being on the right track.

[Bob Zelin]

if you have multiple projects, that require this much data, you don't buy 4 QNAP's - you buy one BIG QNAP. If you have 4 projects on the horizon, you need large storage to hold all of this data. This is what every successful production company deals with as they grow.
Instead of looking at a TVS-672XT, you should be looking at a QNAP Enterprise product like the TS-h1683XU-RP. With sixteen 18 TB Segate EXOS drives, you will have 252 TB of usable storage in the single QNAP system after RAID 6.

I build these systems for video professionals every day. That's all I do.
bobzelin@icloud.com

[ver2ual]

...you should be looking at a QNAP Enterprise product like the TS-h1683XU-RP. With sixteen 18 TB Segate EXOS drives, you will have 252 TB of usable storage in the single QNAP system after RAID 6."

Super helpful. Looking at that model now. We are also discovering that some models have flash media for direct editing- but perhaps the 10GBe will help with that- again not my turf or knowledge base. I have a Divinci project file loaded just to look at- and some files are stuttering- but I'm sure there are factors beyond transfer rate that can cause this...

I'm curious- once we decide on a more scalable QNAP, do we start from scratch or can drives be transferred from the 675 XT directly to the next? We are trying to find 10gbe/TB3 bridges but they are (as you mentioned) all backordered.

Also, checked out the EXOS 18TB vs IronWolf 20TB. Just FYI, I chose the IronWolf 20TB only becasue the first editors recommendation, and the IronWolf 10TB are what's populated in the 675 now.

I would consider paying for some remote consultation as we move forward. If interested or know someone let me know.

[ver2ual]

Having something like a paranoia thinking about data safety is a sign of just being on the right track.

Ha- good to know. When I received the QNAP I felt so insecure looking at this thing and knowing it was the one-and-only full copy of a year's worth of work. We have various backups here and there and I could probably rebuild most of it in pieces, if required- but still...

In the meantime, we now have at least one full back-up, housed in a different location, and are closely looking for a adequate back-up strategy.

Thanks for your input.

[Bob Zelin]

bobzelin@icloud.com