Mount QNAP Drives to Linux

Questions about SNMP, Power, System, Logs, disk, & RAID.
PlanetMan
New here
Posts: 7
Joined: Sun Sep 16, 2018 1:08 pm

Re: Mount QNAP Drives to Linux

Post by PlanetMan » Wed Sep 19, 2018 9:32 pm

storageman wrote:It's a strange reason to move, no one should be worried about needing to read disks like this if they keep backups.


Then you keep right on using them. I'm not using a storage solution that locks me into the brand, full stop.

There's no feature of QNAP you can list me that's better than the ability to quickly hook up an array to literally any computer and have a temporary storage system up and running while you sort out a hardware failure.

Not to mention the support I got from QNAP was absolutely horrible, it took them a solid week to say "nope too bad".

User avatar
storageman
Ask me anything
Posts: 5511
Joined: Thu Sep 22, 2011 10:57 pm

Re: Mount QNAP Drives to Linux

Post by storageman » Thu Sep 20, 2018 3:52 pm

PlanetMan wrote:
storageman wrote:It's a strange reason to move, no one should be worried about needing to read disks like this if they keep backups.


Then you keep right on using them. I'm not using a storage solution that locks me into the brand, full stop.

There's no feature of QNAP you can list me that's better than the ability to quickly hook up an array to literally any computer and have a temporary storage system up and running while you sort out a hardware failure.

Not to mention the support I got from QNAP was absolutely horrible, it took them a solid week to say "nope too bad".


You mean like IBM and Netapp and EMC that allow you to read their proprietary arrays?
No one is worried about doing this like you, not even huge enterprise businesses.
This is why anyone sensible keep backups. But you keep on worrying about it. :wink:

PlanetMan
New here
Posts: 7
Joined: Sun Sep 16, 2018 1:08 pm

Re: Mount QNAP Drives to Linux

Post by PlanetMan » Thu Sep 20, 2018 6:25 pm

No one is worried about doing this like you, not even huge enterprise businesses.


Hahahahahahahahahahahahahahahahahahahahahaha.

OK buddy. You keep on thinking that.

User avatar
MrVideo
Experience counts
Posts: 4729
Joined: Fri May 03, 2013 2:26 pm

Re: Mount QNAP Drives to Linux

Post by MrVideo » Fri Sep 21, 2018 11:41 am

We sure as he[double hocky sticks] didn't care about doing anything like that where I worked and we were a Unix shop. Constant backups were being done.
QTS 4.1.n/4.2.n/4.3.n MANUAL
Submit QNAP Support Ticket - QNAP Tutorials, FAQs, Downloads, Wiki - Product Support Status - Moogle's QNAP FAQ help V2
When you ask a question, please include the following
(Thanks to Toxic17 for the links)
QNAP md_checker nasreport (release 20180525)
===============================
Model: TS-869L -- RAM: 3G -- FW: QTS 4.1.4 Build 20150522 (used for data storage)
WD60EFRX-68L0BN1(x1)/68MYMN1(x7) Red HDDs -- RAID6: 8x6TB -- Cold spare: 1x6TB
Entware
===============================
Model: TS-451A -- RAM: 2G -- FW: QTS 4.3.6 Build 20190531 (used as a video server)
WL3000GSA6472(x3) White label NAS HDDs -- RAID5: 3x3TB
Entware -- MyKodi 17.3 (default is Kodi 16)
===============================
My 2017 Total Solar Eclipse Photos

philippelt
New here
Posts: 5
Joined: Mon Jan 21, 2019 11:44 pm

Re: Mount QNAP Drives to Linux

Post by philippelt » Tue Jan 22, 2019 12:42 am

Hello,

considering the price range, device kind and target customers of QNAP, mostly home customers, I am not sure it could compare to enterprise datacenters with multiple sites, SAN, backup systems, 20G networks and so on...

Most consumers will put their data in the NAS expecting that they survive problems. I am quite sure that 99% of the customer base did not realize that they should buy a second NAS to backup the first to ensure that they will never loose data.

Unfortunately, the QNAP team made design choices that create complexity where it is not required. Complexity is the worst enemy of security and reliability.

I own a QNAP NAS (TVS 472-XT) with 4x6To data, only two drives are in RAID as others are backup/archives. A drive can fail, the system/backplane can fail but in 40 years of computer job I never faced a system were system AND drives all failed simultaneously. This has happen in one company I worked for 30 years ago when a water circuit failure create a "data lake" in the data center and all servers where under water but if such scenario happen for a consumer, then NAS recovery would not be the biggest issue they have to deal with.

In normal life, you loose 1 drive or your server and you are usually interested in recovering whatever is on surviving drives. In case of disaster, anything that can be done to recover data must not only be facilitated but even properly documented because when you need it, you will have no time to do research nor the mindset to do experiments to try to save things.

This is why such a procedure SHOULD be properly documented by QNAP to maximise data recovery in emergency conditions.

This is also, why, despite my NAS working without troubles, I did some experience to prepare myself for such situation.

I extracted a drive from my NAS using the Storage service and the option "Safely remove drive".
In fact, safely means that the NAS will remove ALL shared folders definition, ANY service activated using the drive (like NFS) and even CRASH (not shutdown) any KVM Virtual machine that you have that can be using the drive as NFS client.

It is a one way trip, once you will have remove the drive, you will NEVER be able to replug it and recover the content from your NAS.

As the drive are mounted in your NAS using a heavy stack of software 1) Soft MD Raid 2) LVM 3) DRBD even if you just request that the disk be used standalone with no bells and whistles, mounting it elsewhere will require some steps. Don't think too much about cache management with such pile of software or you will start to make nightmare about the reliability of any database working directly on the drives as there are chances that buffered I/O will make your database engine consider the transaction committed despite the journal not yet really been written on disk.

If you mount the removed drive on a regular Linux system, I used an Ubuntu 14.04, you can retrieve the content with the following procedure :
  • you will need mdadm (raid tools) and lvm2 (logical volume management), if you don't have them :

    Code: Select all

    $ apt-get install mdadm, lvm2
  • plug the drive in, locate the device created

    Code: Select all

    $ dmesg | tail
     ...
     [5443747.331855]  sdb: sdb1 sdb2 sdb3 sdb4 sdb5
    [5443747.332777] sd 1:0:0:0: [sdb] Attached SCSI disk
    ...
    
    You should find lines like the ones above that show that a new drive /dev/sdb has been registered
  • the device usually have 5 partitions, the one that is important to you is partition 3 -> /dev/sdb3
  • Check that the raid metadata is accessible

    Code: Select all

    $ mdadm -E /dev/sdb3
    /dev/sdb3:
              Magic : a92b4efc
            Version : 1.0
        Feature Map : 0x0
         Array UUID : aba5df7b:d6d628c0:551ab9e9:e7315cbc
               Name : 4
      Creation Time : Sat Jan  5 12:00:13 2019
         Raid Level : raid1
       Raid Devices : 1
    
     Avail Dev Size : 5840623240 (2785.03 GiB 2990.40 GB)
         Array Size : 2920311616 (2785.03 GiB 2990.40 GB)
      Used Dev Size : 5840623232 (2785.03 GiB 2990.40 GB)
       Super Offset : 5840623504 sectors
              State : clean
        Device UUID : edea2d79:79cef3ba:8d5454c8:d1a9013e
    
        Update Time : Mon Jan 21 15:27:39 2019
           Checksum : 92a0b9ec - correct
             Events : 4
    
    
       Device Role : Active device 0
       Array State : A ('A' == active, '.' == missing)
    
  • if the metadata was found, mount the raid array with the volume as single on an unused array, here /dev/md100

    Code: Select all

    $ mdadm -A -R /dev/md100 /dev/sdb3
    mdadm: /dev/md100 has been started with 1 drive.
    
  • Scan the drives for logical volumes : pvscan This should display a supported volume on your recently create array (/dev/md100)

    Code: Select all

    $ pvscan
      PV /dev/md100   VG vg290   lvm2 [2,72 TiB / 0    free]
      Total: 1 [2,72 TiB] / in use: 1 [2,72 TiB] / in no VG: 0 [0   ]
    
  • Check volume group and available volumes

    Code: Select all

    $ vgdisplay
      --- Volume group ---
      VG Name               vg290
      System ID             
      Format                lvm2
      Metadata Areas        1
      Metadata Sequence No  15
      VG Access             read/write
      VG Status             resizable
      MAX LV                0
      Cur LV                2
      Open LV               0
      Max PV                0
      Cur PV                1
      Act PV                1
      VG Size               2,72 TiB
      PE Size               4,00 MiB
      Total PE              712966
      Alloc PE / Size       712966 / 2,72 TiB
      Free  PE / Size       0 / 0   
      VG UUID               ntiXEb-nXWZ-2hY7-cQLX-yHDg-hqRi-mXHWAz
    $ lvdisplay
      --- Logical volume ---
      LV Path                /dev/vg290/lv547
      LV Name                lv547
      VG Name                vg290
      LV UUID                P0PakB-I7XU-OYzG-DQYw-nQNE-YJNO-oXYKuc
      LV Write Access        read/write
      LV Creation host, time NASBox, 2019-01-05 12:00:16 +0100
      LV Status              available
      # open                 0
      LV Size                27,90 GiB
      Current LE             7142
      Segments               2
      Allocation             inherit
      Read ahead sectors     8192
      Block device           252:0
       
      --- Logical volume ---
      LV Path                /dev/vg290/lv6
      LV Name                lv6
      VG Name                vg290
      LV UUID                r8FDzu-7qwf-eSFi-Vq1c-e6dm-8Vqf-UGviWj
      LV Write Access        read/write
      LV Creation host, time NASBox, 2019-01-05 12:00:21 +0100
      LV Status              available
      # open                 0
      LV Size                2,69 TiB
      Current LE             705824
      Segments               1
      Allocation             inherit
      Read ahead sectors     8192
      Block device           252:1
    
  • You are almost done, you just now have to mount the logical volume (here /dev/vg290/lv6 is the candidate due to the size)

    Code: Select all

    $ mount /mnt/anywhere /dev/vg290/lv6
    $ ls /mnt/anywhere
    
    You should retrieve, under /mnt/anywhere the data that were previously located on the QNAP physical drives in one or more shared folders.
Hope this may help anyone who is looking desperately on a way to recover data from a failed NAS. Any feedback would be appreciated particularly if you are facing units for which this procedure is not working.

I should add that all these operations can not be done as an external volume on a QNAP Nas. You will have to find a regular Linux system to do the job (you can just boot on a Linux USB Key and plug your drive using a dock station in a laptop).

And finally, as such procedure is not documented by QNAP and heavily rely on design choices of QNAP that can change any moment between firmware upgrades, I have no warranty that what was working TODAY will work the day I will need it. Not a consumer friendly attitude.

User avatar
dolbyman
Guru
Posts: 21157
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Mount QNAP Drives to Linux

Post by dolbyman » Tue Jan 22, 2019 12:51 am

thanks for the hints, but external USB backup would be so much more easy to mount anywhere you please (that why you should never backup to internal NAS disks)

philippelt
New here
Posts: 5
Joined: Mon Jan 21, 2019 11:44 pm

Re: Mount QNAP Drives to Linux

Post by philippelt » Tue Jan 22, 2019 1:43 am

You are right. Much more easy.

But when you have a 4 units NAS with, like me at home, 18To of effective storage (2x6 + 6 + 6), backing up on USB is just unrealistic. Chances that the day a disaster will happen you do not have backups of all datas are 100%.

In some professional situations, I used USB backups for non system users and for small remote location where network is not an option. But the key success factor for such situation is almost full automation.

Just tell me how I can, by program, eject an USB drive to give operators some basic rules such as every morning pick up the backup disk and put it in the safe and place the one in the safe in the dock. Currently, I did not found ANY way to just automatically EJECT an external drive once backup is done.

I am not even requesting an API to do such BASIC operation, a simple CLI utilities could to the job. If you know one, thanks in advance for your help.

It makes YEARS that we use full infrastructure automation in entreprises, among other reasons, for reliability precisely :-)
The greatest risk factor of production systems today are operators interventions, not hardware failures.

User avatar
dolbyman
Guru
Posts: 21157
Joined: Sat Feb 12, 2011 2:11 am
Location: Vancouver BC , Canada

Re: Mount QNAP Drives to Linux

Post by dolbyman » Tue Jan 22, 2019 1:52 am

I backup my 951X to a 419p+ and several external 8TB disks, works fine for me

if you need ejectable storage, QNAP has support for RDX
https://www.qnap.com/en/news/2018/qnap- ... isk-backup

philippelt
New here
Posts: 5
Joined: Mon Jan 21, 2019 11:44 pm

Re: Mount QNAP Drives to Linux

Post by philippelt » Tue Jan 22, 2019 2:04 am

Thanks for the info, I didn't had noticed.

I am not sure such setup are reasonable for a 'home setup' :-) but why not. I am a smaller player and I will continue to buy ordinary disk drives (for much less $/To) plugged on USB as my handling is probably more delicate than the one of an entreprise operator.

But I agree it makes sense for small business users as it is more efficient than LT0-x tapes that costs too much and requires SAS attachments.

I am glad to learn that you can 'eject' by program a RDX cartridge, I just hope that they will understand that it is the minimum that they should provide to USB users...

claykin
Getting the hang of things
Posts: 79
Joined: Mon May 26, 2008 6:42 am

Re: Mount QNAP Drives to Linux

Post by claykin » Tue Jan 22, 2019 2:24 am

Has anyone tried Reclaime? They claim to be able to read and extract Qnap RAID.

https://www.reclaime.com/library/qnap-recovery.aspx

While I'll agree with others who push for external backup plan, I cannot help but agree with those who are frustrated by QNAPs decision to use a custom RAID that cannot easily be mounted in a Linux distro. Synology uses LVM and their's is easily mounted in Ubuntu. even their btrfs volume.

Given the proprietary hardware and fact it often takes days/weeks to get repairs for non enterprise units, it would make sense for qnap to be as compatible as possible in emergency situations.
TS-453Pro

Thisisnotmyname
Easy as a breeze
Posts: 353
Joined: Mon Nov 19, 2018 1:21 am

Re: Mount QNAP Drives to Linux

Post by Thisisnotmyname » Tue Jan 22, 2019 2:59 am

philippelt wrote:
Tue Jan 22, 2019 2:04 am
Thanks for the info, I didn't had noticed.

I am not sure such setup are reasonable for a 'home setup' :-) but why not. I am a smaller player and I will continue to buy ordinary disk drives (for much less $/To) plugged on USB as my handling is probably more delicate than the one of an entreprise operator.

But I agree it makes sense for small business users as it is more efficient than LT0-x tapes that costs too much and requires SAS attachments.

I am glad to learn that you can 'eject' by program a RDX cartridge, I just hope that they will understand that it is the minimum that they should provide to USB users...

wait, we're talking about reasonable steps for a "home setup" but manually mounting a multi-device array in linux is reasonable?!?! I think you massively misjudge the skillset of "home setup" users. It's a small percentage that will have any linux distro setup to begin with and a smaller still set that will be comfortable working with mdadm and lvm. This isn't a massive hole that QNAP needs to close here.

philippelt
New here
Posts: 5
Joined: Mon Jan 21, 2019 11:44 pm

Re: Mount QNAP Drives to Linux

Post by philippelt » Tue Jan 22, 2019 3:14 am

As of my (short) exprience with QNAP, I haven’t found anything proprietary in their volumes setup.

They just use a lot of layers configured even when not required but once you dig in it, any Linux box can read the drive. I don’t see where there would be vendor lock-in.

Of course, if you use Raid5, I don’t know if it is a soft/hard/hybrid Raid 5 for Qnap but in such case, forget any direct drive recovery (with any tool) other than Qnap utilities.

Raid5 for home users in my opinion is just the shortest path to data disaster.

By the way, when you loose data, you will try anything to recover it !

it should not be impossible to find a linux friend somewhere to apply the procedure. As you can see, it does not require core dump analysis or partition offset computations. Not exactly advanced linux kernel knowledge required :-)

Thisisnotmyname
Easy as a breeze
Posts: 353
Joined: Mon Nov 19, 2018 1:21 am

Re: Mount QNAP Drives to Linux

Post by Thisisnotmyname » Tue Jan 22, 2019 3:39 am

philippelt wrote:
Tue Jan 22, 2019 3:14 am
By the way, when you loose data, you will try anything to recover it !

Awesome, you're in luck! https://www.datarecovery.net

philippelt
New here
Posts: 5
Joined: Mon Jan 21, 2019 11:44 pm

Re: Mount QNAP Drives to Linux

Post by philippelt » Tue Jan 22, 2019 4:17 am

Off course, you can always pay someone to solve your problem, each one can choose the path that best fit.

You could also use an Amazon/Google/Microsoft-Azure drive to store your data, you will no longer have to bother with backups.

Just a question of cost ... and requirements

The day you will have 18To of home data in the cloud, think twice before cancelling your storage plan to switch to an other provider :-)

martijnatlico
New here
Posts: 6
Joined: Thu May 26, 2016 4:17 pm

Re: Mount QNAP Drives to Linux

Post by martijnatlico » Thu Jan 31, 2019 2:37 am

Interesting how people stress backups. The lock-in issue makes them useless once you want to step out of the Qnap ecosystem. I have backups of my failed QNAP on Glacier but those are also in a proprietary format :roll: Seems the only reasonable way forward is buying another Qnap, which I'm not going to do after the disappointments I've had with recent Qnap gear (including a pretty expensive rackmount model). I'll see if I can recover the 4-drive RAID10 in Linux and report back here.

Post Reply

Return to “System & Disk Volume Management”