[SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-500P)

Introduce yourself to us and other members here, or share your own product reviews, suggestions, and tips and tricks of using QNAP products.
louiscar
Easy as a breeze
Posts: 265
Joined: Mon Aug 10, 2015 4:32 am

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-50

Post by louiscar »

alyf80 wrote:
JeffryPaul wrote:after installing the "thin-provisioning-tools" package, I was able to step past the thin-pool problem. Now I'm stuck with the vg1 not being activated due to the invalid "thick" segment type, which appears to be unique to QNAP, since there is no LVM documentation supporting such a segment type. Reviewed the segtype.h file in the lvm2 project found here: https://git.fedorahosted.org/cgit/lvm2.git
Apparently QNAP extended the standard thin provisioning model of LVM2 in order to support new features (my guess is that this has to do with the ability to have thick-provisioned volumes within a thin-provisioning pool, which is a very desirable thing from a flexibility standpoint).
There are changes both in the userspace tools and in the kernel; this pretty much rules out the possibility of an easy recovery using a vanilla linux system.

Having said that, the QNAP GPL tarball at http://sourceforge.net/projects/qosgpl/ contains all the relevant sources, so I was able to build a working recovery environment by compiling and installing QNAP's kernel and LVM tools on a standard system.
would love to have a bootable image of that recovery if possible for the more unix challenged amongst us or perhaps you could do a step by step tut for creating same?
Model : TS-453 Pro
Firmware : 5.0.0.1828
4x WD RED 3TB - Raid 5
alyf80
New here
Posts: 3
Joined: Fri Jan 23, 2015 1:50 am

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-50

Post by alyf80 »

louiscar wrote:
alyf80 wrote: Having said that, the QNAP GPL tarball at http://sourceforge.net/projects/qosgpl/ contains all the relevant sources, so I was able to build a working recovery environment by compiling and installing QNAP's kernel and LVM tools on a standard system.
would love to have a bootable image of that recovery if possible for the more unix challenged amongst us or perhaps you could do a step by step tut for creating same?
What I cobbled together was more of a horrible hack than a real recovery environment; building something that can be used without a fair level of linux-fu would take a considerable amount of time, which I really don't have right now.
I can outlne the process, but you will probably have to adapt things based on your environment and any problems you might encounter.

Start with a clean linux install (you need persistent storage, so a live distro is probably not the best idea). I cloned a gentoo system I happened to have at hand, but any distribution should do.
You need a working development environment, so make sure you install the relevant packages (C compiler, binutils, glibc and kernel headers,...). I don't have a complete list, you will have to sort out the missing dependencies from any errors you get.

The first step is to install thin-provisioning-tools; these are just needed for a configure check so don't bother with the sources in the QNAP tarball: just installing your distribution's package should be fine.

From the tarball extract src/LVM2.2.02.96, configure it with --with-thin=internal, compile and install. If you get errors during the configuration or complilation, check your dependencies.

Now comes the fun part, i.e. the kernel.
Extract src/linux-3.12.6; for configuration I picked a default from the kernel_cfg folder in the tarball (I used TS870/linux-3.12.6-x86_64-hal.cfg, which is for an x86 NAS with specs reasonably similar to a recent PC) and adjusted it to include drivers for the hardware in my rescue system.
Everything you need (console, input, storage, network, ...) should be compiled in rather than built as a module: this is a rather ancient kernel and the userspace tools from your distribution might not play nice with it (in my case module loading did not work at all).

Compiling the kernel is a bit of a hassle, as QNAP seems to have sprinkled the sources with calls to code in a module of their own (picd) which is disabled in the configuration and cannot be enabled as it fails to compile.
I was not interested in debugging kernel build issues, so as a quick hack I left picd disabled and added

Code: Select all

#define send_message_to_app(...) do {} while(0)
at the end of include/qnap/pic.h, just before the final #endif; this effectively converts the offending calls to no-ops.

Then you need to reboot using the kernel image you just compiled; it might take some trials to get everything right, so be sure to keep the orginal kernel around so you can always go back and make changes.
At this point you should have a working system running a patched kernel and with a patched LVM installed; if all RAID volumes are connected and running, the usual

Code: Select all

# vgchange -ay
should not give any errors, and a subsequent

Code: Select all

# lvdisplay
should output a list of all your volumes in "available" state, which you can then mount.

Now, the usual disclaimers apply: this is what I did, and it worked for me. If you try to do the same and it kills your cat (or destroys your data), well, tough luck.
I did not write a detailed HOWTO for a good reason: if what I wrote above looks like gibberish to you, you should definitely not attempt to follow it.
Also, the method above is only intended as an emergency measure to gain read-only access to your volumes so that you can copy data off them. Do not mount the volumes read-write, and do not attempt any provisioning operations on the pools!

Hope this helps,
Andrea
User avatar
marshalleq
Easy as a breeze
Posts: 303
Joined: Mon Jul 23, 2012 2:44 pm
Contact:

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-500P)

Post by marshalleq »

There is no like button in these forums, so I'm just going to say thanks to father_mande. :)

Great post!
TS669-L (3GB RAM) fw 4.3.2.0050 Build 20170104 with 6x4TB Seagate ST4000VN000-1H41SC43 2x2TB Hitachi HDS723020BLA642 2x8TB ST8000VN0022-2EL112
QNAP UX-500P External 5 Bay USB3 Expansion Cabinet
Raspberry Pi 1,2 & 3
Mac Mini
Apple TV 3/4
Windows
Sabayon Linux
Ubuntu Linux
Android
Docker
Ryzen 1800x custom system

Web
https://www.tech-knowhow.com

Twitter
@Marshalleq
@TechKn0wH0w
buonboy
Starting out
Posts: 11
Joined: Fri Oct 05, 2018 6:30 pm

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-500P)

Post by buonboy »

Hello,
My NAS (TS231P2) died and does not turn on anymore, and also damaged my HDDs (RAID 1), so I can't access my data.
I got this error for both HDDs (with lvdisplay or vgdisplay):
WARNING: Unrecognized segment type tier-thin-pool
LV tp1, segment 1 invalid: does not support flag ERROR_WHEN_FULL. for tier-thin-pool segment.
Internal error: LV segments corrupted in tp1.

Any idea?
Thanks a lot.
User avatar
marshalleq
Easy as a breeze
Posts: 303
Joined: Mon Jul 23, 2012 2:44 pm
Contact:

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-500P)

Post by marshalleq »

I'm not sure why your disks would have died, but if I were in your shoes I would contact QNAP support. Assuming you've got a new NAS. Failing that - I think there are some instructions floating around on these forums about how to get your data off the drives with a PC - would need some linux skills from memory. Probably the best option is to buy a new NAS / get your old one going (Power Supply?) and use the Helpdesk feature on it - I think it allows them to connect in and fix it up.
TS669-L (3GB RAM) fw 4.3.2.0050 Build 20170104 with 6x4TB Seagate ST4000VN000-1H41SC43 2x2TB Hitachi HDS723020BLA642 2x8TB ST8000VN0022-2EL112
QNAP UX-500P External 5 Bay USB3 Expansion Cabinet
Raspberry Pi 1,2 & 3
Mac Mini
Apple TV 3/4
Windows
Sabayon Linux
Ubuntu Linux
Android
Docker
Ryzen 1800x custom system

Web
https://www.tech-knowhow.com

Twitter
@Marshalleq
@TechKn0wH0w
buonboy
Starting out
Posts: 11
Joined: Fri Oct 05, 2018 6:30 pm

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-500P)

Post by buonboy »

Thanks for reply.
My NAS is still under warranty, it died in less than two monthes, and QNAP support already told me to send it back and they will repair or change it, but the problem is that the HDDs are corrupted after NAS failure and QNAP do not accept it, neither care about it... So how can I be sure after NAS replacment the HDDs will work again and I will have access to my data?
And in the meantime I need to access to my data... I tried with some Recovery Software in Windows, they recover my data but they don't recognize my directory structure...
I'm not a Linux expert but I tried to follow some instruction around, however I can't go further, as I get that error.
User avatar
marshalleq
Easy as a breeze
Posts: 303
Joined: Mon Jul 23, 2012 2:44 pm
Contact:

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-500P)

Post by marshalleq »

I'm guessing a lot here, but when you say hard drives are damaged, I assume you mean the nas died while running, therefore some part of the data structure was corrupted. That does not = hard drive is damaged. So short of a lightening strike or some very odd power supply explosion the drives should not be damaged, but maybe the data is, or maybe just the part that boots the NAS. QNAP does and will help you get your data back, but if you tell them your drives are damaged they are likely to look at you with suspicious eyes.. :D

Give you are able to get some kind of data off with windows, again this seems to indicate your drives are not damaged.

My advice is get the NAS replaced, then try to get it working again, you might find the drives just work as is. Another option is to restore the NAS firmware (again need working NAS first) as it may then provide access to data - it may just be the firmware (which is just software on these) that is corrupted.

It is normal for directory structures to be wonky and other things when you restore from an alternate operating system like Windows. Though that also may mean that the actual partitions have corrupted which is more serious. Nevertheless I expect the drives are still not faulty - just the data. I don't know what your tech capability is, but if you must try to get the data back now - put all the drives from your raid set in a single PC, and hunt down the recovery instructions on this site. Alternately you could try downloading one of those linux rescue disks (bootable CD / USB). Linux recovers this kind of data a lot better than windows does and it's a free solution.
TS669-L (3GB RAM) fw 4.3.2.0050 Build 20170104 with 6x4TB Seagate ST4000VN000-1H41SC43 2x2TB Hitachi HDS723020BLA642 2x8TB ST8000VN0022-2EL112
QNAP UX-500P External 5 Bay USB3 Expansion Cabinet
Raspberry Pi 1,2 & 3
Mac Mini
Apple TV 3/4
Windows
Sabayon Linux
Ubuntu Linux
Android
Docker
Ryzen 1800x custom system

Web
https://www.tech-knowhow.com

Twitter
@Marshalleq
@TechKn0wH0w
P3R
Guru
Posts: 13190
Joined: Sat Dec 29, 2007 1:39 am
Location: Stockholm, Sweden (UTC+01:00)

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-500P)

Post by P3R »

buonboy wrote:My NAS is still under warranty, it died in less than two monthes, and QNAP support already told me to send it back and they will repair or change it...
Yes, that's what the warranty cover so that's what I'd expect from Qnap in this case.
...but the problem is that the HDDs are corrupted after NAS failure...
Since your NAS had failed, how could you determine your disks had been corrupted?
...and QNAP do not accept it, neither care about it...
Well if the disks are really corrupt, there's nothing they can do about it. What do you expect them to do?

You're responsible for your data, that's why you're always adviced by everyone with at least basic experience to do backups of the data you care about.
So how can I be sure after NAS replacment the HDDs will work again and I will have access to my data?
Obviously you can't until you try it. But if you have put the disks in some other machine trying whatever instructions you found on the internet, there's a good chance that you corrupted the disks by doing that.

If the disks are RAID 1, I hope that you at least didn't try those stunts on both disks. With RAID 1, always keep one disk separate and try what you think you need to do on the other.
RAID have never ever been a replacement for backups. Without backups on a different system (preferably placed at another site), you will eventually lose data!

A non-RAID configuration (including RAID 0, which isn't really RAID) with a backup on a separate media protects your data far better than any RAID-volume without backup.

All data storage consists of both the primary storage and the backups. It's your money and your data, spend the storage budget wisely or pay with your data!
buonboy
Starting out
Posts: 11
Joined: Fri Oct 05, 2018 6:30 pm

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-500P)

Post by buonboy »

Yes the NAS died while was running. And as I said in my last post the HDDs are corrupted, and the error I get is why they are corrupted. I sent QNAP support all the necessary information but they don't care and they say it's impossibile that a NAS can corrupt a HDD and they gave me just some basic procedure how to recover data from a disk which was part of a RAID, and obviously those procedures are not working. I sent the same information to three different Recovery Data companies and all of them confirmed that the HDDs are corrupted cause of NAS failure. So I don't trust QNAP anymore.
I have to find out how I can fix that error I get. But in the meantime I will try with some linux rescue disk.
Anyway I thought someone could help me with that error, but I receive more criticism than help. Thanks anyway.
User avatar
marshalleq
Easy as a breeze
Posts: 303
Joined: Mon Jul 23, 2012 2:44 pm
Contact:

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-500P)

Post by marshalleq »

I don't think anyone is intentionally trying to criticise you. The way you have written items above such as 'the NAS damaged my disks' made everyone suspicious. I don't know your experience in IT, but in order to help someone, specifics matter. They matter a lot. Then you correct damaged to corrupted and say QNAP refuse to accept this - so wondering which term you used with QNAP, but I know QNAP do help get your raid going. But they obviously can't do this without a functioning NAS. My recommendation is to start there. If it's RAID 1 (sorry I missed that in my original post) then there's a good chance you can put one of those disks into a linux recovery system if you want to go that way. Seriously though I'd just replace the NAS with another QNAP if you can wait.
TS669-L (3GB RAM) fw 4.3.2.0050 Build 20170104 with 6x4TB Seagate ST4000VN000-1H41SC43 2x2TB Hitachi HDS723020BLA642 2x8TB ST8000VN0022-2EL112
QNAP UX-500P External 5 Bay USB3 Expansion Cabinet
Raspberry Pi 1,2 & 3
Mac Mini
Apple TV 3/4
Windows
Sabayon Linux
Ubuntu Linux
Android
Docker
Ryzen 1800x custom system

Web
https://www.tech-knowhow.com

Twitter
@Marshalleq
@TechKn0wH0w
rramalho
New here
Posts: 7
Joined: Mon Oct 15, 2012 3:10 am

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-500P)

Post by rramalho »

Has anyone got any news on reading LVM2 with the dreaded tier-thin-pool segment type on normal Linux systems?

Has QNAP released those sources (kernel patches + LVM2 patches) to the public as the GPLv2 license demands?

My harddrives are OK, I know that my data is there, but I can't access it.

QNAP is being a jerk to its users...
User avatar
marshalleq
Easy as a breeze
Posts: 303
Joined: Mon Jul 23, 2012 2:44 pm
Contact:

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-500P)

Post by marshalleq »

If you can boot and get into your NAS, why not let them connect in via the Helpdesk app and fix it, or at least log a ticket? Unless this has already been done, or can't be done for some reason? That's what I would do if I couldn't fix myself.
TS669-L (3GB RAM) fw 4.3.2.0050 Build 20170104 with 6x4TB Seagate ST4000VN000-1H41SC43 2x2TB Hitachi HDS723020BLA642 2x8TB ST8000VN0022-2EL112
QNAP UX-500P External 5 Bay USB3 Expansion Cabinet
Raspberry Pi 1,2 & 3
Mac Mini
Apple TV 3/4
Windows
Sabayon Linux
Ubuntu Linux
Android
Docker
Ryzen 1800x custom system

Web
https://www.tech-knowhow.com

Twitter
@Marshalleq
@TechKn0wH0w
singlecell
First post
Posts: 1
Joined: Tue May 30, 2023 4:03 pm

Re: [SMB] HOW-TO RECOVER data from LVM volume on a PC (UX-500P)

Post by singlecell »

TVS-x72XT died - typical motherboard death. Use to have several in the office, so moved the drives to another unit, get it all back up and running and if under warranty - RMA otherwise landfill. All of the previous units died with 3 years - MB issues, and we quit buying them. Recently, I found one unit in a desk - never been used. So, I set it up and starting using it. It made just past 4 months before the MB red death light. Since it had been sitting in box for around 3 years - it was out of warranty. More money down the drain. But it did give me a opportunity to see if I could transfer my disks to a PC with a bunch of drive bays (8) - and recover the data. Here my notes:

Raid6 configurations with 1 volume group and 3 logical volumes - 1 volume encrypted and 1 directory encrypted.
Ubuntu 22.04 - no go. Too new. The LVM tools kept insisting I had a corrupt drive blocks - mainly because it was using LVM2 and the disks seemed to be LVM1 or something closer to that. Had to back up a few version to get things to start working.
Ubuntu 16.04 - Disk and Raid setup worked. So, first layer down. Decided this is a good starting point. Volume Group was visible and not active. Activated it - failed.
Found this thread - and noticed the step about building QNAP's version the LVM tools. Tried that - pulled the sources - rebuilt the tools and vgscan showed a volume group - success. lvscan shows 3 logical volumes - more success. try activating them - corrupt volume. It lied. After some digging around it was failing on the IOCTL in the kernel and giving a scary message. Also pointed out in this thread - need to pull in some tweaked kernel drivers to get it going.
So - as suggested, I pulled the kernel from QNAPs sources - a bit newer now - linux-5.10. merged the ubuntu and QNAP x72 configs and started building a new kernel. There were many errors. Most were easy enough to resolve - usually caused by QNAP additions. Modules worked for me - so, that was nice. Once I had a kernel - installed it and rebooted. Well, it didn't crash - but no raid now. Took a bit of debugging to figure out the problem was with a QNAP #endif statement being 1 line off which stopped the kernel from incrementing the device id's - essentially making all disks /dev/sda. Fixed that and the raid showed up w/o issue.

After that things went well. vgscan showed the volumes and lvscan showed the LV's. Activation of the VG activated all LV's as well. Mounted the LV's and there was my data. Success. Thanks to @alyf80 for giving enough info to get things going.

As for the encrypted stuff - for the encrypted LV - see viewtopic.php?t=133127
For the encrypted folders - I found the folder .__En__<MyFolder> in the LV I created it in. Not sure if all encrypted folders are at the root - my folder happened to be created at the top level. However it was easy enough to mount. QNAP uses the ecryptfs for folders. Just apt install ecryptfs-utils. once installed, use mount to mount the directory. i.e. 'mount -t ecryptfs /raid1/.__En__MyFolder' /<some-mount-point>'. In this case - raid1 is where I mounted my LV with the encrypted folder MyFolder and you can give any folder for the destination mount point.
It will prompt for the passphrase and ask for the crypt algo - 'AES' and ask for the key length - '32'. Success.
Locked

Return to “Users' Corner”