How to recover data from a QNAP Storage Pool (aka lvm2) in RAID1 configuration

Backup, Restore, Netbak Replicator, Cloud Storage Services
Post Reply
telemach
Getting the hang of things
Posts: 58
Joined: Tue Feb 28, 2012 4:01 am

How to recover data from a QNAP Storage Pool (aka lvm2) in RAID1 configuration

Post by telemach » Thu Sep 10, 2020 12:16 am

I’ve been running a RAID1 with primary (aka “legacy”) partition for quite a long time.
Recently, one of my drives started failing before its scheduled replacement, and I thought lockdown would be a good opportunity to move to an lvm2 logical volume (QNAP calls it “Storage Pool”) set-up.
(Disclaimer: I do have a backup, so I was able to take some risks).

Traditionally, I always replaced a drive with a bigger one, so I was able to grow the RAID over time. In this spirit, I bought a new, larger, drive, inserted it fresh, and built a single-drive storage pool.

As various other posts have suggested, it is not possible to re-insert the old drive and simply copy over the data, since QNAP doesn’t use UUIDs to name their partitioning scheme, but rather uses the same names for each “first” drive, so with two “first” drives, it is not possible to mount the second one due to name duplication (NAS becomes unresponsive when inserting).

Anyway, no problem, I mounted it externally via a SATA-to-USB adapter, and copied over the data. Easy.
My plan was now to re-insert the old drive and build the single-drive storage pool back into a RAID 1.
Unfortunately, I had made a school boy mistake here: The new drive was bigger than the old one, and while it is possible to join a bigger drive to a smaller RAID, to opposite does not work. I was not able to “Migrate the RAID”. :|

No problem I thought - I had already copied the data over to the new drive, so I would just start again.
I wiped the old (smaller) drive, and built a new Storage Pool on that. I connected the new (larger) drive via USB, and …. nothing :?: :!: .
I was not able to see or mount the logical volume with the data on it. Snap.
My first hunch was to connect it to an independent Linux system (CentOS in my case), but again, I had no luck (https://www.linuxjournal.com/article/8874 explains the reason, I was not willing to try the suggested workaround at this point).

After some more researching, I was able to piece together the steps for mounting an internal QNAP drive externally via a SATA-to-USB adapter, and they turned out to be the following:
  1. SSH into your box as admin (aka root, no sudo here).
  2. Run 'parted -l' for orientation. In my case, the external drive was '/dev/sdd', and the physical partition '/dev/sdd3' (the largest on this drive)
  3. Use 'mdadm --examine /dev/sdd' to check if there is a valid array on the disk:

    Code: Select all

    [~] # mdadm -E /dev/sdd3                 
    /dev/sdd3:
              Magic : a92f4efc
            Version : 1.0
        Feature Map : 0x0
         Array UUID : 01f75caf:43d55bc5:23030a17:5853cdbf
               Name : 1
      Creation Time : Sun Aug 30 09:28:58 2020
         Raid Level : raid1
       Raid Devices : 1
    
     Avail Dev Size : 11701135240 (5579.54 GiB 5990.98 GB)
         Array Size : 5850567616 (5579.54 GiB 5990.98 GB)
      Used Dev Size : 11701135232 (5579.54 GiB 5990.98 GB)
       Super Offset : 11701135504 sectors
       Unused Space : before=0 sectors, after=264 sectors
              State : clean
        Device UUID : d455b081:0e516ae2:aa86fbf2:27241bd7
    
        Update Time : Tue Sep  8 23:30:17 2020
      Bad Block Log : 512 entries available at offset -8 sectors
           Checksum : e1ac8fd - correct
             Events : 7
    
    
       Device Role : Active device 0
       Array State : A ('A' == active, '.' == missing, 'R' == replacing)
      
    Looks good.
  4. Run 'cat /proc/mdstat', and make a mental not of a number that is not used. E.g in my case, md1, md13, md256, md322, and md9 were in use, so I decided on md42

    Code: Select all

    [~] # cat /proc/mdstat
    Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] 
    md1 : active raid1 sda3[0]
          3897063616 blocks super 1.0 [1/1] [U]
          
    md322 : active raid1 sda5[0]
          7235136 blocks super 1.0 [2/1] [U_]
          bitmap: 1/1 pages [4KB], 65536KB chunk
    
    md256 : active raid1 sda2[0]
          530112 blocks super 1.0 [2/1] [U_]
          bitmap: 1/1 pages [4KB], 65536KB chunk
    
    md13 : active raid1 sda4[0]
          458880 blocks super 1.0 [64/1] [U_______________________________________________________________]
          bitmap: 1/1 pages [4KB], 65536KB chunk
    
    md9 : active raid1 sda1[0]
          530048 blocks super 1.0 [64/1] [U_______________________________________________________________]
          bitmap: 1/1 pages [4KB], 65536KB chunk
    
    unused devices: <none>
    
  5. Run 'mdadm --assemble /dev/md42 /dev/sdd3', where the first parameter is /dev/md + the unused number path from the step above, and the second the partition I want to mount

    Code: Select all

    [~] # mdadm --assemble /dev/md42 /dev/sdd3
    	mdadm: failed to get exclusive lock on mapfile - continue anyway...
    	mdadm: /dev/md42 has been started with 1 drive.
    
  6. At this point, you may need to rename your logical volume groups. Run 'vgs -o +vg_uuid' to find the UUID of the drive in question

    Code: Select all

    [~] # vgs -o +vg_uuid
    	WARNING: duplicate PV RIyfffCz2nOmbxMUYWiZ3kdRtRq7Lkyw is being used from both devices /dev/drbd1 and /dev/md1
    	Found duplicate PV RIyfffCz2nOmbxMUYWiZ3kdRtRq7Lkyw: using /dev/drbd1 not /dev/md1
    	Using duplicate PV /dev/drbd1 from subsystem DRBD, ignoring /dev/md1
    	VG   #PV #LV #SN Attr   VSize VFree VG UUID                               
    	vg1    1   3   0 wz--n- 3.63t    0  gjUwmz-fahK-Gpzj-ycN2-DB2N-Xdv3-fNmXxt
    	vg1    1   3   0 wz--n- 5.45t    0  VZjqKb-BJ0w-1HI7-t6qm-cFSB-TTsY-V3tKuS
    
    You can see we have two volume groups called 'vg1' here - that is not going to work.
  7. Use 'lvm vgrename' to rename the RAID, for example:

    Code: Select all

    lvm vgrename gjUwmz-fahK-Gpzj-ycN2-DB2N-Xdv3-fNmXxt vg7  
    
  8. Run 'vgscan --cache'. This will discover the logical volumes in the above attached RAID array

    Code: Select all

    [~] # vgscan  --cache
    	Reading all physical volumes.  This may take a while...
    	Found volume group "vg7" using metadata type lvm2
    	WARNING: duplicate PV YqfzJVroGAx5SFpNJUrC2f0ijgHUm0Y7 is being used from both devices /dev/drbd1 and /dev/md1
    	Found duplicate PV YqfzJVroGAx5SFpNJUrC2f0ijgHUm0Y7: using existing dev /dev/drbd1
    	Found volume group "vg1" using metadata type lvm2
    
  9. Run 'lvdisplay'. This will show all logical volumes. Pick the one you want to mount (/dev/vg7/lv1 in my case - again going by size should give a good hint).

    Code: Select all

    	--- Logical volume ---
    	LV Path                /dev/vg7/lv1
    	LV Name                lv1
    	VG Name                vg7
    	LV UUID                wXLuQK-jYz1-1dSL-fLL2-SSit-YlVB-k3fD89
    	LV Write Access        read/write
    	LV Creation host, time skc-nas-4, 2020-08-30 09:36:16 +0100
    	LV Pool name           tp1
    	LV Status              available
    	# open                 1
    	LV Size                3.60 TiB
    	Mapped size            100.00%
    	Mapped sectors         7731150848
    	Current LE             943744
    	Segments               1
    	Allocation             inherit
    	Read ahead sectors     8192
    	Block device           253:15
    
  10. Now run 'lvchange -ay /dev/vg7/lv1' to activate the volume (so vg7 is the renamed volume group, and lv1 is the first volume in this group)

    Code: Select all

    [~] # lvchange -ay /dev/vg7/lv1
    	#No output
    
  11. Create a directory to mount the volume in: 'mkdir /mnt/sdd'
  12. Now mount it in that directory: 'mount -t ext4 -o ro /dev/vg7/lv1 /mnt/sdd'
  13. 'ls /mnt/sdd' should now hopefully show the shared folder structure
  14. From here, start copying the old files to the new drive, e.g. 'rsync -aAXvv --exclude={'@Recently-Snapshot','@Recycle','.DS_Store','._.DS_Store','.@__thumb','.@__qini'} /mnt/sdd/Shared/ /share/CACHEDEV1_DATA/Shared'

    Code: Select all

    [~] # rsync -aAXvv  --exclude={'@Recently-Snapshot','@Recycle','.DS_Store','._.DS_Store','.@__thumb','.@__qini'} /mnt/sdd/Shared/ /share/CACHEDEV1_DATA/Shared
    	#Lots of output
    
That's it!
Once you're done, you can unmout the volume like so:

Code: Select all

[~] #  umount /dev/vg7/lv1  # unmount the filesystem
[~] #  lvchange -an /dev/vg7/lv1  # deactivate the volume
[~] #  mdadm -S /dev/md42  # stop the RAID
Note: When exporting and re-importing a QNAP configuration backup, it seems to remember the exported storage pool.
This is a deal breaker, so unfortunately, it looks like all configuration needs to be built from scratch.

RAID5 and other RAID configuration should work in a similar way, except that when rebuilding the RAID with mdadm, you would need to group the various drives together.
Also thanks to viewtopic.php?f=25&t=112889&start=15, which got me started, and https://wiki.elvanor.net/index.php/QNAP_NAS, which directed me to the right path.

This was done on a TS-253 Pro using firmware 4.4.3.1400.
Hope this helps somebody.

sfagg
Starting out
Posts: 13
Joined: Tue Jan 04, 2011 7:00 pm

Re: How to recover data from a QNAP Storage Pool (aka lvm2) in RAID1 configuration

Post by sfagg » Sun Sep 20, 2020 11:40 pm

Ah!
This is almost exactly what I'm wanting to do, except, I'm wanting to mount the drive in one of the 6 internal bays.

I got all the way up to step 12 but it's saying 'wrong fs type'.

This is my specific drive information from parted:

Code: Select all

[~] # parted /dev/sdd print
Model: WDC WD20EARX-00PASB0 (scsi)
Disk /dev/sdd: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name     Flags
 1      20.5kB  543MB   543MB   ext3            primary
 2      543MB   1086MB  543MB   linux-swap(v1)  primary
 3      1086MB  1991GB  1990GB                  primary
 4      1991GB  1992GB  543MB   ext3            primary
 5      1992GB  2000GB  8554MB  linux-swap(v1)  primary
 
The output from mdadm:

Code: Select all

[~] # mdadm --examine /dev/sdd3
/dev/sdd3:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x1
     Array UUID : 20b4fccc:63625242:896a8e26:d3228c13
           Name : 1
  Creation Time : Tue Apr  7 12:10:51 2015
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 3887119240 (1853.52 GiB 1990.21 GB)
     Array Size : 1943559616 (1853.52 GiB 1990.21 GB)
  Used Dev Size : 3887119232 (1853.52 GiB 1990.21 GB)
   Super Offset : 3887119504 sectors
   Unused Space : before=0 sectors, after=256 sectors
          State : clean
    Device UUID : 200aaa88:99c260f2:71fdc5cf:fe5a31ad

Internal Bitmap : -16 sectors from superblock
    Update Time : Sat Sep 19 21:05:00 2020
       Checksum : 563aa3fc - correct
         Events : 342027


   Device Role : Active device 1
   Array State : .A ('A' == active, '.' == missing, 'R' == replacing)
The output from mdstat looked the same as yours, so I too opted for md42.

At step 5, I also got the same output as you, stating that 'failed to get exclusive lock on mapfile - continue anyway...' but '/dev/md42 has been started with 1 drive.'... so assuming good so far?

At step 6 there was no output from vgs -o +vg_uuid. Which is expected as this is the only array currently live.

At step 8, vgscan produced:

Code: Select all

[~] # vgscan --cache
  Reading all physical volumes.  This may take a while...
  Found volume group "vg1" using metadata type lvm2
At step 9, lvdisplay output:

Code: Select all

[~] # lvdisplay
  --- Logical volume ---
  LV Path                /dev/vg1/lv544
  LV Name                lv544
  VG Name                vg1
  LV UUID                8FJgr5-9kp7-HCD2-Zcjd-hSow-2XmV-ZeU3Cu
  LV Write Access        read/write
  LV Creation host, time QNAP, 2015-04-07 12:10:54 +0100
  LV Status              NOT available
  LV Size                18.54 GiB
  Current LE             4745
  Segments               1
  Allocation             inherit
  Read ahead sectors     8192

  --- Logical volume ---
  LV Path                /dev/vg1/lv1
  LV Name                lv1
  VG Name                vg1
  LV UUID                PzkhNN-co1k-Uj0E-ZHrM-GRFP-Y1xL-HJ7h5X
  LV Write Access        read/write
  LV Creation host, time QNAP, 2015-04-07 12:11:03 +0100
  LV Status              NOT available
  LV Size                1.79 TiB
  Current LE             469756
  Segments               1
  Allocation             inherit
  Read ahead sectors     8192
The LV Status concerns me a bit as it states 'NOT available'... could this be the cause of my problem later on?

The rest was fine up until step 12 where I get:

Code: Select all

[~] # mount -t ext3 -o ro /dev/vg1/lv1 /mnt/sdd
mount: wrong fs type, bad option, bad superblock on /dev/vg1/lv1,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
The output of dmesg isn't particularly helpful either, just stating it can't find ext4 filesystem :(

Code: Select all

[~] # dmesg | tail
[37636.970837] md/raid:md13: report qnap hal event: raid_id=13, pd_name=/dev/(null), spare=/dev/(null), pd_repair_sector=0
[37638.610880] md: md9: recovery done.
[37638.614396] md: Recovering done: md9
[37638.617974] md/raid:md9: report qnap hal event: type = HAL_EVENT_RAID, action = REBUILDING_COMPLETE
[37638.627013] md/raid:md9: report qnap hal event: raid_id=9, pd_name=/dev/(null), spare=/dev/(null), pd_repair_sector=0
[60551.360114] md: md42 stopped.
[60551.391671] md/raid1:md42: active with 1 out of 2 mirrors
[60551.412540] md42: detected capacity change from 0 to 1990205046784
[60754.943923] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
[60805.162313] EXT4-fs (dm-0): VFS: Can't find ext4 filesystem
I'm hoping you might have an idea as to what's gone on with it.

Just FYI, I had the setup as stated in my signature previously with 4x3TB drives in RAID5 and 2x2TB drives in RAID1. One of the 2TB drives had died and one of the 3TB drives was giving errors when the SMART data was trying to be read (I mean, the NAS literally *could not* read the SMART data, not that there SMART data reported some errors).
So, I bought 4 new 4TB drives (as I was running low on space I thought I'd up capacity a bit) and made use of 2 of the old 3TB drives to make a large storage pool - obviously, with the remaining 2TB drive removed as I've got a 6 bay NAS.

So, this morning, I wake up and think... Oh, hang on, how am I going to get the data off the 2TB drive because I didn't have anywhere to move that data to before migrating it all back onto the new array.
At this point, I removed all volumes on the new storage pool and then removed the storage pool.
I then re-inserted the old 2TB drive - expecting the QNAP to go 'Ooo, here's a RAID1 array in degraded performance mode, I'll just mount it for you'. But, no. Sadly not.

And this is the point I'm at, I just need to get this drive mounted, then I can re-create the storage pool and volumes, copy the data off, remove the drive, then re-add the 2 3TB drives, increase the capacity and that should be it...!

Hopefully you might be able to assist me on the mounting bit.

Thanks
Steve
TS-653 Pro
RAID 5: 3 x 3TB WD30EFRX
RAID 2: 2 x 2TB WD20EARX

sfagg
Starting out
Posts: 13
Joined: Tue Jan 04, 2011 7:00 pm

Re: How to recover data from a QNAP Storage Pool (aka lvm2) in RAID1 configuration

Post by sfagg » Wed Sep 23, 2020 5:43 am

It seems I forgot that this drive had been encrypted too...
I used e2fsck_64 to attempt to check the file system.
It complained that the superblock was invalid and suggested I tried 8193... Trying that it stated that the file system was 'crypt_LUKS'.

A google search for that brought me to this site https://www.linux-howto.info/mount-qnap ... ed-volume/.
So I followed your steps all the way up to step 10, then continued from the above site.
I managed to mount the array and see my files.
I now need to get my array sorted out now that I know I can access my files!

Thanks for your original post. It helped greatly.

Steve
TS-653 Pro
RAID 5: 3 x 3TB WD30EFRX
RAID 2: 2 x 2TB WD20EARX

craigliu
New here
Posts: 6
Joined: Tue Sep 22, 2015 4:43 am

Re: How to recover data from a QNAP Storage Pool (aka lvm2) in RAID1 configuration

Post by craigliu » Thu Oct 22, 2020 2:15 am

I have done exactly the same as yours, but I met a problem. The error msg is "WARNING: Unrecognised segment type thick". So I cannot activate the lv.

[root@mill-srv ~]# vgscan
Reading volume groups from cache.
WARNING: Unrecognised segment type thick
Found volume group "vg6" using metadata type lvm2
[root@mill-srv ~]# lvscan
WARNING: Unrecognised segment type thick
ACTIVE '/dev/vg6/lv549' [20.00 GiB] inherit
inactive '/dev/vg6/tp6' [3.59 TiB] inherit
inactive '/dev/vg6/lv9' [3.59 TiB] inherit
[root@mill-srv ~]# lvchange -a y /dev/vg6/tp6
WARNING: Unrecognised segment type thick
Check of pool vg6/tp6 failed (status:1). Manual repair required!
[root@mill-srv ~]# lvchange -a y /dev/vg6/lv9
WARNING: Unrecognised segment type thick
Refusing activation of LV vg6/lv9 containing an unrecognised segment.

What should I do? Thanks very much. I posted my problem in viewtopic.php?f=15&t=157479.

Post Reply

Return to “Backup & Restore”