Invisible Snapshot that block all space

[Razzorhead]

Hello

I don't know what to do. My storage pool run full but I didnt do anything, when I checked a lot of space is used by Snapshots - however Snapshots are deactivated. It was a thick volume, I can't do anything because it is Protected (Read/Delete Possible) now the only possible thing was to turn it into a thin volume which I did but there is where my possibilities end.

I also increased the Space on the Volume as a last resort, thats why it is over provisioned now, but it was not before.

qnap1.JPG

qnap2.JPG

qnap3.JPG

(It is in german but there is not much to see anyway - everything deactivated, no snapshots visible, but snapshot space used.

[dolbyman]

Please always switch your UI to English when posting in the English Forum

There is also a German forum
https://forum.qnapclub.de

Volume type is not visible on your screenshots, any error messages on the volume are also cut off there

[dombo]

Razzorhead were you able to solve this? I'm in the same situation.

[dolbyman]

You probably need to delete some snapshots to free up pool space..several threads about this have been created since this one here was opened

[dombo]

I have no snapshots on my qnap and they are not visible in logical volumes:

Code: Select all

[~] # lvs -v    
    Using logical volume(s) on command line.
    cache_pool text_import start
    cache_pool text_import end
    cache_pool export chunk_size:2048, client_num:0
    cache_pool export fin
    cache_pool text_import start
    cache_pool text_import end
    target_name:cache
    Checking cache status for LV vg256/lv256.
    Checking cache status for LV vg256/lv256.
    _cache_text_import1
    cache text_import flag:265 end
    cache export client_id=10002, cache_mode:writeback, policy:smqv, io_mode:read-write, clean:0
    _cache_text_import1
    cache text_import flag:265 end
    target_name:thin-pool
    target_name:thin
    target_name:thin
  LV     VG    #Seg Attr       LSize   Maj Min KMaj KMin Pool Origin Data%  Meta%  Move Cpy%Sync Log Convert LV UUID                                LProfile
  lv1    vg1      1 Vwi-aot---  52.00t  -1  -1  253   10 tp1         99.81                                   kJc9jO-iWWj-Npfa-6JFh-j3ON-YpOJ-tQT2qt         
  lv1312 vg1      1 -wi-a-----   6.70g  -1  -1  253   11                                                     yfPKhG-wJe9-zrYc-71Iq-cgfi-bOku-F0trh9         
  lv2    vg1      1 Vwi-aot---   1.00g  -1  -1  253   13 tp1         4.00                                    C9qCZ4-aTJR-B80e-4c7u-3LgC-5CcJ-Q4ddyi         
  lv544  vg1      1 -wi------- 137.30g  -1  -1   -1   -1                                                     WiqZCh-WlZs-0VVQ-yrTY-5red-3oMY-hIyNCt         
  tp1    vg1      1 twi-aot---  65.22t  -1  -1  253    8             99.28  9.16                             3n1X1h-X0iJ-8Wwv-V80L-zMgH-y27p-lUIOTk         
  lv256  vg256    1 Cwi-aoC--- 826.44g  -1  -1  253    6             85.38  0.49                             FEa1sp-jWDj-2N7z-BP5F-qZcI-ShWj-OszZoL         
  lv545  vg256    1 -wi-------   8.50g  -1  -1   -1   -1                                                     90iPTt-rdqL-PuQQ-6El6-Ebpi-cCka-G1nZs5         
[~] # 

so pool has 65.22TB (tp1) with almost 100% usage (99.28%) while virtual volumes (lv1 and lv2) in this pool are occupying max 52TB+1GB, so ~13TB is missing somewhere

[dombo]

I was able to fix my problem by following this procedure. https://pariswells.com/blog/research/qn ... -repairing

[sroslevich]

Dombo, Does this fix the issue moving forward as well or will you have to go through the same steps when ghost-snapshots fill up the space again? Also, does that method preserve any snapshots that are correctly being stored? I am having the same issue as the others but do not want to lose the good snapshots.

[sarangtc@gmail.com]

Model: TS-1273U-RP
FW: 5.0.0.1932

I have tried the following
https://docs.qnap.com/operating-system/ ... CEA8E.html
says "there are no snapshots yet"

going directly to the snapshot section
the snapshot space utilization graph shows over 10TB used by snapshot

Screenshot 2022-03-07 221040.png

tried:
viewtopic.php?t=154206#p774188
viewtopic.php?f=25&t=133225#p619286

got:

Code: Select all

[~] # lvs
  LV        VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv1       vg1  Vwi-aot---  80.00t tp1         77.25
  lv1312    vg1  -wi-a-----   7.44g
  lv544     vg1  -wi------- 144.00g
  snap10005 vg1  Vwi-a-t---  80.00t tp1  lv1    82.91
  tp1       vg1  twi-aot---  72.47t             99.98  9.63
[~] # umount /dev/mapper/vg1-snap10005
umount: /dev/mapper/vg1-snap10005: not mounted
[~] # lvremove -ff vg1/snap10005
  Failed to delete snapshot snap10005 due to wrong passphrase
[~] #

tried viewtopic.php?f=25&t=133225#p619288
got:

Code: Select all

[~] #  dmsetup status vg1-snap10005
0 171798691840 thin 142430385152/0 171798430719 8497 0 0
[~] # lvdatac vg1/snap10005 -discard
lvdatac: invalid option -- 'i'
  Error during parsing of command line.
[~] # lvdatac vg1/snap10005 --discard
lvdatac: unrecognized option '--discard'
  Error during parsing of command line.
[~] # lvdatac vg1/snap10005 --discard
lvdatac: unrecognized option '--discard'
  Error during parsing of command line.
[~] # lvdatac vg1/snap10005 -discard
lvdatac: invalid option -- 'i'
  Error during parsing of command line.
[~] # lvdatac vg1/snap10005

[sarangtc@gmail.com]

I was able to fix my problem by following this procedure. https://pariswells.com/blog/research/qn ... -repairing

These instructions are fairly daunting.
Will they work with Firmware version 5.0.0.1932 ?
I couldn't understand the tp part.

If my snapshot is vg1-snap10005 how should I proceed?

[neoviktor]

Sorry to resurrect this thread, but I'm facing same problem, do anyone found how to fixit?

[dolbyman]

If it still the case with your (unknown) NAS and (unknown) firmware, best to open a ticket with qnap so they can fix that bug

[michasm]

I was struggling with same issue but I fixed it. It took me few days but solution was simple.
Model: TS-431X
FW: 5.0.1.2173

Code: Select all

[~] # lvs
  LV        VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv1       vg1  Vwi-aot---  80.00t tp1         77.25
  lv1312    vg1  -wi-a-----   7.44g
  lv544     vg1  -wi------- 144.00g
  snap10005 vg1  Vwi-a-t---  80.00t tp1  lv1    82.91
  tp1       vg1  twi-aot---  72.47t             99.98  9.63
[~] # umount /dev/mapper/vg1-snap10005
umount: /dev/mapper/vg1-snap10005: not mounted
[~] # lvremove -ff vg1/snap10005
  Failed to delete snapshot snap10005 due to wrong passphrase
[~] #

Same as here but for me broken snapshot was snap10002. It was invisible in snapshot manager and it was growing all the time.
Above sarangtc@gmail.com has broken snap10005.
To fix it I made some free space to make single proper snapshot.
Then I Went to
Storage/Snapshots->Snapshot Manager->Pool Guaranteed Snapshot Space->Configure->Snapshot Retention->Maximum number of snapshots to keep: set to 2
Then I made another proper snapshot. It took about 10 minutes and then invisible snap10002 was replaced by proper one.
I had back a lot of space.

So in case above where you have invisible snap10005 you should create at least 5 proper snapshots so the 5th one will replace broken one.
Those are numbered as snap10001, snap10002, ..., snap10005 and so on.

Still I have no idea why whis was growing all the time (up tu 2,2TB on 11TB space...) but it's fixed.

I hope this will help someone.

[denormative]

Same as here but for me broken snapshot was snap10002. It was invisible in snapshot manager and it was growing all the time.
Above sarangtc@gmail.com has broken snap10005.
To fix it I made some free space to make single proper snapshot.
Then I Went to
Storage/Snapshots->Snapshot Manager->Pool Guaranteed Snapshot Space->Configure->Snapshot Retention->Maximum number of snapshots to keep: set to 2
Then I made another proper snapshot. It took about 10 minutes and then invisible snap10002 was replaced by proper one.
I had back a lot of space.

So in case above where you have invisible snap10005 you should create at least 5 proper snapshots so the 5th one will replace broken one.
Those are numbered as snap10001, snap10002, ..., snap10005 and so on.

Still I have no idea why whis was growing all the time (up tu 2,2TB on 11TB space...) but it's fixed.

I hope this will help someone.

Thanks! Was fighting with trying to solve the same problem for a week and this is the only thing that worked. I had 3.5TB of snapshots somehow stuck as snap10004 and snap10006, and basically the same process worked for me. Delete enough space on my raid array to be able to make snapshots, and then snap six of them one by one and it eventually freed up the space. 10004 and 10006 took a long time, but I'm guessing that was because it was freeing up all the space the old snapshot was taking.

[dolbyman]

but again...tickets should be opened, as manual fiddling with snapshots via SSH should not be required (unless it's a bug)