Replace cache SSD on TVS-EC2480U-SAS

[AndyAlreadyInUse]

Hi!

We have a TVS-EC2480U-SAS running. Now it reports an error with one of the "Cache modules". So I ordered two new mSata SSD drives. Going to replace both.
I wonder if there is anything I should pay attention to before/while replacing the SSD.

The NAS is still online, but files are inaccessible. All data drives are green, no issues. Plan is to shut down the device. Open the cover, find the SSD modules, disconnect old ones, connect the new ones. Power up unit.

Any suggestions?
Are the SSD cache modules mandatory, or could the NAS run without? They are supposed to be a perfomance feature, but might be part of the design.

Thanks,
Andy

P.S. I hope that all data is still on the data drives, but currently it says 0Bytes and it says there are no folders.

[QNAPDanielFL]

The NAS is still online, but files are inaccessible. All data drives are green, no issues. Plan is to shut down the device. Open the cover, find the SSD modules, disconnect old ones, connect the new ones. Power up unit.

Any suggestions?
Are the SSD cache modules mandatory, or could the NAS run without? They are supposed to be a perfomance feature, but might be part of the design.

Is your cache set to READ/Write or READ Only?
I don't know why the files are not accessible, but if you have any kind of Write cache, cache going down or there being a power outage can have some risk. You can make a support ticket to make sure you deal with this in the right way if you have Write or READ/Write cache.

The NAS does not need cache to work, but some SSD cache set to random only can help with the random performance.

[AndyAlreadyInUse]

@QNAPDanielFL - thanks a lot.

Checked that the cache is read-only - as I understand this is good news. Tried to turn off caching, but got an error that the cache drives cannot be disabled:
2022-07-04 08:58:33 System 127.0.0.1 Storage & Snapshots Cache Acceleration [Storage & Snapshots] Failed to disable SSD cache on SSD " Host Internal Cache Module: 1 2". RAID type: RAID0, Cache type: Read-only.

Today or tomorrow the new SSD drives should arrive. Will then either find some time to take the QNAP out of the rack to replace the drives (no space to access it in the rack) or will need to convince some of my more hardware related colleagues to do that.

[AndyAlreadyInUse]

Tried to remove Cache but that failed too: Storage & Snapshots Cache Acceleration [Storage & Snapshots] Failed to remove SSD cache.

And saw that the storage volume is in some safe mode - write protected. Multiple possible reasons mentioned, with one being a RAID issue. The only issue currently reported is the one with the cache.

Any suggestion on how to force disable/remove cache in such a condition?

Thanks!

[QNAPDanielFL]

I think it would be best to make a support ticket.
I don't know why you are having this problem, but support can advise on how to best protect your data since something is wrong enough to make your pool go to safe mode.

[QNAPDanielFL]

Has Tech Support been able to help?
I am discussing your situation and if you tell me the ticket number we would like to look closer at your case.
If you have dump logs to share on the ticket that would also help.

[AndyAlreadyInUse]

@QNAPDanielFL - thanks for asking. No, I have not opened a support ticket yet.

Today I got the replacement SSDs for the cache modules. Turns out I ordered the wrong format sata instead of mSata (Amazon fooled me by adding sata results to my mSata search and I did not pay attention).
So, I removed the broken SSD cache modules today. Rebooted qnap without the modules.

All drives are green but the Volume is "unmounted". I guess the missing cache is messing up the mounting.
File system check does not run: Fehler 2022-07-08 11:43:50 System 127.0.0.1 Storage & Snapshots Volume [Storage & Snapshots] Failed to check file system. Storage pool: 1.


Some console output:

Code: Select all

/mnt/HDA_ROOT/.logs$ md_checker

Welcome to MD superblock checker (v2.0) - have a nice day~

Scanning system...


RAID metadata found!
UUID:		6224a1cf:5365547a:44b00ef0:80ebeb2e
Level:		raid5
Devices:	24
Name:		md2
Chunk Size:	512K
md Version:	1.0
Creation Time:	Aug 26 09:58:36 2018
Status:         [1;32mONLINE[1;0m (md2) [UUUUUUUUUUUUUUUUUUUUUUUU]
===============================================================================================
 Enclosure | Port | Block Dev Name | # | Status |   Last Update Time   | Events | Array State
===============================================================================================
 NAS_HOST      13        /dev/sde3   0   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      14        /dev/sdj3   1   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      15        /dev/sdv3   2   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      16        /dev/sdw3   3   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      17        /dev/sdf3   4   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      18        /dev/sdi3   5   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      19        /dev/sdp3   6   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      20        /dev/sdq3   7   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      21        /dev/sdg3   8   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      22        /dev/sdh3   9   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      23        /dev/sdn3  10   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      24        /dev/sdo3  11   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      11        /dev/sdx3  12   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      10        /dev/sdk3  13   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST       9        /dev/sdd3  14   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST      12        /dev/sdy3  15   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST       8        /dev/sds3  16   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST       7        /dev/sdr3  17   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST       6        /dev/sdl3  18   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST       5        /dev/sdc3  19   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST       4        /dev/sdt3  20   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST       3        /dev/sdu3  21   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST       2        /dev/sdm3  22   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
 NAS_HOST       1        /dev/sdb3  23   [0;94mActive[0m   Jul  8 11:26:25 2022   111327   AAAAAAAAAAAAAAAAAAAAAAAA 
===============================================================================================

Code: Select all

/mnt/HDA_ROOT/.logs$ pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/drbd2 vg1  lvm2 a--  208.99t    0 

Code: Select all

/mnt/HDA_ROOT/.logs$ lvs -a 
  LV                      VG   Attr       LSize   Pool Origin                  Data%  Meta%  Move Log Cpy%Sync Convert
  lv1312                  vg1  -wi-a-----  21.40g                                                                     
  lv2                     vg1  Vwi---t--- 187.86t tp1                                                                 
  lv545                   vg1  -wi-a----- 144.00g                                                                     
  tp1                     vg1  twi---t--- 208.76t                                                                     
  [tp1_tierdata_0]        vg1  vwi-a-v---   4.00m                                                                     
  [tp1_tierdata_1]        vg1  vwi-a-v---   4.00m                                                                     
  [tp1_tierdata_2]        vg1  Cwi---C--- 208.76t      [tp1_tierdata_2_fcorig]                                        
  [tp1_tierdata_2_fcorig] vg1  owi-a-C--- 208.76t                                                                     
  [tp1_tmeta]             vg1  ewi-a-----  64.00g                                                                     

Code: Select all

/mnt/HDA_ROOT/.logs$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
none                    409600    300716    108884  73% /
devtmpfs               8126556         4   8126552   0% /dev
tmpfs                    65536       500     65036   1% /tmp
tmpfs                  8141200       136   8141064   0% /dev/shm
tmpfs                    16384        84     16300   1% /share
/dev/sda5                 8028        28      8000   0% /mnt/boot_config
tmpfs                    16384         0     16384   0% /mnt/snapshot/export
/dev/md9                505300    151852    353448  30% /mnt/HDA_ROOT
cgroup_root            8141200         0   8141200   0% /sys/fs/cgroup
/dev/md13               426984    411444     15540  96% /mnt/ext
tmpfs                    32768     27900      4868  85% /samba_third_party
/dev/ram2               444276      2318    441958   1% /mnt/update
tmpfs                        4         0         4   0% /tmp/default_dav_root
tmpfs                    65536     29960     35576  46% /samba

Code: Select all

/mnt/HDA_ROOT/.logs$ vgchange -ay vg1
  Device /dev/mapper/vg256-lv256 not found
  Failed to set name for /dev/mapper/vg256-lv256 rename.
  Failed to dm info cache client pool /dev/mapper/vg256-lv256 to get major & minor
  Device /dev/mapper/vg256-lv256 not found
  Failed to set name for /dev/mapper/vg256-lv256 rename.
  Failed to dm info cache client pool /dev/mapper/vg256-lv256 to get major & minor
  2 logical volume(s) in volume group "vg1" now active

Code: Select all

/mnt/HDA_ROOT/.logs$ /etc/init.d/init_lvm.sh
Changing old config name...
Reinitialing...
/bin/cat: /sys/bus/pci/devices/0000:00:1c.0/0000:08:00.0/subsystem_vendor: No such file or directory
/bin/cat: /sys/bus/pci/devices/0000:00:1c.0/0000:08:00.0/subsystem_vendor: No such file or directory
Detect disk(65, 128)...
dev_count ++ = 0Detect disk(8, 80)...
dev_count ++ = 1Detect disk(8, 224)...
dev_count ++ = 2Detect disk(65, 96)...
dev_count ++ = 3Detect disk(8, 48)...
dev_count ++ = 4Detect disk(8, 192)...
dev_count ++ = 5Detect disk(65, 64)...
dev_count ++ = 6Detect disk(8, 16)...
dev_count ++ = 7Detect disk(8, 160)...
dev_count ++ = 8Detect disk(65, 32)...
dev_count ++ = 9Detect disk(8, 128)...
dev_count ++ = 10Detect disk(65, 0)...
dev_count ++ = 11Detect disk(8, 96)...
dev_count ++ = 12Detect disk(65, 112)...
dev_count ++ = 13Detect disk(8, 64)...
dev_count ++ = 14Detect disk(8, 208)...
dev_count ++ = 15Detect disk(65, 80)...
dev_count ++ = 16Detect disk(8, 32)...
dev_count ++ = 17Detect disk(8, 176)...
dev_count ++ = 18Detect disk(65, 48)...
dev_count ++ = 19Detect disk(8, 0)...
ignore non-root enclosure disk(8, 0).
Detect disk(8, 144)...
dev_count ++ = 20Detect disk(65, 16)...
dev_count ++ = 21Detect disk(8, 112)...
dev_count ++ = 22Detect disk(8, 240)...
dev_count ++ = 23Detect disk(65, 128)...
Detect disk(8, 80)...
Detect disk(8, 224)...
Detect disk(65, 96)...
Detect disk(8, 48)...
Detect disk(8, 192)...
Detect disk(65, 64)...
Detect disk(8, 16)...
Detect disk(8, 160)...
Detect disk(65, 32)...
Detect disk(8, 128)...
Detect disk(65, 0)...
Detect disk(8, 96)...
Detect disk(65, 112)...
Detect disk(8, 64)...
Detect disk(8, 208)...
Detect disk(65, 80)...
Detect disk(8, 32)...
Detect disk(8, 176)...
Detect disk(65, 48)...
Detect disk(8, 0)...
ignore non-root enclosure disk(8, 0).
Detect disk(8, 144)...
Detect disk(65, 16)...
Detect disk(8, 112)...
Detect disk(8, 240)...
  WARNING: duplicate PV IGFDxnV1u9GvcOgXycjFT5selGioTCq1 is being used from both devices /dev/drbd2 and /dev/md2
  Found duplicate PV IGFDxnV1u9GvcOgXycjFT5selGioTCq1: using /dev/drbd2 not /dev/md2
  Using duplicate PV /dev/drbd2 from subsystem DRBD, ignoring /dev/md2
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
  LV Status              NOT available
sys_startup_p2:got called count = -1
Done

[AndyAlreadyInUse]

Created a ticket with the German speaking support: Q-202207-36067. Hopefully they can help.
I would have expected that a cache is a cache(it was read-only), but seems like it might be integrated deeper.

[AndyAlreadyInUse]

Update: QNAP support sent me some instructions, got access to the data (currenly only via manual mount)

Main steps:
backup pool config:

# cd /dev/mapper
# vgcfgbackup vg1 -f vg1.lvm
# vgcfgbackup vg1 -f vg1.lvm.original

Edit vg1.lvm:
- change cacheVersion from 3 to 2
- remove specific section from the config (I believe that section defined the usage of the cache), but I can't tell exactly

Then restore modified pool header, bring it online, mount to test, where {ID} had to be looked up from my mapper directory:

# vgcfgrestore vg1 -f vg1.lvm --force
# lvchange -ay vg1
# mount -t ext4 /dev/mapper/vg1-lv{ID} /mnt/test -o ro,noload

Now I can access all data. Now I hope QNAP support will also help me to get it back auto-mounted and bring it back to "normal".

[dolbyman]

With access to the files..full backups should be No1 priority..no?

[AndyAlreadyInUse]

Thanks, yes, backup would be fine, but I have no storage device to backup 120 TB of data. However, once this is working again I might want to split the pool into multiple pools with something better than RAID5 and maybe something more easy to recover than LVM (I am new to all of this, so will need more time to figure out). But at first I hope to get it back up and running. I mean the RAID did not have an issue, only the cache modules.

[dolbyman]

As harsh as it sounds..with lots of data come lots of backups...without backups, you WILL lose these files at one point.

But only you know the files and their importance...

btw..any and all modes of operation on the NAS will use LVM..no way around it

[AndyAlreadyInUse]

I fully agree and understand your point. It's a company storage, company does not want to pay for backup, but is happy to pay me for trying to fix that broken thing. The data stored is kind of very important, but then again not important enough to buy a top tier EMC storage. I am happy to work on that as this is something I have never been in touch with, so yey, new and interesting stuff:)