Page 1 of 1

Firmware Upgrade Disaster Recovery

Posted: Sat Apr 10, 2021 3:37 am
by L54Solutions
I have a QNAP TS-412, and have been using it since 2013. Last year, I had to remove it from the site where it was used, and it went to storage for about twelve months.
It was joined to a domain, and the network ports had static IPs assigned to them.
When I powered it up, after a year in storage, I could no longer connect to the device using the IP addresses that I knew was assigned to it. The domain to which it was joined, also no longer existed. I tried absolutely everything to gain access, but I could not even figure out what the current IPs were, and if they had somehow changed in the meantime.
I downloaded and installed QFinder, but not event QFinder could find the device.

So I decided to start with a Reset. https://www.it-support-singapore.com/it ... -function/
The 3 second reset allowed the device to get an IP from my DHCP server. Now QFinder found it, but because the drives were not plugged in, I could not access the web interface to log in.
So I decided to plug the drives in. Still no luck. So I decided to use QFinder to upgrade the Firmware, but QNAP, in their infinate wisdom, only has the firmwares released during 2019-2020 on their website. There is no firmware that predates that, and absolutely nowhere to download it. If only somebody at QNAP had the foresight to keep the old firmware versions in an archive somewhere.

So I downloaded the latest firmware, and used QFinder to apply it. It failed. I restarted the device, and once again, it's not accessible. I shut it down, unplug the drives, and restart. Still nothing. I do a three second reset again, and QFinder finds it, but now it shows that the firmware is version 4.0.1, the original firmware.

I go through this process probably four more times, each time, hoping that the upgrade will work. But it fails each time.

In pure frustration, I do the 10 second reset. Now the device does not start up at all. All the drive and the status light remains red. So I find this link https://wiki.qnap.com/wiki/Firmware_Recovery and I follow the VMare Player route. Once the update is complete, I restart the device, and once again it is not available. Great!!!
I do another 3 second reset and try the firmware update again. Nothing....the thing is as dead as a door nail. While going through all the menu items in QFinder, I notice that SSH is available. I try the SSH connect and it fails....no surprise there. So I search the internet, and find a few article talking about SSH connections using Putty rather than QFinder's SSH client.

I download Putty, and do the three second reset again. Qfinder finds the device and I use Putty to connect to it using admin/admin as the login. Now I have access to the internal system drive in the device. I spend a whole afternoon trying to figure out how to update the firmware using SSH, and I find this link https://www.it-support-singapore.com/it ... -recovery/. I try that, but after a reboot, the device is dead again.

3 second reset -> QFinder -> Putty : I'm in again. I do the first few steps of https://www.it-support-singapore.com/it ... -recovery/ again, but stop short of the reboot. I use the # df command in SSH, and notice that the drives were mounted, but I cannot see if the data on them are still in tack. I go back to QFinder, and double click the device. The web page opens, but tells me that the device has not yet been initialized, and that the firmware needs to be loaded. I don't read the message properly, and miss the part where it says that ALL THE DRIVES WILL BE INITIALIZED as part of the firmware update. Guess what, the firmware update fails again, but now....oh now....I have four freshly initialized drives, with probably no data left on them.

At this point I decide that I have been beaten, and decide to just try and get the device running again, so that I can reformat the drives and setup RAID5 again.

In the mean time, I found this link https://wiki.qnap.com/wiki/Manually_Updating_Firmware which describes how to manually update the firmware, but it relies on the fact that I can copy the firmware to one of the shares on the device. How freaking stupid is that!!!?? Why on earth would I be at the point where I feel the need to do a manual firmware update, if I still had access to the shares on the device???

So I copy the firmware IMG file to a USB stick and plug it into the device......but for some reason, I cannot get it to mount. So I go searching on how to copy files using Putty, and I discover PSCP on the Putty web site. I download PSCP, and install it. I have the required firmware image file sitting on my computers hard drive. I decide to try and copy the file to the device, so in the end, here are the steps that I followed to manually install the latest firmware on a bricked QNAP:

From <link removed>
(1) Please make sure there is /mnt/HDA_ROOT/update folder.
If it does not exist, create it using

Code: Select all

# mkdir /mnt/HDA_ROOT/update
(2) Confirm no /mnt/update file or folder.
If it exists, you need to delete it using:

Code: Select all

# rm -rf /mnt/update
if you get a message saying:
“rm: unable to remove ‘/mnt/update’: Device or resource busy”, unmount the folder and try the rm line above again. To unmount use:

Code: Select all

# umount /mnt/update
Then I used PSCP on my local computer to copy the IMG file to the device using this command:

Code: Select all

pscp -scp -P 22 C:\IMG\TS-412_20201006-4.3.3.1432.img admin@NASCF91B8:/mnt/HDA_ROOT/update/TS-412_20201006-4.3.3.1432.img
Once the file was copied to the device, I went back to Putty and ran this command:

Code: Select all

/etc/init.d/update.sh /mnt/HDA_ROOT/update/TS-412_20201006-4.3.3.1432.img
The firmware installed successfully, but complained at the end about not being able to make a backup. I ignored that, and restarted the device with the hard drives plugged in. I figured that the drives were initialized anyway, so there was now no more harm that can come to them.

The device took forever to boot, and as luck would have it, I left it over night to do whatever it needed to do.

When I looked at it again the next morning, QFinder showed that it was alive and well, and I could access the web interface. I immediately logged in, and went to the File Manager. My data that was on the drives were gone. I went to the Control Panel and took and look at the storage. Storage showed me that the drive size was 8Tb, but it also showed me that there was 3Tb worth of something on the drives. So I figured that there must be something there. I logged into the device with Putty again, and checked the mounted drives. The drives were all there, and using the # ls command, I finally established that my data was ALL STILL THERE!!! Absolutely amazing.

But why could I not see them in the File Manager. I found a link on the internet that spoke about just recreating the shares again, and pointing them to the same drives. I did that, and to my utter amazement, all my data showed up again in the File Manager.

So even after resetting the device multiple times, initializing the drives as part of the failed firmware updates, and manually updating the firmware, my data survived the journey, so if anybody else ever find themselves in this position, this is how I fixed it.

Re: Firmware Upgrade Disaster Recovery

Posted: Sat Apr 10, 2021 4:12 am
by OneCD
It's great you were able to fix this, but these fixes are quite well-known here.

You've had a QNAP NAS since 2013 but just posted here for the first time. You might find it beneficial to participate more in forum topics to get yourself up-to-speed. :geek:

Re: Firmware Upgrade Disaster Recovery

Posted: Sat Apr 10, 2021 4:28 am
by dolbyman
@OneCD , can you remove the qnap support link further UP .. the forum screws up the link and it currently spews malware again

Re: Firmware Upgrade Disaster Recovery

Posted: Sat Apr 10, 2021 4:32 am
by OneCD
Thanks @dm, I must have skipped-over that one. Image

QNAP should really be redirecting to a page they control. :evil:

Re: Firmware Upgrade Disaster Recovery

Posted: Sat Apr 10, 2021 4:36 am
by dolbyman
OneCD wrote: Sat Apr 10, 2021 4:32 am QNAP should really be redirecting to a page they control. :evil:
Worst part is that they give out links to that site themselves

viewtopic.php?f=185&t=156628&p=785148 (you could cleanup that topic to) .. maybe via shortening ?

Complete nonsense

Re: Firmware Upgrade Disaster Recovery

Posted: Sat Apr 10, 2021 4:38 am
by OneCD
At-least it's only a holding-page at the moment. ;)

Re: Firmware Upgrade Disaster Recovery

Posted: Sat Apr 10, 2021 4:53 am
by dolbyman
I checked it in a sandbox and it was flagged for malware .. maybe a holding page or malware depending on browser and/location

Re: Firmware Upgrade Disaster Recovery

Posted: Sat Apr 10, 2021 4:55 am
by OneCD
Ah, good point. Image