Malware?

[mustard]

I started seeing in the event logs "[firmware] detected unauthorized changes to **firmware number** while booting."

Support logged in and attempted to remove malware. I'm not convinced they actually found anything though. Malware scanner never identified anything.

Rebooted, same warning.

Support said to carry out DOM recovery.

Carried out DOM recovery. Running latest firmware version.

Installed disks and booted.

Same warning, but referencing previous firmware version?

Updated firmware from within GUI. Ran malware detector. Nothing found. Rebooted.

Same warning, but referencing previous firmware version.

Do I have an issue, or is this a false positive? No other indication of infection.

[mustard]

Just to add, I can't find any reference to anyone else seeing similar in the event logs?

[dolbyman]

Never heard of any issues like this ..but if qnap support has no clue...I guess you are SOL

was/is your NAS WAN exposed?

[Mousetick]

Seems you're the only one to ever have experienced this issue and reported it. Can you provide more details?

I started seeing in the event logs "[firmware] detected unauthorized changes to **firmware number** while booting."

When? Was it right after a reboot or out of the blue while the NAS had been running for a while?
If this was right after a reboot, what was the reason for the reboot? Was the NAS rebooted as the result of a firmware update?

Carried out DOM recovery. Running latest firmware version.

Which firmware version? Where is that version displayed on your NAS?

Same warning, but referencing previous firmware version?

Which firmware version? Was this firmware version previously installed on the NAS, or was it never installed?

Do I have an issue, or is this a false positive? No other indication of infection.

There is definitely an issue, but not necessarily caused by malicious activity. The fact that the unauthorized changes warning references a previous version leads me to believe a firmware update may have corrupted something somehow on the disks. Your answers to the questions above can give some clues.

The message "[Firmware] Detected unauthorized changes to <firmware-version> while booting." is emitted by the shell script /etc/init.d/cs_fw_verify.sh which is executed during boot. If you're familiar with Bash shell scripting, you can examine the contents of this file to figure out what it's doing. Basically, it verifies the digital signatures of the Linux boot image and firmware tarball packages stored on the DOM's 2nd partition.

This script writes a log /tmp/check_fw_code_signing.log which you can examine. The format is <status code>:<status code>:/path/to/file/being/verified. The first status code indicates the state of the digital certificate used to digitally sign the packages (0 is success). The second status code indicates verification success or failure (0 is success).

You should be able to run this script on demand from the command line (I have not tried it), but it's not necessary if you just want to look at the log file (the one generated during boot remains in /tmp):

Code: Select all

# /etc/init.d/cs_fw_verify.sh verify_fw_code_signing

This is what the log file looks like on a healthy system:

Code: Select all

$ cat /tmp/check_fw_code_signing.log
FW code signing verification start
0:0:/tmp/.verify_fw_code_signing/boot/bzImage
0:0:/tmp/.verify_fw_code_signing/boot/initrd.boot
0:0:/tmp/.verify_fw_code_signing/boot/qpkg.tar
0:0:/tmp/.verify_fw_code_signing/boot/rootfs2.bz
0:0:/tmp/.verify_fw_code_signing/boot/rootfs_ext.tgz
FW code signing verification end

The digital signature validation script relies on background services named cs_daemon and cs_qdaemon, both started during boot by the script /etc/init.d/cs_daemon.sh . This script shows the existence of various digital signature databases, /etc/config/nas_sign_fw.db for example, but I can't establish a clear relationship between these databases and the firmware validation process and I can't tell if or how they're used.

The firmware verification script /etc/init.d/cs_fw_verify.sh uses the contents of the file /etc/config/uLinux.conf to display the firmware version number in the "unauthorized changes" messages. This file is stored on disk, not on the DOM.

Top of /etc/config/uLinux.conf on a system running 4.5.4.1715 Build 20210630:

Code: Select all

[System]
...
Version = 4.5.4
Build Number = 20210630
Number = 1715
...

On a healthy system, all these files in the boot directory on the DOM's 2nd partition, should have the same the same modification date/time, which is that of the firmware installation date/time. My system is running 4.5.4.1715 Build 20210630:

Code: Select all

-rw-r--r-- 1 admin administrators   7566032 Jul  5 11:09 bzImage
-rw-r--r-- 1 admin administrators        75 Jul  5 11:09 bzImage.cksum
-rw-r--r-- 1 admin administrators      2410 Jul  5 11:09 bzImage.sign
-rw-r--r-- 1 admin administrators  20396264 Jul  5 11:09 initrd.boot
-rw-r--r-- 1 admin administrators        81 Jul  5 11:09 initrd.boot.cksum
-rw-r--r-- 1 admin administrators      2410 Jul  5 11:09 initrd.boot.sign
-rw-r--r-- 1 admin administrators 120657920 Jul  5 11:09 qpkg.tar
-rw-r--r-- 1 admin administrators        78 Jul  5 11:09 qpkg.tar.cksum
-rw-r--r-- 1 admin administrators      2410 Jul  5 11:09 qpkg.tar.sign
-rw-r--r-- 1 admin administrators  52719124 Jul  5 11:09 rootfs2.bz
-rw-r--r-- 1 admin administrators        80 Jul  5 11:09 rootfs2.bz.cksum
-rw-r--r-- 1 admin administrators      2410 Jul  5 11:09 rootfs2.bz.sign
-rw-r--r-- 1 admin administrators  13059198 Jul  5 11:09 rootfs_ext.tgz
-rw-r--r-- 1 admin administrators        83 Jul  5 11:09 rootfs_ext.tgz.cksum
-rw-r--r-- 1 admin administrators      2410 Jul  5 11:09 rootfs_ext.tgz.sign

These files on the system should also have the same modification date/time as the firmware on the DOM:

Code: Select all

-rw-r--r-- 1 admin administrators    5120 Jul  5 11:16 /etc/config/nas_sign_fw.db
-rw-r--r-- 1 admin administrators 4770816 Jul  5 11:19 /etc/config/nas_sign_qpkg.db

You can use all this information to inspect your NAS and compare the results. Report back with your findings if you like.

[mustard]

That's really helpful, thank you. I'll respond in full when I get a chance. I've also managed to get the issue escalated with QNAP and I'm in the process of arranging some time for a different support engineer to run a remote session with me.

[mustard]

Warning has now disappeared. The only change in-between reboots was AI Core getting updated. After escalation to a really helpful Qnap engineer, they've not been able to find an issue. First engineer flat-out told me my NAS was unrecoverable, deeply compromised by malware, needed to be RMAd (320USD) and that the mainboard/DOM needed replacing. Go figure.

[snarf007]

Hi, had the same issue on a TS-251+
After reading that an update of IACore fix the issue if installed and removed IACore.
Then a reboot and the issue was gone.

[ganxiyun]

I had the same issue, thanks @Mousetick, I found it's caused by the qnap.tar.

Go to App Center, found the Apps installed with Signature issue (Certification expired), remove it, and let reboot. The warning is gone.