I started seeing in the event logs "[firmware] detected unauthorized changes to **firmware number** while booting."
Support logged in and attempted to remove malware. I'm not convinced they actually found anything though. Malware scanner never identified anything.
Rebooted, same warning.
Support said to carry out DOM recovery.
Carried out DOM recovery. Running latest firmware version.
Installed disks and booted.
Same warning, but referencing previous firmware version?
Updated firmware from within GUI. Ran malware detector. Nothing found. Rebooted.
Same warning, but referencing previous firmware version.
Do I have an issue, or is this a false positive? No other indication of infection.
Malware?
-
- Getting the hang of things
- Posts: 87
- Joined: Sat Jun 15, 2013 7:24 pm
Re: Malware?
Just to add, I can't find any reference to anyone else seeing similar in the event logs?
- dolbyman
- Guru
- Posts: 35273
- Joined: Sat Feb 12, 2011 2:11 am
- Location: Vancouver BC , Canada
Re: Malware?
Never heard of any issues like this ..but if qnap support has no clue...I guess you are SOL
was/is your NAS WAN exposed?
was/is your NAS WAN exposed?
-
- Experience counts
- Posts: 1081
- Joined: Thu Aug 24, 2017 10:28 pm
Re: Malware?
Seems you're the only one to ever have experienced this issue and reported it. Can you provide more details?
If this was right after a reboot, what was the reason for the reboot? Was the NAS rebooted as the result of a firmware update?
The message "[Firmware] Detected unauthorized changes to <firmware-version> while booting." is emitted by the shell script /etc/init.d/cs_fw_verify.sh which is executed during boot. If you're familiar with Bash shell scripting, you can examine the contents of this file to figure out what it's doing. Basically, it verifies the digital signatures of the Linux boot image and firmware tarball packages stored on the DOM's 2nd partition.
This script writes a log /tmp/check_fw_code_signing.log which you can examine. The format is <status code>:<status code>:/path/to/file/being/verified. The first status code indicates the state of the digital certificate used to digitally sign the packages (0 is success). The second status code indicates verification success or failure (0 is success).
You should be able to run this script on demand from the command line (I have not tried it), but it's not necessary if you just want to look at the log file (the one generated during boot remains in /tmp):
This is what the log file looks like on a healthy system:
The digital signature validation script relies on background services named cs_daemon and cs_qdaemon, both started during boot by the script /etc/init.d/cs_daemon.sh . This script shows the existence of various digital signature databases, /etc/config/nas_sign_fw.db for example, but I can't establish a clear relationship between these databases and the firmware validation process and I can't tell if or how they're used.
The firmware verification script /etc/init.d/cs_fw_verify.sh uses the contents of the file /etc/config/uLinux.conf to display the firmware version number in the "unauthorized changes" messages. This file is stored on disk, not on the DOM.
Top of /etc/config/uLinux.conf on a system running 4.5.4.1715 Build 20210630:
On a healthy system, all these files in the boot directory on the DOM's 2nd partition, should have the same the same modification date/time, which is that of the firmware installation date/time. My system is running 4.5.4.1715 Build 20210630:
These files on the system should also have the same modification date/time as the firmware on the DOM:
You can use all this information to inspect your NAS and compare the results. Report back with your findings if you like.
When? Was it right after a reboot or out of the blue while the NAS had been running for a while?
If this was right after a reboot, what was the reason for the reboot? Was the NAS rebooted as the result of a firmware update?
Which firmware version? Where is that version displayed on your NAS?Carried out DOM recovery. Running latest firmware version.
Which firmware version? Was this firmware version previously installed on the NAS, or was it never installed?Same warning, but referencing previous firmware version?
There is definitely an issue, but not necessarily caused by malicious activity. The fact that the unauthorized changes warning references a previous version leads me to believe a firmware update may have corrupted something somehow on the disks. Your answers to the questions above can give some clues.Do I have an issue, or is this a false positive? No other indication of infection.
The message "[Firmware] Detected unauthorized changes to <firmware-version> while booting." is emitted by the shell script /etc/init.d/cs_fw_verify.sh which is executed during boot. If you're familiar with Bash shell scripting, you can examine the contents of this file to figure out what it's doing. Basically, it verifies the digital signatures of the Linux boot image and firmware tarball packages stored on the DOM's 2nd partition.
This script writes a log /tmp/check_fw_code_signing.log which you can examine. The format is <status code>:<status code>:/path/to/file/being/verified. The first status code indicates the state of the digital certificate used to digitally sign the packages (0 is success). The second status code indicates verification success or failure (0 is success).
You should be able to run this script on demand from the command line (I have not tried it), but it's not necessary if you just want to look at the log file (the one generated during boot remains in /tmp):
Code: Select all
# /etc/init.d/cs_fw_verify.sh verify_fw_code_signing
Code: Select all
$ cat /tmp/check_fw_code_signing.log
FW code signing verification start
0:0:/tmp/.verify_fw_code_signing/boot/bzImage
0:0:/tmp/.verify_fw_code_signing/boot/initrd.boot
0:0:/tmp/.verify_fw_code_signing/boot/qpkg.tar
0:0:/tmp/.verify_fw_code_signing/boot/rootfs2.bz
0:0:/tmp/.verify_fw_code_signing/boot/rootfs_ext.tgz
FW code signing verification end
The firmware verification script /etc/init.d/cs_fw_verify.sh uses the contents of the file /etc/config/uLinux.conf to display the firmware version number in the "unauthorized changes" messages. This file is stored on disk, not on the DOM.
Top of /etc/config/uLinux.conf on a system running 4.5.4.1715 Build 20210630:
Code: Select all
[System]
...
Version = 4.5.4
Build Number = 20210630
Number = 1715
...
Code: Select all
-rw-r--r-- 1 admin administrators 7566032 Jul 5 11:09 bzImage
-rw-r--r-- 1 admin administrators 75 Jul 5 11:09 bzImage.cksum
-rw-r--r-- 1 admin administrators 2410 Jul 5 11:09 bzImage.sign
-rw-r--r-- 1 admin administrators 20396264 Jul 5 11:09 initrd.boot
-rw-r--r-- 1 admin administrators 81 Jul 5 11:09 initrd.boot.cksum
-rw-r--r-- 1 admin administrators 2410 Jul 5 11:09 initrd.boot.sign
-rw-r--r-- 1 admin administrators 120657920 Jul 5 11:09 qpkg.tar
-rw-r--r-- 1 admin administrators 78 Jul 5 11:09 qpkg.tar.cksum
-rw-r--r-- 1 admin administrators 2410 Jul 5 11:09 qpkg.tar.sign
-rw-r--r-- 1 admin administrators 52719124 Jul 5 11:09 rootfs2.bz
-rw-r--r-- 1 admin administrators 80 Jul 5 11:09 rootfs2.bz.cksum
-rw-r--r-- 1 admin administrators 2410 Jul 5 11:09 rootfs2.bz.sign
-rw-r--r-- 1 admin administrators 13059198 Jul 5 11:09 rootfs_ext.tgz
-rw-r--r-- 1 admin administrators 83 Jul 5 11:09 rootfs_ext.tgz.cksum
-rw-r--r-- 1 admin administrators 2410 Jul 5 11:09 rootfs_ext.tgz.sign
Code: Select all
-rw-r--r-- 1 admin administrators 5120 Jul 5 11:16 /etc/config/nas_sign_fw.db
-rw-r--r-- 1 admin administrators 4770816 Jul 5 11:19 /etc/config/nas_sign_qpkg.db
-
- Getting the hang of things
- Posts: 87
- Joined: Sat Jun 15, 2013 7:24 pm
Re: Malware?
That's really helpful, thank you. I'll respond in full when I get a chance. I've also managed to get the issue escalated with QNAP and I'm in the process of arranging some time for a different support engineer to run a remote session with me.
-
- Getting the hang of things
- Posts: 87
- Joined: Sat Jun 15, 2013 7:24 pm
Re: Malware?
Warning has now disappeared. The only change in-between reboots was AI Core getting updated. After escalation to a really helpful Qnap engineer, they've not been able to find an issue. First engineer flat-out told me my NAS was unrecoverable, deeply compromised by malware, needed to be RMAd (320USD) and that the mainboard/DOM needed replacing. Go figure.
-
- Starting out
- Posts: 40
- Joined: Sun Jan 16, 2011 6:47 am
Re: Malware?
Hi, had the same issue on a TS-251+
After reading that an update of IACore fix the issue if installed and removed IACore.
Then a reboot and the issue was gone.
After reading that an update of IACore fix the issue if installed and removed IACore.
Then a reboot and the issue was gone.
-
- New here
- Posts: 5
- Joined: Thu Sep 15, 2022 12:12 pm
Re: Malware?
I had the same issue, thanks @Mousetick, I found it's caused by the qnap.tar.
Go to App Center, found the Apps installed with Signature issue (Certification expired), remove it, and let reboot. The warning is gone.
Go to App Center, found the Apps installed with Signature issue (Certification expired), remove it, and let reboot. The warning is gone.