[OMG] QNAP Forced Remote Firmware 4.1.1 Shellshock Upgrade?

[disappointed_user]

WOW.

I *CANNOT* believe that QNAP would be so shortsighted as to forcibly upgrade user's firmware remotely, without warning or notification, even in relation to a security alert such as shellshock - however, as it happens, it seems this happened to me this morning...

I have a QNAP TS-559+ Pro, which - until this morning (22nd October 2014 / 2014-10-22) - was running a firmware version prior to 4.1.1 and I had intentionally avoided upgrading the firmware for many, many months.

The power to my house was cut this morning, and upon boot up when power was restored, I received email notifications from my device regarding improper shutdown, scanning of the [Media Library], and indications that both RAID array filesystems were not clean and required checking.

The firmware running now (Version 4.1.1, firmware dated "2014/10/03" - which I imagine is the date the "remote, forced firmware auto-upgrade" actually happened, and I will be scouring my system logs for evidence of this, which I consider to be a complete violation of my system, and a system compromise by QNAP of my device/server/data, and may even attract criminal penalties in various jurisdictions!) is significantly newer and different to the previous firmware version I was running.

Upon first login to my QNAP after the power outage this afternoon, via the SSL admin page, I received a 502 Gateway Proxy Error - which caused me to fret immediately that corruption of the core QNAP firmware/system had occurred, and that the admin login page had been affected. After a deep breath to calm my nerves, upon pressing refresh in my browser, a NEW, COMPLETELY DIFFERENT login screen presented itself, with logos top-right regarding "MyQNAPCloud", yet the IP address was still my internal network's IP address.

Gingerly logging in with my sensive admin/root credentials to this potentially compromised, potentially security MITM attack on my device masquerading as my internal IP, my login was successful, I was presented with the new firmware, a completely different style "desktop" or "dashboard" interface, a notification regarding firmware upgrades because of the shellshock vulnerability, and details of My QNAP Cloud service, etc.

The very fact that this is possible is beyond belief, and I will be expecting a full response from QNAP directly to my email, and/or on this forum, with an apology, and/or specific details of whichever user agreement could possibly consent to allowing this remote forced firmware upgrade to occur, and if so, evidence of the exact date on which I personally agreed to QNAP's ability to remotely force upgrades.

Not only is this a complete intrustion of the trust of users and control of their own systems (e.g. I had SPECIFICALLY not upgraded the firmware for some months, despite being fully aware of the potential shellshock vulnerabilities and how they can be exploited) but opens the door to potential DATA CORRUPTION caused by QNAP'S COMPLETE LACK OF FORETHOUGHT in FORCING upgrades remotely.

INDEED - there may be many other users like myself, who - unbeknownst to them, have had their firmware upgrade pre-downloaded/pre-applied/pre-scheduled to apply upon the next boot, and who have not had the opportunity to make a full, recent backup. There may be others in the same boat as I, who may suffer an unplanned power outage, and result in data corruption, possibly made worse by QNAP's forced remote auto-upgrade of firmware, opening QNAP up to legal action in the event of cataclysmic data failure in business critical operations.

SO BE WARNED:

FOR THOSE WHO HAVE NOT REBOOTED THEIR QNAP SINCE 27th OF SEPTEMBER 2014 - BACKUP YOUR QNAP DATA NOW!

I have Googled for other reports/complaints/users reports of this, but have found none - so I expect this is just a timebomb waiting to go off, as I DEFINITELY have not upgraded my firmware automatically, and now have a potentially inconsistent state, and no obvious downgrade path, if firmware downgrade is indeed possible.

As per:

http://geekbeat.tv/qnap-releases-qts-fi ... erability/ (dated: September 29, 2014)

"QTS 4.1.1 Build 0927 is now available for update directly on the Turbo NAS management interface (QTS) and on QNAP’s official download site (http://www.qnap.com/download) for the following Turbo NAS models..."

QNAP - PLEASE: Think before you act.

Regards,

One VERY Disappointed QNAP Owner.
(QNAP TS-559+ Pro, Running Firmware 4.1.1)

[Briain]

Hi

Welcome to the Qnap community forum.

That's a somewhat surprising (and unlikely) thing to read about! It's the first time I've heard of anything like that happening, and I access this forum several times per day. Just out of interest, was your Qnap exposed to the Internet and if so, I wonder if someone (other than Qnap) used the Shellshock vulnerability to access it and kick off the live update script via the shell, for example (which is at least a little less nefarious than most of the other things that they could have done to it); just an interesting thought to ponder before accusing Qnap of doing it. In fact, I am so surprised by your findings that I'm now even wondering if live update has somehow been kicked off after the power outage crash corrupting something; I'm no expert, but that sounds a little far fetched, to me (I'm clutching at straws, but that's how surprised I am to hear of all this).

It's all very, very intriguing and I wonder if you could please post more details on which services were exposed (and ports opened in your router) and which firmware version you were running before this update, and finally, please do let us all know what you find in the logs; I'd imagine there will be many folks eager to know such things. It will certainly be interesting to hear what the regular posters have to say about all this, as to my knowledge, there have - as yet - been no other indications of any similar such events.

Oh well, to look on the brighter side, at least it's now patched and considering the other (RAID related) issues, I think you should seriously ponder the merits of getting a UPS (money very well spent, IMHO).

Bri

[schumaku]

When you have stopped shouting, and calmed down...continue reading:

QNAP does not have an ability to do what you define as "remote, forced firmware auto-upgrade". If they would so something the like ... why do you think they still have the warning note up on all the QNAP Web pages? Figure: There would be simply no need for these nag messages.

Screen Shot 2014-10-22 at 10.40.05.png

So in absence of such an implementation (confirmed by my QNAP connections)... I have some isolated legacy systems up and running on v3.8.4 build0816 - a TS-459 Pro and a TS-509 Pro. Rebooted the both several times for you - can't see any time bomb - except of the to me mitigate-able risk of permitting these systems Internet access.

With the live update check enabled on the firmware update page - the firmware update notification is shown on each login (resp on entering the admin control panel on v3). Disabling the update check does also disable the notification pop-up:

Screen Shot 2014-10-22 at 11.18.45.png

With the live update check enabled, every time when accessing the NAS admin panel, the update information is shown:

Screen Shot 2014-10-22 at 11.17.23.png

Here a simple "Enter" (potentially queued on the client or browser side) does trigger the update - because of the "default" action is [Ok].

...and I will be expecting a full response from QNAP directly to my email, and/or on this forum...

Once more for the records, this is a community forum - not a formal challenge-QNAP-and-get-response channel.

If you desperately want to revert to a v3.x build - feel free to do so. (edit: Let me know if you want me to check if a downgrade is possible NAS models similar to yours.)

Regards,
-Kurt.

[Briain]

Ah, the demon of the defaults. That sounds like a highly plausible scenario. I've not enabled that feature and was thus unaware of that pop up message (with focus on the 'go' button).

All the best
Bri

[disappointed_user]

Hi Bri / schumaku,

Thanks for your prompt replies, and for the critical thinking applied to how this could have happened.

Firstly, some security background:

Access to my QNAP web login interface is forced SSL, which I'm fairly sure uses the QNAP self-signed certificate (not commercial), which I have a permanent exception for in my browser, so had it changed, or if I use a different browser, I would have received an SSL warning. My QNAP admin password is also extremely long, not used elsewhere, uses a combination of uppercase/lowercase/numeric/non-alphanumeric, etc. and has security alerts on to email when any IP could potentially bruteforce the account, and are automatically blacklisted after a conservative number of attempts. My WIFI uses WPA2 encryption, with an extremely long, complicated password, also not used elsewhere, and I monitor my network using ARPwatch for new station IDs/MAC addresses in the event someone breaks into my WIFI and onto the network segment where the QNAP resides. (More on this soon.)

Brian: No, my QNAP is NOT exposed to the internet, and runs no active services apart from SSH and CIFS/Samba.
Brian: I find it (extremely) unlikely that someone who managed to hack my network (somehow without my knowledge) would have gratuitously upgraded my firmware for me. (But, stranger things have been known to happen, like the Blaster worm wars in 2003 when the Blaster Worm was hacking Windows hosts via RPC, and the other worm was hacking into hosts, fixing the vulnerability, and terminating the Blaster Worm.)
Brian: Thank you for responding with positive criticism, and suggestions, even while clutching at straws, and yes, the benefit is no shellshock vulnerability, I'm just hoping my RAID filesystem checks come up clean!

schumaku: I wasn't shouting (OK, I was typing one sentence - out of how many? - in CAPS, but that was to draw attention to the message for other users who may potentially be in a worse situation than I, as a potentially legitimate warning) and yes, I did Google for other reports, I admit with both of you that it seems unlikely - seriously, who would be so shortsighted to do that? - but someone is always "first", and given many QNAP devices don't get rebooted that often, and the datestamp of the firmware is only 1 week since the release of the 4.1.1 firmware, it doesn't seem like too far-fetched.

schumaku: Thank you for the screenshots showing the "Live Update" feature, and the "We recommend you upgrade", however, the "We recommend you upgrade" screenshot looks very much to me from the style and surrounding features to be a later firmware, e.g. 4.1.1 or perhaps at least a 4.x firmware - I believe I may have been running firmware 3.5.9 but that's from vague memory (I can't immediately find the firmware file I downloaded, which I will have somewhere to confirm, but will get back to you.)

The example re the "live update" feature, and the "OK" default, seems odd to me - definitely a bad design decision if this is indeed the case. I believe I did have the "Live Update" feature enabled, but each time I logged in, it would ask me if I wanted to, and each and every time - paranoid about data corruption on my multiple TB of precious, precious data - attempt at humour in case you miss it - I decline; I hadn't actually known the default was "OK", and if it has, *MAYBE* I've entered "OK", but if so, it would have been a VERY accidental enter/space bar key when logging in, and I am always VERY careful with firmware upgrades, as if you haven't guessed, I'm an I.T. professional.

schumaku: Two points I have for you are:

1. Your "assertion" that QNAP does not have the ability to force auto-upgrade:

Could you please clarify how you know this? Do you have some visibility into the source code?
Are you an ex/current QNAP employee? As you seem to disclaim from the forum being a community one that you have any authority as a QNAP employee, how can you possibly know that there is not an "emergency" upgrade value that can be set when Live Update checks in?

Have you simply done a netstat? Over what time period? Did you tcpdump the packets over a reasonable time period and analyse the unencrypted HTTP requests sent to the QNAP live-update server? Did the absence of anything saying "EMERGENCY FORCE UPDATE", however, encoded in the HTTP request, over the time period could have monitored you monitored (no more than 1 hour and 2 minutes) allow you to confirm that the absence of it means it can't be done?

I'm sorry, but whether you think I'm shouting or not, you simply do not have the credentials or the authority to assert what you have asserted - I will check my logs until I find something, and if (when?) I find I clicked the "OK" to the Live Update prompt, I will happily eat humble pie, respond on this forum, and apologise for this rant (yes, I'll admit it was a rant).

2. Your other legacy systems might not have access to the internet and/or the Live Update enabled?

Are you confirming that you:
(a) restrict your legacy QNAP devices from accessing the internet?
(b) don't have Live Update enabled?
I would be interested if you allowed/enabled this, waited a week, then rebooted, what effect that might have.

The question is, would you put your money where your mouth is?

And, for both of you/the general community, if I wanted to look for evidence of firmware upgrades/historical firmware versions/decisions made by the system, are there any magical log files on the file system (other than commonly used Linux/UNIX/BSD logs and conventional log file locations, or even SQLLITE3/SQL databases with database record log entries) that I might use to confirm/deny any of these issues?

I'll leave you with one final question:

If you click "OK" and perform a firmware upgrade, does the system itself not need to immediately reboot, apply the firmware upgrade, and boot into the new firmware immediately?

Have a think about that, because I've done a firmware upgrade on my QNAP, and I'm fairly certain - like all other firmware upgrades I've done in my life, on 8086/386/486/Pentium/HP Blade Servers/Network WAPs/Cisco Switches/Juniper and F5 Load Balancers/and Cisco UCS hardware firmware upgrades, that firmware doesn't just (Read: VERY, VERY, VERY RARELY DOESN'T) get "updated" while you're running - say, in the Windows OS - it's usually a boot disk or some other boot mechanism that is loaded by the BIOS of the system as a part of reading the MBR, or by some other magic that can't actually happen while the system is on, because if firmware modifications don't happen correctly, you often brick your device.

I might have sounded angry, but don't equate angry with incompetent, or a n00b.

Fact is, I was angry, and I've calmed down somewhat now, but will get to the bottom of it, even if I have to admit it's my fault by clicking OK almost 2 weeks ago, and the firmware only upgrading on boot, but that seems even less likely than someone hacking my network to upgrade it for me, no?

Cheers!

[Toxic17]

raise a ticket with QNAP. they will not reply here or even look into your issue. link is in my Signature...

[schumaku]

Some interesting topics in your post I'd like to come back on - don't want to appear ignorant, have some other hobbies the next few hours. So please allow me some time before a reply will follow.

Regards,
-Kurt.

[TheOGTonyG]

QNAP is short-sighted in many ways. This is not one of them.

They do not have the ability to remotely-upgrade the firmware of ANY NAS without significant user action, much less one that is properly-secured.

I'd suggest you apply some of the patience obviously lacking in your post to your future interactions with your storage hardware. Next time, don't get so "clicky" or bang the keys when an important notification pops up in the interface.

[andand]

Interesting theory disappointed_user.

Have you been able to confirm your data is indeed intact at this point?

If you look at the system logs assuming you can access them you should be able to find the log entry for [Firmware Upgrade] with a source IP and user attached. It will also show one entry as System from 127.0.0.1 when it actually starts applying it. Check out the time difference between that entry and the previous [Firmware Upgrade] entry and you can tell what the delay between the initiation of the firmware upgrade is, and when it was actually applied. Does it match the timing of your power outage?

I hesitate to make absolute statements in the world of IT unless I have all of the facts. These forums are full of folks making diagnosis' with only half of the evidence. As detailed as your posts are, the only way to find a firm answer is look at the whole picture. Sounds like you know what you're doing, so please keep us up to date on the outcome. If QNAP is retaining a backdoor into our systems, we all want to know about it. As you suggest, maybe it is time for you to run a wireshark for a day or so to look for traffic to and from qnap. Maybe you want to use a syslog server to collect the logs too.

Good luck in your hunt.

[Don]

FWIW, I have been using QNAP devices since the days of the 209 which is 6+ years ago. I have been a QNAP beta tester and forum participant for almost as long. I have never seen a forced firmware upgrade nor have I ever seen anyone report this on the forum. I would be very surprised if they could do this as I'm sure it would open them up all kinds of lawsuits as well as the accompanying bad press and presumably loss of sales. Imagine what would happen if businesses using QNAP devices found out that they had a back door into their systems. I don't think it happened but anything is possible.


Sent from my iPad using Tapatalk HD

[disappointed_user]

Toxic17: Thanks for the idea to raise a ticket with QNAP, I just might, but still looking through logs, etc.

schumaku: No worries, take your time, I dare say this is a wild goose chase, some kind of bug, or a forced upgrade. And my logs may not go back far enough.

TheOGTonyG: No comment. Stop </trolling> and go over to one of those Apple forums where you probably fit in more.

Don: I agree, I would not have (didn't until it happened!) suspected it or considered a thought so paranoid (or stupid on QNAP's behalf in respect of the legal implications) but I certainly don't doubt it's possible. It's a no brainer to program in a simple check into the QNAP firmware to parse the response from the Live Update server for an additional token, and if it exists, to upgrade the firmware without user intervention. Such a check would be so opaque without access to the source code, and only appear as something like "PUSH AX,1 JNE 80408, JMP 80699" potentially, if you were to boil it down to assembly code And not only that, it could be different depending on which QNAP device you run, and which firmware revision, so many factors to consider, and nobody on this forum with access to the firmware to back up their apparently categorical claims that "they can't do it"... And you could Wireshark/tcpdump/ssldump until the cows come home, you might not ever find the presence of the potential additional "force-uprade" token, unless you managed to catch it prior to experiencing the upgrade check!

andand: No, can't confirm yet, want to do a full backup before I run a filesystem scan (just in case a full scan corrupts anything - but I suspect it will be fine) but I have a semi-recent full backup which may have more logs on it, so don't want to overwrite them.

My logs go back as far as the 21st of October 2014 (on the live system) and I've told them to roll over into new files so I don't lose them from now on (Really? Only 10,000 log lines? Including individual file accesses? Dang! They seem to be saved in an SQLITE or an SQL database of some sort as I suspected, as there are no standard log files in the areas I've looked, and the "log rollover" files produce CSV, so clearly from a record/field system, rather than plaintext logfiles.)

The power outage happened the morning I posted, so, Wed Oct 22, 2014 5:24 pm minus about 6 hours, so I've definitely got logs prior to the outage, and there's nothing in the logs about a firmware upgrade.

I successfully logged in via the HTTP admin console the night before, Tue Oct 21, via the admin console, which is suspiciously close to the next day when I lost power, but it's confirmation that when I logged in that night, I had the old firmware, because I had the old interface.

So unless my previous version of the firmware doesn't LOG firmware upgrades (which seems VERY unlikely) then it seems I have "conclusive"(?) proof that the upgrade happened between the 21st and 22nd, somewhere between being online, then suddenly losing power, then booting up.

There are no records in the logs as you suggest regarding "[Firmware Upgrade]" with source IP nor any log entries with 127.0.0.1 when the firmware was applied.

Can you confirm my recollection with firmware upgrades on the QNAP that the firmware upgrade happens then and there, not "applies" and waits for the next reboot?

I need to go to my QNAP full backup of a few weeks/a month or so ago, and hope that it's AFTER the 3rd October (when the firmware date claims to be) and has logs going back before the firmware date.

The only possible explanations I can come up with at this point is that:

1) On boot, the Live Update feature applied the firmware upgrade / forced / auto-upgrade, OR
2) The night before, I accidentally clicked OK to the firmware upgrade (I strongly doubt it) and it upgraded, pending a reboot which just happened to occur the following morning due to a power outage (but where are the logs re: the upgrade, and are firmware upgrades immediate, or can they be "installed, pending reboot"?
3) The sudden power outage caused the [Firmware Upgrade] logs not to be written (doubtful)
4) I accidentally clicked "OK" to the Live Updated (again, strongly doubt it) back on the 3rd of October, and the firmware upgrade has been pending a reboot since then.

But most of these suppositions depend on the firmware upgrade being applied immediately, which - having done at least one of these before with my own QNAP - is my recollection; it "installs it" once you confirm, then it immediately reboots into the new firmware.

Does anyone have any more experience with firmware upgrades on the QNAP TS-559 Pro+ specifically, or any others in general about how firmware upgrades are applied?

Also - please, no trolls. If you're going to criticise and not have the creds to back it up, just go find an Apple forum to post on.

As they say: "Never argue with a fool - they'll drag you down to their level, then beat you with experience."

Thoughts, anyone?

[spamkutu]

If the Nas's default setting liveupdate on may be someone reseted machine on back side than firmware updated automaticaly that time thats only possibilty i though.