Azure Backup fails due to UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuation

[smithimage]

Hi all!

When I try to use the new Azure Backup feature it always stop with an error pointing at some sort of encoding issue (se log statement below).

Does anyone now a solution to this?

Tnx in advance!


2016-01-09 10:01:25,585 PID:15058 backupscannerworkerthread.py:814 cc ERROR not handled exception
Traceback (most recent call last):
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 723, in run
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 508, in check_local_files
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 452, in _check_local_files_for_each
File "./package/src/shared/bin/diskdataprocessor.py", line 181, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 561, in walk
File "./package/src/shared/python-lib/scandir.py", line 536, in scandir
File "/share/MD0_DATA/.qpkg/AzureStorage/python/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuation byte
2016-01-09 10:01:26,005 PID:15058 backupscannerworkerthread.py:823 cc DEBUG BackupScannerWorkerThread exit: (CloudConnectorError(...), -1, 'of unknown error. Please try again')
2016-01-09 10:01:26,062 PID:15058 statistics.py:356 cc DEBUG [backup] update sub_state: 'Scanning' -> 'To be Stopped'
2016-01-09 10:01:26,352 PID:15058 connectorworkerthread.py:147 cc DEBUG stop ConnectorWorkerThread because (ThreadTerminateError(...), -1005, 'the thread is stopped')
2016-01-09 10:01:26,357 PID:15058 connectorworkerthread.py:157 cc DEBUG ConnectorWorkerThread exit: False
2016-01-09 10:01:30,093 PID:15058 qlog.py:73 cc DEBUG log_tool: Appending a log to database...
Done. Return value is 0

[ikk01]

This is caused by an old "scandir" version that is included with the Azure, S3, Glacier, etc. backup QPKG provided by QNAP.

This has been fixed a couple years ago: https://github.com/benhoyt/scandir/issues/42

Is there any chance an official representative of QNAP can provide any information here. Can you give this information to the developers of the QNAP Azure/Glacier/... plugin, and have them add an updated version of "scandir" in the python-lib folder of the plugin?

This happens in Glacier/Azue latest version 1.1.379.

I am sure this is happening to many users.

[smithimage]

I suspected that the problem had something to do with a python lb. Thank you for finding out which one..
Is there anyway we as users could upgrade the scandir version our selves?

[ikk01]

I didn't find a way to upgrade scandir. However, I found the solution in fixing the broken folder/filenames on my shares. See here: http://forum.qnap.com/viewtopic.php?t=11413#p125565

Basically this utility converts the folder/filenames into UTF-8 which is readable by QNAP (at least in my case).

You will need "Entware-NG" (a package manager) first: https://github.com/Entware-ng/Entware-n ... n-QNAP-NAS

Then in /root/ or somewhere else (log in via SSH), you need to install perl and convmv:

ipkg install perl
wget http://www.j3e.de/linux/convmv/convmv-1.14.tar.gz
tar xzvf convmv-1.14.tar.gz
cd convmv*
sed -i -e 's/usr/opt/1p' convmv
./convmv

Now the first times you run convmv some errors will show up that some perl libraries are missing. Look at the error messages and install these libraries using "ipkg install perl-whatever". Repeat until ./convmv shows the help.

Before you proceed, I would recommend taking a backup of the files/folders, as real folders/files will be renamed.

Then you can do this:

./convmv -r "/share/YOURSHARE/" -f iso-8859-1 -t utf8

This is a "test mode" and will first just show what folders/files would be renamed, i.e. where the utility finds "wrong" characters in the filename.

Then if you are confident you want to rename the files/folders you can run it using --notest:

./convmv -r "/share/YOURSHARE/" -f iso-8859-1 -t utf8 --notest

It will then rename the files so they can be read by QNAP Glacier/S3 Plus/Azure backup packages. It works for me now.

Good luck!

[smithimage]

Thank you for the tip..
I will try it..

[Don]

This is caused by an old "scandir" version that is included with the Azure, S3, Glacier, etc. backup QPKG provided by QNAP.

This has been fixed a couple years ago: https://github.com/benhoyt/scandir/issues/42

Is there any chance an official representative of QNAP can provide any information here. Can you give this information to the developers of the QNAP Azure/Glacier/... plugin, and have them add an updated version of "scandir" in the python-lib folder of the plugin?

This happens in Glacier/Azue latest version 1.1.379.

I am sure this is happening to many users.

This is a community forum so to get QNAP engaged you need to open a ticket with the help desk.

[QNAP_EmmaS]

Hi,

I am the PM of CloudBackup Station.
We have found the root cause and will fixed in the next release.
After released, I will update the release version here.
Thanks a lot.

[QNAP_EmmaS]

Hello,
We had released the v1.1.387 to fixed this problem.
You can update now. Thank you.

[smithimage]

Hi again!

I just upgraded to version 1.1.387. Unfortunately i still get the same error..


2016-05-15 09:43:54,034 PID:13045 backupscannerworkerthread.py:817 cc ERROR not handled exception
Traceback (most recent call last):
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 726, in run
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 508, in check_local_files
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 452, in _check_local_files_for_each
File "./package/src/shared/bin/diskdataprocessor.py", line 181, in walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 603, in _walk
File "/share/MD0_DATA/.qpkg/AzureStorage/python/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 87: invalid continuation byte
2016-05-15 09:44:10,232 PID:13045 engine.py:404 cc ERROR CloudConnectorError: of unknown error. Please try again
2016-05-15 09:44:11,600 PID:13045 engine.py:1038 cc INFO clear watchdog and var/garbage folder
2016-05-15 09:44:12,682 PID:13045 engine.py:1055 cc INFO end backup job:2

[smithimage]

Hi again!

FYI.. I just reported a issue at the Qnap helpdesk..