Hi all!
When I try to use the new Azure Backup feature it always stop with an error pointing at some sort of encoding issue (se log statement below).
Does anyone now a solution to this?
Tnx in advance!
2016-01-09 10:01:25,585 PID:15058 backupscannerworkerthread.py:814 cc ERROR not handled exception
Traceback (most recent call last):
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 723, in run
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 508, in check_local_files
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 452, in _check_local_files_for_each
File "./package/src/shared/bin/diskdataprocessor.py", line 181, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 590, in walk
File "./package/src/shared/python-lib/scandir.py", line 561, in walk
File "./package/src/shared/python-lib/scandir.py", line 536, in scandir
File "/share/MD0_DATA/.qpkg/AzureStorage/python/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuation byte
2016-01-09 10:01:26,005 PID:15058 backupscannerworkerthread.py:823 cc DEBUG BackupScannerWorkerThread exit: (CloudConnectorError(...), -1, 'of unknown error. Please try again')
2016-01-09 10:01:26,062 PID:15058 statistics.py:356 cc DEBUG [backup] update sub_state: 'Scanning' -> 'To be Stopped'
2016-01-09 10:01:26,352 PID:15058 connectorworkerthread.py:147 cc DEBUG stop ConnectorWorkerThread because (ThreadTerminateError(...), -1005, 'the thread is stopped')
2016-01-09 10:01:26,357 PID:15058 connectorworkerthread.py:157 cc DEBUG ConnectorWorkerThread exit: False
2016-01-09 10:01:30,093 PID:15058 qlog.py:73 cc DEBUG log_tool: Appending a log to database...
Done. Return value is 0
Azure Backup fails due to UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuation
-
- New here
- Posts: 7
- Joined: Mon Apr 26, 2010 9:40 pm
-
- New here
- Posts: 2
- Joined: Tue Apr 05, 2016 11:42 pm
Re: Azure Backup fails due to UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuat
This is caused by an old "scandir" version that is included with the Azure, S3, Glacier, etc. backup QPKG provided by QNAP.
This has been fixed a couple years ago: https://github.com/benhoyt/scandir/issues/42
Is there any chance an official representative of QNAP can provide any information here. Can you give this information to the developers of the QNAP Azure/Glacier/... plugin, and have them add an updated version of "scandir" in the python-lib folder of the plugin?
This happens in Glacier/Azue latest version 1.1.379.
I am sure this is happening to many users.
This has been fixed a couple years ago: https://github.com/benhoyt/scandir/issues/42
Is there any chance an official representative of QNAP can provide any information here. Can you give this information to the developers of the QNAP Azure/Glacier/... plugin, and have them add an updated version of "scandir" in the python-lib folder of the plugin?
This happens in Glacier/Azue latest version 1.1.379.
I am sure this is happening to many users.
-
- New here
- Posts: 7
- Joined: Mon Apr 26, 2010 9:40 pm
Re: Azure Backup fails due to UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuat
I suspected that the problem had something to do with a python lb. Thank you for finding out which one..
Is there anyway we as users could upgrade the scandir version our selves?
Is there anyway we as users could upgrade the scandir version our selves?
-
- New here
- Posts: 2
- Joined: Tue Apr 05, 2016 11:42 pm
Re: Azure Backup fails due to UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuat
I didn't find a way to upgrade scandir. However, I found the solution in fixing the broken folder/filenames on my shares. See here: http://forum.qnap.com/viewtopic.php?t=11413#p125565
Basically this utility converts the folder/filenames into UTF-8 which is readable by QNAP (at least in my case).
You will need "Entware-NG" (a package manager) first: https://github.com/Entware-ng/Entware-n ... n-QNAP-NAS
Then in /root/ or somewhere else (log in via SSH), you need to install perl and convmv:
ipkg install perl
wget http://www.j3e.de/linux/convmv/convmv-1.14.tar.gz
tar xzvf convmv-1.14.tar.gz
cd convmv*
sed -i -e 's/usr/opt/1p' convmv
./convmv
Now the first times you run convmv some errors will show up that some perl libraries are missing. Look at the error messages and install these libraries using "ipkg install perl-whatever". Repeat until ./convmv shows the help.
Before you proceed, I would recommend taking a backup of the files/folders, as real folders/files will be renamed.
Then you can do this:
./convmv -r "/share/YOURSHARE/" -f iso-8859-1 -t utf8
This is a "test mode" and will first just show what folders/files would be renamed, i.e. where the utility finds "wrong" characters in the filename.
Then if you are confident you want to rename the files/folders you can run it using --notest:
./convmv -r "/share/YOURSHARE/" -f iso-8859-1 -t utf8 --notest
It will then rename the files so they can be read by QNAP Glacier/S3 Plus/Azure backup packages. It works for me now.
Good luck!
Basically this utility converts the folder/filenames into UTF-8 which is readable by QNAP (at least in my case).
You will need "Entware-NG" (a package manager) first: https://github.com/Entware-ng/Entware-n ... n-QNAP-NAS
Then in /root/ or somewhere else (log in via SSH), you need to install perl and convmv:
ipkg install perl
wget http://www.j3e.de/linux/convmv/convmv-1.14.tar.gz
tar xzvf convmv-1.14.tar.gz
cd convmv*
sed -i -e 's/usr/opt/1p' convmv
./convmv
Now the first times you run convmv some errors will show up that some perl libraries are missing. Look at the error messages and install these libraries using "ipkg install perl-whatever". Repeat until ./convmv shows the help.
Before you proceed, I would recommend taking a backup of the files/folders, as real folders/files will be renamed.
Then you can do this:
./convmv -r "/share/YOURSHARE/" -f iso-8859-1 -t utf8
This is a "test mode" and will first just show what folders/files would be renamed, i.e. where the utility finds "wrong" characters in the filename.
Then if you are confident you want to rename the files/folders you can run it using --notest:
./convmv -r "/share/YOURSHARE/" -f iso-8859-1 -t utf8 --notest
It will then rename the files so they can be read by QNAP Glacier/S3 Plus/Azure backup packages. It works for me now.
Good luck!
-
- New here
- Posts: 7
- Joined: Mon Apr 26, 2010 9:40 pm
Re: Azure Backup fails due to UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuat
Thank you for the tip..
I will try it..
I will try it..
- Don
- Guru
- Posts: 12289
- Joined: Thu Jan 03, 2008 4:56 am
- Location: Long Island, New York
Re: Azure Backup fails due to UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuat
This is a community forum so to get QNAP engaged you need to open a ticket with the help desk.ikk01 wrote:This is caused by an old "scandir" version that is included with the Azure, S3, Glacier, etc. backup QPKG provided by QNAP.
This has been fixed a couple years ago: https://github.com/benhoyt/scandir/issues/42
Is there any chance an official representative of QNAP can provide any information here. Can you give this information to the developers of the QNAP Azure/Glacier/... plugin, and have them add an updated version of "scandir" in the python-lib folder of the plugin?
This happens in Glacier/Azue latest version 1.1.379.
I am sure this is happening to many users.
Use the forum search feature before posting.
Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.
NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
Use RAID and external backups. RAID will protect you from disk failure, keep your system running, and data accessible while the disk is replaced, and the RAID rebuilt. Backups will allow you to recover data that is lost or corrupted, or from system failure. One does not replace the other.
NAS: TVS-882BR | F/W: 5.0.1.2346 | 40GB | 2 x 1TB M.2 SATA RAID 1 (System/VMs) | 3 x 1TB M.2 NMVe QM2-4P-384A RAID 5 (cache) | 5 x 14TB Exos HDD RAID 6 (Data) | 1 x Blu-ray
NAS: TVS-h674 | F/W: 5.0.1.2376 | 16GB | 3 x 18TB RAID 5
Apps: DNSMasq, PLEX, iDrive, QVPN, QLMS, MP3fs, HBS3, Entware, DLstation, VS, +
-
- New here
- Posts: 5
- Joined: Fri Aug 21, 2015 9:45 am
Re: Azure Backup fails due to UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuat
Hi,
I am the PM of CloudBackup Station.
We have found the root cause and will fixed in the next release.
After released, I will update the release version here.
Thanks a lot.
I am the PM of CloudBackup Station.
We have found the root cause and will fixed in the next release.
After released, I will update the release version here.
Thanks a lot.
-
- New here
- Posts: 5
- Joined: Fri Aug 21, 2015 9:45 am
Re: Azure Backup fails due to UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuat
Hello,
We had released the v1.1.387 to fixed this problem.
You can update now. Thank you.
We had released the v1.1.387 to fixed this problem.
You can update now. Thank you.
-
- New here
- Posts: 7
- Joined: Mon Apr 26, 2010 9:40 pm
Re: Azure Backup fails due to UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuat
Hi again!
I just upgraded to version 1.1.387. Unfortunately i still get the same error..
2016-05-15 09:43:54,034 PID:13045 backupscannerworkerthread.py:817 cc ERROR not handled exception
Traceback (most recent call last):
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 726, in run
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 508, in check_local_files
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 452, in _check_local_files_for_each
File "./package/src/shared/bin/diskdataprocessor.py", line 181, in walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 603, in _walk
File "/share/MD0_DATA/.qpkg/AzureStorage/python/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 87: invalid continuation byte
2016-05-15 09:44:10,232 PID:13045 engine.py:404 cc ERROR CloudConnectorError: of unknown error. Please try again
2016-05-15 09:44:11,600 PID:13045 engine.py:1038 cc INFO clear watchdog and var/garbage folder
2016-05-15 09:44:12,682 PID:13045 engine.py:1055 cc INFO end backup job:2
I just upgraded to version 1.1.387. Unfortunately i still get the same error..
2016-05-15 09:43:54,034 PID:13045 backupscannerworkerthread.py:817 cc ERROR not handled exception
Traceback (most recent call last):
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 726, in run
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 508, in check_local_files
File "./package/src/shared/bin/workerthread/backupscannerworkerthread.py", line 452, in _check_local_files_for_each
File "./package/src/shared/bin/diskdataprocessor.py", line 181, in walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 654, in _walk
File "./package/src/shared/python-lib/scandir.py", line 603, in _walk
File "/share/MD0_DATA/.qpkg/AzureStorage/python/lib/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 87: invalid continuation byte
2016-05-15 09:44:10,232 PID:13045 engine.py:404 cc ERROR CloudConnectorError: of unknown error. Please try again
2016-05-15 09:44:11,600 PID:13045 engine.py:1038 cc INFO clear watchdog and var/garbage folder
2016-05-15 09:44:12,682 PID:13045 engine.py:1055 cc INFO end backup job:2
-
- New here
- Posts: 7
- Joined: Mon Apr 26, 2010 9:40 pm
Re: Azure Backup fails due to UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 138: invalid continuat
Hi again!
FYI.. I just reported a issue at the Qnap helpdesk..
FYI.. I just reported a issue at the Qnap helpdesk..