QNAP Hybrid Sync Failing

Backup, Restore, Netbak Replicator, Cloud Storage Services
Locked
neorad84
New here
Posts: 3
Joined: Mon Mar 11, 2019 10:44 pm

QNAP Hybrid Sync Failing

Post by neorad84 »

Hi All, I have been fighting this issues for about a week now, I have 2 large (over 1tb in size) remote backups running to this system. it seems to get through most of it and then I get the following error.

QNAP Info

Model: TS-863U-RP
CPU: AMD GX-420MC SOC (4 Core)
Mem: 4GB
FW: 4.3.6.0875 Build 20190308
Disk: 32.69TB (Raid 6 Array 8 total disks)
Storage Pool:
Used: 1.87TB
Cap: 17.93TB


Severity Level Date Time Users Source IP Application Category Content
Error 2019/03/11 04:26:04 System 127.0.0.1 Hybrid Backup Sync Job Status [Hybrid Backup Sync] Failed to complete Sync job: "Backup". Encountered insufficient storage space or I/O error on destination device. Error code: -121

I have checked the firmware and it is up to date and when I look at the log for this backup job it appears that is is completing it but not sure (See below) (Directory names / Ip Address / Actual Job names are omitted for security)

[2019/03/11 03:18:33] Finished synchronizing.
[2019/03/11 03:18:33] Sync statistic: Total number of the folders: 56089. Total number of the files: 860861. Total updated size: 1.30 TB. Average updating speed: 48.33 (MB/sec). Time elapsed: 7 hours 51 minutes 33 seconds
[2019/03/11 03:18:34] # WARNING: Job encountered a remote I/O error or no space left on the destination device. Begin to retry the job
[2019/03/11 03:21:36] Synchronize files from a local folder to a remote folder.
[2019/03/11 03:21:36] Server IP address: xx.xx.xx.xx, port number 1.
[2019/03/11 03:21:36] The SSL connection option is disabled.
[2019/03/11 03:21:36] The number of folder pairs = 1.
[2019/03/11 03:21:36] Pair1 = [/mnt/RTRR_CIFS/].
[2019/03/11 03:21:36] Schedule type: Weekly (week day = 0, hour = 7, minute = 0).
[2019/03/11 03:21:36] The deleting extra files option is disabled.
[2019/03/11 03:21:36] The checking data content option is disabled.
[2019/03/11 03:21:36] The detecting sparse files option is disabled.
[2019/03/11 03:21:36] The extended attributes option is disabled.
[2019/03/11 03:21:36] Symbolic link files are included.
[2019/03/11 03:21:36] The detail log is enabled with the maximum size 10 MB.
[2019/03/11 03:21:36] Maximum retry times: 5; Retry interval in seconds: 180; Connection timeout in seconds: 60; Current retry number: 1.
[2019/03/11 03:26:40] # WARNING: Unable to create the " file/directory (Permission denied), it will be skipped. (-13)
[2019/03/11 03:32:03] Finished synchronizing.
[2019/03/11 03:32:03] Sync statistic: Total number of the folders: 56089. Total number of the files: 860861. Total updated size: 0 byte. Average updating speed: N/A. Time elapsed: 10 minutes 25 seconds
[2019/03/11 03:32:05] # WARNING: Job encountered a remote I/O error or no space left on the destination device. Begin to retry the job
[2019/03/11 03:35:07] Synchronize files from a local folder to a remote folder.
[2019/03/11 03:35:07] Server IP address: XX.XX.XX.XX, port number 1.
[2019/03/11 03:35:07] The SSL connection option is disabled.
[2019/03/11 03:35:07] The number of folder pairs = 1.
[2019/03/11 03:35:07] Pair1 = [/mnt/RTRR_CIFS/].
[2019/03/11 03:35:07] Schedule type: Weekly (week day = 0, hour = 7, minute = 0).
[2019/03/11 03:35:07] The deleting extra files option is disabled.
[2019/03/11 03:35:07] The checking data content option is disabled.
[2019/03/11 03:35:07] The detecting sparse files option is disabled.
[2019/03/11 03:35:07] The extended attributes option is disabled.
[2019/03/11 03:35:07] Symbolic link files are included.
[2019/03/11 03:35:07] The detail log is enabled with the maximum size 10 MB.
[2019/03/11 03:35:07] Maximum retry times: 5; Retry interval in seconds: 180; Connection timeout in seconds: 60; Current retry number: 2.
[2019/03/11 03:39:41] # WARNING: Unable to create the "" file/directory (Permission denied), it will be skipped. (-13)
[2019/03/11 03:45:06] Finished synchronizing.
[2019/03/11 03:45:06] Sync statistic: Total number of the folders: 56089. Total number of the files: 860861. Total updated size: 0 byte. Average updating speed: N/A. Time elapsed: 9 minutes 57 seconds
[2019/03/11 03:45:09] # WARNING: Job encountered a remote I/O error or no space left on the destination device. Begin to retry the job
[2019/03/11 03:48:10] Synchronize files from a local folder to a remote folder.
[2019/03/11 03:48:10] Server IP address: xx.xx.xx.xx, port number 1.
[2019/03/11 03:48:10] The SSL connection option is disabled.
[2019/03/11 03:48:10] The number of folder pairs = 1.
[2019/03/11 03:48:10] Pair1 = [/mnt/RTRR_CIFS/].
[2019/03/11 03:48:10] Schedule type: Weekly (week day = 0, hour = 7, minute = 0).
[2019/03/11 03:48:10] The deleting extra files option is disabled.
[2019/03/11 03:48:10] The checking data content option is disabled.
[2019/03/11 03:48:10] The detecting sparse files option is disabled.
[2019/03/11 03:48:10] The extended attributes option is disabled.
[2019/03/11 03:48:10] Symbolic link files are included.
[2019/03/11 03:48:10] The detail log is enabled with the maximum size 10 MB.
[2019/03/11 03:48:10] Maximum retry times: 5; Retry interval in seconds: 180; Connection timeout in seconds: 60; Current retry number: 3.
[2019/03/11 03:52:50] # WARNING: Unable to create the file/directory (Permission denied), it will be skipped. (-13)
[2019/03/11 03:58:42] Finished synchronizing.
[2019/03/11 03:58:42] Sync statistic: Total number of the folders: 56089. Total number of the files: 860861. Total updated size: 0 byte. Average updating speed: N/A. Time elapsed: 10 minutes 28 seconds
[2019/03/11 03:58:43] # WARNING: Job encountered a remote I/O error or no space left on the destination device. Begin to retry the job
[2019/03/11 04:01:43] Synchronize files from a local folder to a remote folder.
[2019/03/11 04:01:43] Server IP address: xx.xx.xx.xx, port number 1.
[2019/03/11 04:01:43] The SSL connection option is disabled.
[2019/03/11 04:01:43] The number of folder pairs = 1.
[2019/03/11 04:01:43] Pair1 = [/mnt/RTRR_CIFS/].
[2019/03/11 04:01:43] Schedule type: Weekly (week day = 0, hour = 7, minute = 0).
[2019/03/11 04:01:43] The deleting extra files option is disabled.
[2019/03/11 04:01:43] The checking data content option is disabled.
[2019/03/11 04:01:43] The detecting sparse files option is disabled.
[2019/03/11 04:01:43] The extended attributes option is disabled.
[2019/03/11 04:01:43] Symbolic link files are included.
[2019/03/11 04:01:43] The detail log is enabled with the maximum size 10 MB.
[2019/03/11 04:01:43] Maximum retry times: 5; Retry interval in seconds: 180; Connection timeout in seconds: 60; Current retry number: 4.
[2019/03/11 04:06:23] # WARNING: Unable to create the "" file/directory (Permission denied), it will be skipped. (-13)
[2019/03/11 04:12:19] Finished synchronizing.
[2019/03/11 04:12:19] Sync statistic: Total number of the folders: 56089. Total number of the files: 860861. Total updated size: 0 byte. Average updating speed: N/A. Time elapsed: 10 minutes 34 seconds
[2019/03/11 04:12:22] # WARNING: Job encountered a remote I/O error or no space left on the destination device. Begin to retry the job
[2019/03/11 04:15:24] Synchronize files from a local folder to a remote folder.
[2019/03/11 04:15:24] Server IP address: xx.xx.xx.xx, port number 1.
[2019/03/11 04:15:24] The SSL connection option is disabled.
[2019/03/11 04:15:24] The number of folder pairs = 1.
[2019/03/11 04:15:24] Pair1 = [/mnt/RTRR_CIFS.
[2019/03/11 04:15:24] Schedule type: Weekly (week day = 0, hour = 7, minute = 0).
[2019/03/11 04:15:24] The deleting extra files option is disabled.
[2019/03/11 04:15:24] The checking data content option is disabled.
[2019/03/11 04:15:24] The detecting sparse files option is disabled.
[2019/03/11 04:15:24] The extended attributes option is disabled.
[2019/03/11 04:15:24] Symbolic link files are included.
[2019/03/11 04:15:24] The detail log is enabled with the maximum size 10 MB.
[2019/03/11 04:15:24] Maximum retry times: 5; Retry interval in seconds: 180; Connection timeout in seconds: 60; Current retry number: 5.
[2019/03/11 04:20:10] # WARNING: Unable to create the " file/directory (Permission denied), it will be skipped. (-13)
[2019/03/11 04:26:02] Finished synchronizing.
[2019/03/11 04:26:02] Sync statistic: Total number of the folders: 56089. Total number of the files: 860861. Total updated size: 0 byte. Average updating speed: N/A. Time elapsed: 10 minutes 37 seconds
[2019/03/11 04:26:04] # ERROR: Job encountered a remote I/O error or no space left on the destination device.


Is it failing because of the permission denied (hence the I/O error) and I am actually getting everything backed up or is it failing because of something else, needless to say I am completely stumped and I have 2 jobs that do this (the other hasn't ran recently because of these issues)

Steps taken to resolve.

Saturday:

1. Verified that the system could get to all the shares that I am backing up.
2. Still had errors again with space, deleted all snapshot data off device (it was old data wasn't loosing anything)
3. Still had errors with space, went to settings and reinitialized the QNAP device (restored to factory settings)
4. Re setup all of my backup sync jobs on QNAP,
5. Re setup all of my folders on the QNAP device
6. Manually ran all of my backups once they were done waiting till Sunday

Sunday:
1. Have 2 remaining Jobs, all other jobs have processed successfully with out any issues.
2. Began Job in question (this is a last transfer and took most of the day and as you can see went into the morning hours on Monday)

Monday:

1. Logged in to see that the job indicated that it had failed (see message above)
2. Checked file space and found that the device had backed up 1.30 tb of data (the space according to my Equalogics for this directory on disk is only 810GB in size) not sure where it is seeing 1.30TB on its end.
3. Verified directories on both devices, and it appears things are the same
4. Writing this community forum post in hopes of gaining some insight / help with my device

Please if you can assist I would appreciate it I am stumped at this point as to why it is telling me this and backing up more Data than what is in those directories.

Thanks,

Aran
User avatar
storageman
Ask me anything
Posts: 5507
Joined: Thu Sep 22, 2011 10:57 pm

Re: QNAP Hybrid Sync Failing

Post by storageman »

I'm not clear. What are you backing up from/to?
neorad84
New here
Posts: 3
Joined: Mon Mar 11, 2019 10:44 pm

Re: QNAP Hybrid Sync Failing

Post by neorad84 »

i am backing up from a remote destination to the QNAP local storage pool (Update on this we found that the file that it couldn't access we or the end user couldn't access, I have removed said file and attempting a resync operation to see if it still has the same error.
neorad84
New here
Posts: 3
Joined: Mon Mar 11, 2019 10:44 pm

Re: QNAP Hybrid Sync Failing

Post by neorad84 »

Good news !!! That one file was the issue once it was removed it fixed the problem ! Thanks for your assistance everyone !!!
kerton
New here
Posts: 3
Joined: Mon Jul 08, 2019 4:26 am

Re: QNAP Hybrid Sync Failing

Post by kerton »

Ugh. Happy for you, but that's BRUTAL.

The fact that one errant file can sabotage a multi-hour backup???!?!
That seems to me to be just the kind of thing a backup software tool should "work around".
Although, to be fair to QNAP, in 25 years of computing, that's pretty much how backups always are for me. Fail, fail, fail.

If just ONE corrupt file or sector can scupper you, you're doomed.
The odds of finding > 0 corrupt sectors or corrupt files somewhere between two RAID arrays, or a Windows PC and a RAID array, have to be about 100%.

Why not back up everything else, and report the error to us in a log file?
User avatar
OneCD
Guru
Posts: 12037
Joined: Sun Aug 21, 2016 10:48 am
Location: "... there, behind that sofa!"

Re: QNAP Hybrid Sync Failing

Post by OneCD »

* topic locked to prevent further necroposting *

ImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImageImage
Locked

Return to “Backup & Restore”