How do I run a memory test on a NAS that does not have video out?

[BigRedBrent]

I've been downloading large torrent files and they've become immediately corrupted when I simply move them to a different folder. Before I move them to a folder they are verified correctly. I highly suspect the problem is the memory, but I don't know of any way to test this because it doesn't have an HDMI output to use memtest86.

I did a memory check for the Windows 10 client computer that I am using, and it passed without any errors. I have also not experienced any problems on the Windows 10 client computer. So I believe that it is very likely that the problem is with the QNAP NAS.

The current memory that is being used is non-ECC. I have ordered some ECC replacement memory. I would just like to verify if the current memory will give errors or not, so that I may try to get them replaced, to be able to sell the replacements. ECC memory is expensive...

Model: TS-873A

[nibbles2bits]

you can use the serial console for basic output (just noticed u don't have one visible on the rear sorry)
you can enable auto running the specific tests you want and enable autoreporting
that saves the results to the memtest86 boot drive

[BigRedBrent]

I actually discovered that I had a compatible video card laying around and installed it. I will just leave the video card in there and use it for transcoding as well.

[nibbles2bits]

Let us know if it was RAM. That corruption was what I saw when I had a bad stick.

[BigRedBrent]

I honestly hope that it is, because I've already ordered the replacement ecc memory that should be here today, and I'm currently running the test and it is at 12% for the first pass and hasn't showed errors yet. But it is 64 GB of memory.

[nibbles2bits]

that's good news 12% and no errors. iirc bad RAM fails usually right away in the first or second test in the first pass, mine did.

[BigRedBrent]

that's good news 12% and no errors. iirc bad RAM fails usually right away in the first or second test in the first pass, mine did.

It's not good news to me, because if it's not the memory then I don't have a clue what the problem is... :/

[dosborne]

It's not good news to me, because if it's not the memory then I don't have a clue what the problem is... :/

I recall reading a number of posts about people having similar issues due to 5.0.x firmware and I believe it was one of the apple protocols for connecting. Perhaps a search will turn something up. What firmware are you running and how are you connected? You may want to try a simple samba share and a PC if possible for a test.

[BigRedBrent]

It is currently running a memory test. I remember that it ended with 2022, so I believe that it is: h5.0.0.2022

I believe that I have the apple protocol disabled. I use samba with a Windows 10 client. I have tested the memory for the windows 10 client with no errors detected.

I'm using ZFS with triple redundancy on 8 x 8TB enterprise class helium filled hard drives. What could cause data corruption on the drive after a file is downloaded correctly and even after being manually verified, and then becomes corrupt and then no longer passes verification after moving the file into a different folder on the same share, other than the memory?

The replacement memory is running late, and I have not recieved it yet. I plan on using the ecc replacement memory no matter what. But I do want to find out what is the cause of this problem.

[nibbles2bits]

Yeah I was a bit too happy considering.

I'd be curious to see the result of a test where you:

• Use your windows machine as you have been to exhibit the problem moving a file to another directory on the same share.
• ssh into the NAS and attempt to recreate the problem using the mv command on a similar file.

Given the number of drives you mentioned, the other things it could be are worse than RAM,
like a bug in the handling of pointers or an off by 1 error in an array,
a capacitor that's borderline, or a bug in SMB.

The test I suggested will narrow things down more. There should be a way to get some logging,
but I'm not a syslog guy. I'd be curious if it happens on every share, on a new share, on a new filesystem,
or on a USB drive. Is there any sort of fsck that's been run or is appropriate to run?
Did the move command exit with 0 status, as in

Code: Select all

#!/bin/sh

mv -f /path/to/the/file.mkv /path/to/newdir
if [ $? -eq 0 ]
then
    echo "Move is successful"
fi

[BigRedBrent]

One thing that's concerning me, is that it doesn't do it every single time. It's only started doing it in the last couple of weeks, but I got pretty concerned when it did it to two Torrent downloads in a row of a couple gigabytes each, and then I re-downloaded those same torrents and it did it to them again.

I was able to get the files moved where I wanted them by using another program that would copy them into place and then verify the files after they've been copied, and then they were fine. But just simply moving them was causing the issue.

Because I was so worried about possible data corruption of the server, I decided to shut the server down and look into possibly fixing whatever the issue might be.

[BigRedBrent]

The memory test has done one pass without any errors. I find that unfortunate. I'm still going to use the new memory when it gets here, but if it doesn't fix the problem, then I'm back at square one I guess.

I'll get back with you with any additional tests that I perform. At some point I'll try to reproduce the problem. Since it only started happening, and only seem to happen rarely, and only recently happened several times in a row. Unless it just never happens again, I'll probably have no way of knowing if the problem is gone or not, if I'm unable to reproduce it again.

At first I thought it might have been the torrent downloader that was not downloading it correctly, but since then I've set it up to automatically reverify the data once it's finished downloading, and that never seems to detect any problems. It only seems to be when I move it using cut and paste to another folder in the same share. I should also point out that this folder that I move it to is a folder that Plex monitors.

[dosborne]

Use the onboard file manager instead to try moving the files from one folder to another to see if it is a "local" issue or a "connectivity" issue.

[BigRedBrent]

When I logged into the machine after the memory test, it had a new firmware release available, so I updated the firmware. I don't know if the problem is sporadic or fixed, but I have not had the problem as of yet.

I also changed the way I download torrents. I disabled pre-allocation of files, increased the disk cache size to 1GB. I have set it to copy the finished files to a new folder on completion with deduplication disabled, then verify the copied files, and then delete the original files. This is my attempt to reduce fragmentation caused by torrent downloads.

I am most definitely still going to install the ecc memory when it arrives though.

Here is my script if anyone is interested:

Code: Select all

#!/bin/bash

USERNAME="username"
PASSWORD="password"
PORT="9091"
TRANSMISSION_REMOTE="/usr/bin/transmission-remote"
VERIFY=true

TORRENT_HASH=$TR_TORRENT_HASH
if [ -z "$TORRENT_HASH" ]; then
    if [ ${#1} -ne 40 ]; then
        if [ "$1" != "TRANSMISSION_POST_START_COMPLETED_SCRIPT_IN_BACKGROUND_STARTUP_ARGUMENT" ]; then
            "$0" "TRANSMISSION_POST_START_COMPLETED_SCRIPT_IN_BACKGROUND_STARTUP_ARGUMENT" > /dev/null 2>&1 &
            exit 0
        fi
        rm -rf /tmp/transmission-script-*.lock
        sleep 1
        TORRENT_LIST=$($TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --list | sed -e '1d' -e '$d' | awk '{print $1}' | sed -e 's/[^0-9]*//g')
        for TORRENT_ID in $TORRENT_LIST
        do
            TORRENT_DATA=$($TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_ID --info)
            if [ -n "$TORRENT_DATA" ]; then
                TORRENT_HASH=$(echo "${TORRENT_DATA}" | sed -n -e 's/^ *Hash: *//p')
                if [ ${#TORRENT_HASH} -eq 40 ]; then
                    "$0" $TORRENT_HASH > /dev/null 2>&1 &
                fi
            fi
        done
        exit 0
    fi
    TORRENT_HASH=$1
    sleep 5
fi

LOCK_FILE="/tmp/transmission-script-${TORRENT_HASH}.lock"
if [ -f "$LOCK_FILE" ]; then exit 1; fi
touch "$LOCK_FILE"
trap "rm -rf $LOCK_FILE" EXIT
COPY_LOCK_FILE="/tmp/transmission-script-copying.lock"
COPY_LOCKED=false

TORRENT_DATA=$($TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --info)
if [ $? -ne 0 ]; then exit 1; fi
if [ -z "$TORRENT_DATA" ]; then exit 1; fi

TORRENT_STATE=$(echo "${TORRENT_DATA}" | sed -n -e 's/^ *State: *//p')
if [[ $TORRENT_STATE = *"Verify"* ]]; then
    while [ -f "$COPY_LOCK_FILE" ]
    do
        sleep 5
    done
    touch "$COPY_LOCK_FILE"
    trap "rm -rf $LOCK_FILE; rm -rf $COPY_LOCK_FILE" EXIT
    COPY_LOCKED=true
    while true
    do
        sleep 5
        TORRENT_DATA=$($TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --info)
        if [ $? -ne 0 ]; then exit 1; fi
        if [ -z "$TORRENT_DATA" ]; then exit 1; fi
        TORRENT_STATE=$(echo "${TORRENT_DATA}" | sed -n -e 's/^ *State: *//p')
        if [[ $TORRENT_STATE != *"Verify"* ]]; then break 1; fi
    done
fi
if [ "$TORRENT_STATE" != "Finished" ] && [ "$TORRENT_STATE" != "Stopped" ] && [ "$TORRENT_STATE" != "Seeding" ] && [ "$TORRENT_STATE" != "Idle" ]; then exit 1; fi

TORRENT_NAME=$(echo "${TORRENT_DATA}" | sed -n -e 's/^ *Name: *//p')
TORRENT_DIR=$(echo "${TORRENT_DATA}" | sed -n -e 's/^ *Location: *//p')
TORRENT_PERCENT_DONE=$(echo "${TORRENT_DATA}" | sed -n -e 's/^ *Percent Done: *//p')
TORRENT_SEEDING=false
VERIFY_TMP=true
if [ "$TORRENT_PERCENT_DONE" = "100%" ]; then
    if [ "$TORRENT_STATE" = "Seeding" ] || [ "$TORRENT_STATE" = "Idle" ]; then
        TORRENT_SEEDING=true
    fi
else
    VERIFY_TMP=false
    VERIFY=true
fi

TORRENT_PARENT_DIR=$(echo "$TORRENT_DIR" | sed -e 's/\(\/\.incomplete\)$//g')
if [ "$TORRENT_PARENT_DIR" = "$TORRENT_DIR" ]; then
    TORRENT_PARENT_DIR=$(echo "$TORRENT_DIR" | sed -e 's/\(\/\.verify\)$//g')
    if [ "$TORRENT_PARENT_DIR" = "$TORRENT_DIR" ]; then
        TORRENT_PARENT_DIR=$(echo "$TORRENT_DIR" | sed -e 's/\(\/\.tmp\)$//g')
        if [ "$TORRENT_PARENT_DIR" = "$TORRENT_DIR" ]; then
            TORRENT_PARENT_DIR=$(echo "$TORRENT_DIR" | sed -e 's/\(\/\.seeding\)$//g')
            if [ "$TORRENT_PARENT_DIR" = "$TORRENT_DIR" ]; then
                if [ "$TORRENT_PERCENT_DONE" != "100%" ]; then exit 0; fi
                $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --move "$TORRENT_PARENT_DIR/.verify" || exit 1
            elif [ "$TORRENT_PERCENT_DONE" = "100%" ]; then
                if [ "$TORRENT_STATE" = "Finished" ] || [ "$TORRENT_STATE" = "Stopped" ]; then
                    $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --move "$TORRENT_PARENT_DIR" || exit 1
                    sleep 1
                    rm -rf "$TORRENT_PARENT_DIR/.seeding/$TORRENT_NAME" 2> /dev/null
                    rmdir "$TORRENT_PARENT_DIR/.seeding" 2> /dev/null
                    $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --remove || exit 1
                fi
                exit 0
            else
                $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --move "$TORRENT_PARENT_DIR/.incomplete" || exit 1
                sleep 1
                rm -rf "$TORRENT_PARENT_DIR/.seeding/$TORRENT_NAME" 2> /dev/null
                rmdir "$TORRENT_PARENT_DIR/.seeding" 2> /dev/null
                if [ "$TORRENT_STATE" = "Finished" ] || [ "$TORRENT_STATE" = "Stopped" ]; then
                    $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --start || exit 1
                fi
                exit 0
            fi
        else
            $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --find "$TORRENT_PARENT_DIR/.verify" || exit 1
            if [ "$TORRENT_PERCENT_DONE" != "100%" ]; then
                sleep 1
                $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --move "$TORRENT_PARENT_DIR/.incomplete" || exit 1
                sleep 1
                rm -rf "$TORRENT_PARENT_DIR/.verify/$TORRENT_NAME" 2> /dev/null
                rmdir "$TORRENT_PARENT_DIR/.verify" 2> /dev/null
                rm -rf "$TORRENT_PARENT_DIR/.tmp/$TORRENT_NAME" 2> /dev/null
                rmdir "$TORRENT_PARENT_DIR/.tmp" 2> /dev/null
                if [ "$TORRENT_STATE" = "Finished" ] || [ "$TORRENT_STATE" = "Stopped" ]; then
                    $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --start || exit 1
                fi
                exit 0
            fi
        fi
    elif [ "$TORRENT_PERCENT_DONE" != "100%" ]; then
        $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --move "$TORRENT_PARENT_DIR/.incomplete" || exit 1
        sleep 1
        rm -rf "$TORRENT_PARENT_DIR/.verify/$TORRENT_NAME" 2> /dev/null
        rmdir "$TORRENT_PARENT_DIR/.verify" 2> /dev/null
        rm -rf "$TORRENT_PARENT_DIR/.tmp/$TORRENT_NAME" 2> /dev/null
        rmdir "$TORRENT_PARENT_DIR/.tmp" 2> /dev/null
        if [ "$TORRENT_STATE" = "Finished" ] || [ "$TORRENT_STATE" = "Stopped" ]; then
            $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --start || exit 1
        fi
        exit 0
    fi
else
    if [ "$TORRENT_PERCENT_DONE" != "100%" ]; then exit 0; fi
    $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --move "$TORRENT_PARENT_DIR/.verify" || exit 1
    sleep 1
    rm -rf "$TORRENT_PARENT_DIR/.incomplete/$TORRENT_NAME" 2> /dev/null
    rmdir "$TORRENT_PARENT_DIR/.incomplete" 2> /dev/null
fi

sleep 1
while true
do
    if [ "$VERIFY_TMP" = true ]; then
        if [ "$COPY_LOCKED" = false ]; then
            while [ -f "$COPY_LOCK_FILE" ]
            do
                sleep 5
            done
            touch "$COPY_LOCK_FILE"
            trap "rm -rf $LOCK_FILE; rm -rf $COPY_LOCK_FILE" EXIT
            COPY_LOCKED=true
        fi
        mkdir -p "$TORRENT_PARENT_DIR/.tmp" 2> /dev/null
        rm -rf "$TORRENT_PARENT_DIR/.tmp/$TORRENT_NAME" 2> /dev/null
        cp -R "$TORRENT_PARENT_DIR/.verify/$TORRENT_NAME" "$TORRENT_PARENT_DIR/.tmp/" || exit 1
        $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --find "$TORRENT_PARENT_DIR/.tmp" || exit 1
    else
        $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --find "$TORRENT_PARENT_DIR/.verify" || exit 1
    fi

    if [ "$VERIFY" = true ]; then
        sleep 1
        $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --verify || exit 1
        while true
        do
            sleep 5
            TORRENT_DATA=$($TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --info)
            if [ $? -ne 0 ]; then exit 1; fi
            if [ -z "$TORRENT_DATA" ]; then exit 1; fi
            TORRENT_STATE=$(echo "${TORRENT_DATA}" | sed -n -e 's/^ *State: *//p')
            if [[ $TORRENT_STATE != *"Verify"* ]]; then break 1; fi
        done
    fi

    sleep 1
    TORRENT_DATA=$($TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --info)
    if [ $? -ne 0 ]; then exit 1; fi
    if [ -z "$TORRENT_DATA" ]; then exit 1; fi
    TORRENT_STATE=$(echo "${TORRENT_DATA}" | sed -n -e 's/^ *State: *//p')
    TORRENT_PERCENT_DONE=$(echo "${TORRENT_DATA}" | sed -n -e 's/^ *Percent Done: *//p')

    if [ "$TORRENT_STATE" = "Finished" ] || [ "$TORRENT_STATE" = "Stopped" ] || [ "$TORRENT_STATE" = "Seeding" ] || [ "$TORRENT_STATE" = "Idle" ]; then
        if [ "$TORRENT_PERCENT_DONE" = "100%" ]; then
            if [ "$VERIFY_TMP" = true ]; then
                if [ "$TORRENT_SEEDING" = true ]; then
                    $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --move "$TORRENT_PARENT_DIR/.seeding" || exit 1
                    $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --bandwidth-low || exit 1
                else
                    $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --move "$TORRENT_PARENT_DIR" || exit 1
                fi
                sleep 1
                rm -rf "$TORRENT_PARENT_DIR/.verify/$TORRENT_NAME" 2> /dev/null
                rmdir "$TORRENT_PARENT_DIR/.verify" 2> /dev/null
                rm -rf "$TORRENT_PARENT_DIR/.tmp/$TORRENT_NAME" 2> /dev/null
                rmdir "$TORRENT_PARENT_DIR/.tmp" 2> /dev/null
                if [ "$TORRENT_SEEDING" != true ]; then
                    $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --remove || exit 1
                fi
                exit 0
            fi
            VERIFY_TMP=true
        else
            if [ "$VERIFY_TMP" = false ]; then
                $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --move "$TORRENT_PARENT_DIR/.incomplete" || exit 1
                sleep 1
                rm -rf "$TORRENT_PARENT_DIR/.verify/$TORRENT_NAME" 2> /dev/null
                rmdir "$TORRENT_PARENT_DIR/.verify" 2> /dev/null
                rm -rf "$TORRENT_PARENT_DIR/.tmp/$TORRENT_NAME" 2> /dev/null
                rmdir "$TORRENT_PARENT_DIR/.tmp" 2> /dev/null
                $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --start || exit 1
                exit 0
            fi
            VERIFY_TMP=false
        fi
    else
        $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TORRENT_HASH --find "$TORRENT_PARENT_DIR/.verify"
        sleep 1
        rm -rf "$TORRENT_PARENT_DIR/.tmp/$TORRENT_NAME" 2> /dev/null
        rmdir "$TORRENT_PARENT_DIR/.tmp" 2> /dev/null
        exit 1
    fi
    VERIFY=true
done

[nibbles2bits]

ECC RAM should reduce the variables if you're rigorous about being constantly grounded while working on your NAS.

I like the script. Looks like it wasn't too bad to debug Is it your understanding that the exit status of cp would be unreliable and that hashing the files is the correct way? It's not often I have bad RAM to test. I probably would have used the default Bourne Shell because I haven't learned if Bash runs 100% as expected for admin work. Any insights there?