How do I run a memory test on a NAS that does not have video out?

Don't miss a thing. Post your questions and discussion about other uncategorized NAS features here.
BigRedBrent
Starting out
Posts: 37
Joined: Fri Feb 15, 2019 5:57 am

How do I run a memory test on a NAS that does not have video out?

Post by BigRedBrent » Thu Jun 23, 2022 5:24 pm

I've been downloading large torrent files and they've become immediately corrupted when I simply move them to a different folder. Before I move them to a folder they are verified correctly. I highly suspect the problem is the memory, but I don't know of any way to test this because it doesn't have an HDMI output to use memtest86.

I did a memory check for the Windows 10 client computer that I am using, and it passed without any errors. I have also not experienced any problems on the Windows 10 client computer. So I believe that it is very likely that the problem is with the QNAP NAS.

The current memory that is being used is non-ECC. I have ordered some ECC replacement memory. I would just like to verify if the current memory will give errors or not, so that I may try to get them replaced, to be able to sell the replacements. ECC memory is expensive...

Model: TS-873A

nibbles2bits
Starting out
Posts: 34
Joined: Fri Aug 23, 2019 10:17 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by nibbles2bits » Fri Jun 24, 2022 4:41 am

you can use the serial console for basic output (just noticed u don't have one visible on the rear sorry)
you can enable auto running the specific tests you want and enable autoreporting
that saves the results to the memtest86 boot drive

BigRedBrent
Starting out
Posts: 37
Joined: Fri Feb 15, 2019 5:57 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by BigRedBrent » Fri Jun 24, 2022 6:24 am

I actually discovered that I had a compatible video card laying around and installed it. I will just leave the video card in there and use it for transcoding as well. :)

nibbles2bits
Starting out
Posts: 34
Joined: Fri Aug 23, 2019 10:17 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by nibbles2bits » Fri Jun 24, 2022 6:37 am

Let us know if it was RAM. That corruption was what I saw when I had a bad stick.

BigRedBrent
Starting out
Posts: 37
Joined: Fri Feb 15, 2019 5:57 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by BigRedBrent » Fri Jun 24, 2022 6:55 am

I honestly hope that it is, because I've already ordered the replacement ecc memory that should be here today, and I'm currently running the test and it is at 12% for the first pass and hasn't showed errors yet. But it is 64 GB of memory.

nibbles2bits
Starting out
Posts: 34
Joined: Fri Aug 23, 2019 10:17 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by nibbles2bits » Fri Jun 24, 2022 7:24 am

that's good news 12% and no errors. iirc bad RAM fails usually right away in the first or second test in the first pass, mine did.

BigRedBrent
Starting out
Posts: 37
Joined: Fri Feb 15, 2019 5:57 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by BigRedBrent » Fri Jun 24, 2022 7:59 am

nibbles2bits wrote:
Fri Jun 24, 2022 7:24 am
that's good news 12% and no errors. iirc bad RAM fails usually right away in the first or second test in the first pass, mine did.
It's not good news to me, because if it's not the memory then I don't have a clue what the problem is... :/

dosborne
Easy as a breeze
Posts: 478
Joined: Tue May 29, 2018 3:02 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by dosborne » Fri Jun 24, 2022 9:58 am

BigRedBrent wrote:
Fri Jun 24, 2022 7:59 am
It's not good news to me, because if it's not the memory then I don't have a clue what the problem is... :/
I recall reading a number of posts about people having similar issues due to 5.0.x firmware and I believe it was one of the apple protocols for connecting. Perhaps a search will turn something up. What firmware are you running and how are you connected? You may want to try a simple samba share and a PC if possible for a test.
QNAP TS-563-16G 5x10TB Seagate Ironwolf HDD Raid-5 Triple NIC: 2x1GB 1x10GbE
QNAP TS-231P-US 2x18TB Seagate Exos HDD Raid-1 Dual 1GB NIC

BigRedBrent
Starting out
Posts: 37
Joined: Fri Feb 15, 2019 5:57 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by BigRedBrent » Fri Jun 24, 2022 10:49 am

It is currently running a memory test. I remember that it ended with 2022, so I believe that it is: h5.0.0.2022

I believe that I have the apple protocol disabled. I use samba with a Windows 10 client. I have tested the memory for the windows 10 client with no errors detected.

I'm using ZFS with triple redundancy on 8 x 8TB enterprise class helium filled hard drives. What could cause data corruption on the drive after a file is downloaded correctly and even after being manually verified, and then becomes corrupt and then no longer passes verification after moving the file into a different folder on the same share, other than the memory?

The replacement memory is running late, and I have not recieved it yet. I plan on using the ecc replacement memory no matter what. But I do want to find out what is the cause of this problem.

nibbles2bits
Starting out
Posts: 34
Joined: Fri Aug 23, 2019 10:17 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by nibbles2bits » Fri Jun 24, 2022 12:34 pm

Yeah I was a bit too happy considering.

I'd be curious to see the result of a test where you:
  • Use your windows machine as you have been to exhibit the problem moving a file to another directory on the same share.
  • ssh into the NAS and attempt to recreate the problem using the mv command on a similar file.
Given the number of drives you mentioned, the other things it could be are worse than RAM,
like a bug in the handling of pointers or an off by 1 error in an array,
a capacitor that's borderline, or a bug in SMB.

The test I suggested will narrow things down more. There should be a way to get some logging,
but I'm not a syslog guy. I'd be curious if it happens on every share, on a new share, on a new filesystem,
or on a USB drive. Is there any sort of fsck that's been run or is appropriate to run?
Did the move command exit with 0 status, as in

Code: Select all

#!/bin/sh

mv -f /path/to/the/file.mkv /path/to/newdir
if [ $? -eq 0 ]
then
    echo "Move is successful"
fi
Last edited by nibbles2bits on Fri Jun 24, 2022 12:52 pm, edited 1 time in total.

BigRedBrent
Starting out
Posts: 37
Joined: Fri Feb 15, 2019 5:57 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by BigRedBrent » Fri Jun 24, 2022 12:45 pm

One thing that's concerning me, is that it doesn't do it every single time. It's only started doing it in the last couple of weeks, but I got pretty concerned when it did it to two Torrent downloads in a row of a couple gigabytes each, and then I re-downloaded those same torrents and it did it to them again.

I was able to get the files moved where I wanted them by using another program that would copy them into place and then verify the files after they've been copied, and then they were fine. But just simply moving them was causing the issue.

Because I was so worried about possible data corruption of the server, I decided to shut the server down and look into possibly fixing whatever the issue might be.

BigRedBrent
Starting out
Posts: 37
Joined: Fri Feb 15, 2019 5:57 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by BigRedBrent » Fri Jun 24, 2022 1:09 pm

The memory test has done one pass without any errors. I find that unfortunate. I'm still going to use the new memory when it gets here, but if it doesn't fix the problem, then I'm back at square one I guess.

I'll get back with you with any additional tests that I perform. At some point I'll try to reproduce the problem. Since it only started happening, and only seem to happen rarely, and only recently happened several times in a row. Unless it just never happens again, I'll probably have no way of knowing if the problem is gone or not, if I'm unable to reproduce it again.

At first I thought it might have been the torrent downloader that was not downloading it correctly, but since then I've set it up to automatically reverify the data once it's finished downloading, and that never seems to detect any problems. It only seems to be when I move it using cut and paste to another folder in the same share. I should also point out that this folder that I move it to is a folder that Plex monitors.

dosborne
Easy as a breeze
Posts: 478
Joined: Tue May 29, 2018 3:02 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by dosborne » Fri Jun 24, 2022 11:35 pm

Use the onboard file manager instead to try moving the files from one folder to another to see if it is a "local" issue or a "connectivity" issue.
QNAP TS-563-16G 5x10TB Seagate Ironwolf HDD Raid-5 Triple NIC: 2x1GB 1x10GbE
QNAP TS-231P-US 2x18TB Seagate Exos HDD Raid-1 Dual 1GB NIC

BigRedBrent
Starting out
Posts: 37
Joined: Fri Feb 15, 2019 5:57 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by BigRedBrent » Sat Jun 25, 2022 12:19 am

When I logged into the machine after the memory test, it had a new firmware release available, so I updated the firmware. I don't know if the problem is sporadic or fixed, but I have not had the problem as of yet.

I also changed the way I download torrents. I disabled pre-allocation of files, increased the disk cache size to 1GB. I have set it to copy the finished files to a new folder on completion with deduplication disabled, then verify the copied files, and then delete the original files. This is my attempt to reduce fragmentation caused by torrent downloads.

I am most definitely still going to install the ecc memory when it arrives though.

Here is my script if anyone is interested:

Code: Select all

#!/bin/bash

USERNAME="username"
PASSWORD="password"
PORT="9091"
TRANSMISSION_REMOTE="/usr/bin/transmission-remote"

TORRENT_DIR=$(echo "$TR_TORRENT_DIR" | sed -e 's/\(\/\.verify\)$//g')
if [ "$TR_TORRENT_DIR" = "$TORRENT_DIR" ]; then
    $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TR_TORRENT_HASH --move "$TORRENT_DIR/.verify"
fi

sleep 1
VERIFY_TMP=true
while true
do
    if [ "$VERIFY_TMP" = true ] ; then
        rm -rf "$TORRENT_DIR/.tmp/$TR_TORRENT_NAME"
        mkdir -p "$TORRENT_DIR/.tmp"
        cp -R "$TORRENT_DIR/.verify/$TR_TORRENT_NAME" "$TORRENT_DIR/.tmp/"
        $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TR_TORRENT_HASH --find "$TORRENT_DIR/.tmp"
    else
        $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TR_TORRENT_HASH --find "$TORRENT_DIR/.verify"
    fi
    sleep 1

    $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TR_TORRENT_HASH --verify
    sleep 2
    while true
    do
        sleep 5
        TORRENT_DATA=$($TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TR_TORRENT_HASH --info)
        TORRENT_STATE=$(echo "${TORRENT_DATA}" | sed -n -e 's/^ *State: *//p')
        if [[ $TORRENT_STATE != *"Verify"* ]]; then
            break 1
        fi
    done

    sleep 1
    TORRENT_DATA=$($TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TR_TORRENT_HASH --info)
    TORRENT_STATE=$(echo "${TORRENT_DATA}" | sed -n -e 's/^ *State: *//p')

    if [ "$TORRENT_STATE" = "Finished" ] || [ "$TORRENT_STATE" = "Stopped" ]; then
        TORRENT_PERCENT_DONE=$(echo "${TORRENT_DATA}" | sed -n -e 's/^ *Percent Done: *//p')
        if [ "$TORRENT_PERCENT_DONE" = "100%" ]; then
            if [ "$VERIFY_TMP" = true ] ; then
                $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TR_TORRENT_HASH --move "$TORRENT_DIR"
                sleep 1
                rm -rf "$TORRENT_DIR/.verify/$TR_TORRENT_NAME"
                rmdir "$TORRENT_DIR/.verify"
                rmdir "$TORRENT_DIR/.tmp"
                $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TR_TORRENT_HASH --remove
                exit 0
            fi
            VERIFY_TMP=true
        else
            if [ "$VERIFY_TMP" != true ] ; then
                rm -rf "$TORRENT_DIR/.tmp/$TR_TORRENT_NAME"
                rmdir "$TORRENT_DIR/.tmp"
                $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TR_TORRENT_HASH --start
                exit 0
            fi
            VERIFY_TMP=false
        fi
    else
        $TRANSMISSION_REMOTE $PORT --auth $USERNAME:$PASSWORD --torrent $TR_TORRENT_HASH --find "$TORRENT_DIR/.verify"
        sleep 1
        rm -rf "$TORRENT_DIR/.tmp/$TR_TORRENT_NAME"
        rmdir "$TORRENT_DIR/.tmp"
        exit 0
    fi
done
Last edited by BigRedBrent on Sat Jun 25, 2022 7:59 pm, edited 7 times in total.

nibbles2bits
Starting out
Posts: 34
Joined: Fri Aug 23, 2019 10:17 am

Re: How do I run a memory test on a NAS that does not have video out?

Post by nibbles2bits » Sat Jun 25, 2022 6:47 am

ECC RAM should reduce the variables if you're rigorous about being constantly grounded while working on your NAS.

I like the script. Looks like it wasn't too bad to debug :) Is it your understanding that the exit status of cp would be unreliable and that hashing the files is the correct way? It's not often I have bad RAM to test. I probably would have used the default Bourne Shell because I haven't learned if Bash runs 100% as expected for admin work. Any insights there?

Post Reply

Return to “Miscellaneous”