[HOW-TO] Ideal home storage

[rualark]

Hi! In this article I will describe main problems of home data storage and my approach to them.

* Storage choosing *

There are plenty of devices on the market.

Today I believe good home storage needs to have:

- Transfer speed at least 30 Mbytes/sec to and from over SMB (or your main protocol) or greater
- Support at least RAID1, can use at least 2 disks
- Support Ethernet RJ-45 connection
- Support SMB and HTTP protocols (can access all data using both)
- Support automatic backup to external disk
- Debugging capabilities (who is using my storage? what are disks, processor and memory doing?)
- Reliable notifications via sound and email (hardware failures, backup failures, low free space and other)
- Reliable disaster recovery, good technical support and well-discussed on the internet (forums etc.)

So I chose QNAP TS-239 Pro II+

* Choosing disks *

When choosing disks you should:

- Plan needed capacity for 3-8 years ahead. You may make table like this:



- If you want to use disk actively, if you information is important - use server disks, else use consumer disks.
- Check disks officially compatible with your storage. This is vital. For QNAP use http://www.qnap.com/pro_compatibility.asp
- Check forums for issues with the disk chosen

So I chose two Hitachi Ultrastar 7K3000 HUA723030ALA640 server disks

I got these speeds with them (75/62 Mbytes/sec over SMB to RAID1):



* Network devices *

If you want to get more than 8 Mbytes/sec, choose gigabit switch. In general model it is not very important. You usually do not need Jumbo frames or some management capabilities for home usage.

Better minimise the number of network devices between your computers and Storage.

* Initial setup *

Important notice:

- Download latest firmware
- Follow other vendor recommendations for initial setup. For QNAP it is in the current manual http://www.qnap.com/download.asp?pl=1&p_mn=179
- Better insert both disks at once and create your RAID1 from scratch
- I do not recommend putting NAS outdoors, in a box, cupboard, closet, or any other cold/hot/wet environment. Best is a conditioned room.

* Advanced setup *

When your RAID is up and running, begin tuning your NAS. I recommend that you enter web interface and check all the pages and tabs. These are most important:

- Setup NTP. This will help you understand, what is NAS doing looking at your PC clock and understand time in logs



- Setup at least minimal password strength. This will help you if you or you friend forgets about security.



- When setting up DNS servers, I recommend that you use some public DNS server (e.g. I use Google 8.8.8.8 ) AND your provider's DNS server. This gives maximum reliability.



- If provider does not give you static IP-address and you want to access your NAS from Internet (your friends or your vacations), set up DDNS. I use no-ip. Note that User name must be with domain.



- I recommend the following hardware settings



(these are optimal. If you have problems, you may need to disable Write cache)
(I recommend to turn standby mode off, increasing you HDD life. If you have noise or hot problems, turn it on)

- I do not recommend using QNAP High Security Level, because the system is already pretty secure. If you will need in future, you can block some hosts using Medium Security Level.



- I recommend that you turn on Network Access Protection for all protocols that you use.



- I recommend these power settings if you are not sure:



* Disk management *

I recommend that you enable temperature alarm at 50 degrees C. Researches show that best temperature for HDD is 35-45 degrees C and minimal spindowns.

I recommend that you enable automatic rapid tests once a month. If your disk is older that 5 years, you may enable tests every week.



I recommend that you enable Bitmap on the RAID. This may help you in a disaster and does not require much performance or disk space. Best choise is to enable Bitmap after you initially write all the main data on the disks.



I recommend that you don't use iSCSI at home. This protocol does not support sharing and has little benefits over standard ones (SMB, AFP, HTTP). iSCSI may be used for special purposes like virtual machines.

* Setup notifications *

I recommend that you setup notifications, so that you know when something is wrong with your NAS and you are not at home. I use email notifications, because I check email often.

To setup email notifications you can create a google account for your NAS and send mail through it. This keeps your main email password secure:



(Note, that you need to check authentication and SSL/TLS boxes)
(Note, that you can use any Sender, even non-existent)

Do not forget to enter your real email address on Alert Notification page:



* Setup protocols *

Microsoft networking: usually standalone + local master browser for home.





For others use default settings.

* Security *

Recommendations:

- Disable all services that you are not going to really need (usually FTP, NFS, Telnet, SFTP is not needed. Also turn off Apple Networking if you use Windows)
- Divide all your data into Public and Private folders. This will help you to grow.
- Use group permissions instead of user permissions even if you have little users for now. This will help you to grow.
- Always passwords where you can use them. Use SNMP community (not public).

* Data growth *

Data is always growing and if you do not control it, you end up not knowing where to find some files and the disks will always be not enough for you.

This is why you have to divide your data into classes and plan how you are going to store it. For example, you may go from most critical data to least important and build a table like this:



Now you can look at this table 1-2 times a year and delete and archive unneeded data. Frankly speaking, this will make your data two times more accessible and two times smaller in size.

* Data protection *

Usually home users have 3 types of data:

+ Not important data (movies, downloads, games etc.) - this data can be stored on cheap desktop drives in a PC

+ Important data. You do not want to loose this data, but it is not a catastrophy. This data can be stored on a NAS RAID1 (or RAID5) and also you can make a tree of the data and put it in a "very important" data location.

+ Very important data. You do not want to loose this data and it is a catastrophy. This data can be stored on a NAS RAID1 (or RAID5) and be automatically backed up to an external USB-drive. Usually this data is less then 10% of Important data and 2.5 inch drive is enough for backup.



To use external backup I connected 2.5 inch usb disk to QNAP, created Critic folders in both Public and Private and set up backup in web interface (I also backup Cacti scripts folder):



Also I:

- periodically backup QNAP settings to Private/Critic folder
- automatically backup all configurations (/mnt/HDA_ROOT and /mnt/ext) to Private/Critic folder using the following cron job:
- backup Gmail, Google documents, ICQ and Skype logs, mobile phone address book (online data) yearly to this folder Private/Critic/Backup.
15 3 1 * * /share/Main/Private/Critic/Backup/QNAP/backup.sh

[/] # cat /share/Main/Private/Critic/Backup/QNAP/backup.sh
/bin/tar -czf mntext.tgz /mnt/ext
/bin/tar -czf HDA_ROOT.tgz /mnt/HDA_ROOT

To make trees of my data I use the following windows bat script yearly (it resides in Private/Critic/Backup/tree):

SET mycd=%cd%
SET par=/F /A
mkdir %date%
x:
cd x:\
tree %par% > %mycd%\%date%\tree-x-%date%.txt
z:
cd z:\
tree %par% > %mycd%\%date%\tree-z-%date%.txt
n:
cd n:\public
tree %par% > %mycd%\%date%\tree-npu-%date%.txt
n:
cd n:\private
tree %par% > %mycd%\%date%\tree-npr-%date%.txt

Trees help me to remember, which files (software versions or movies) I used and download them again if needed, when my PC disks fail.

Also, I greatly recommend that you enable Network Recycle Bin. This will help you to recover files that you or your friends deleted by mistake.



* Web server *

To use web server to access public files, you can do the following:

- make "files" folder in your Web folder
- create link in "files" folder to other folders that you want to publish:
ln -s /share/Public /share/Web/files/

- now if you want to grant access to the folder with password, create .htaccess and htpasswd files:
[/share/Web/files] # vi .htaccess
Options +Indexes
AuthName "Private zone"
AuthType Basic
AuthUserFile /etc/config/htpasswd
require valid-user

[/share/Web/files] # /usr/local/apache/bin/htpasswd -c /etc/config/htpasswd john
New password: ******
Re-type new password: ******
Adding password for user john

- To make indexes more informative and useful, I recommend adding into the "<IfModule autoindex_module>" section of the Apache config:

[/share/Web/files] # vi /usr/local/apache/conf/apache.conf
Options Indexes FollowSymLinks
IndexOptions FancyIndexing
IndexOptions VersionSort
IndexOptions HTMLTable
IndexOptions FoldersFirst
IndexOptions IconsAreLinks
IndexOptions IgnoreCase
IndexOptions SuppressDescription
IndexOptions SuppressHTMLPreamble
IndexOptions XHTML
IndexOptions IconWidth=16
IndexOptions IconHeight=16
IndexOptions NameWidth=*
IndexOrderDefault Descending Name
AddType text/html .php
HeaderName /index-style/header.html
ReadmeName /index-style/footer.php
<Files "*.php">
AddHandler application/x-httpd-php .php
</Files>

- To display icons near the filenames for better viewing copy icons folder from my archive to /share/Web/ folder (see archive in the end of the post).

Do not forget to restart your web server in admin web interface or console (/usr/local/apache/bin/apachectl restart).
You will have the following:



If you install PHP Autoindex scripts, you will also be able to download total folders as archives: http://autoindex.sourceforge.net/

I also wrote a simple web-interface for fast searching the storage. It has easy filename-search mode and full text search mode. I include it in the archive in the end of the post. You will need to install mlocate and findutils for it to work (via IPKG). This is how it looks like:






* Monitoring *

Latest QNAP firmware (I use 3.4.4) has new features for monitoring: better CPU/memory monitoring with TOP processes, good logs of System events and Connections (does not include SMB and HTTP connections on Web server, only Administration and Web File Manager)

I recommend that you turn on System Connection Logs, so that if somebody tries to hack you, you can go there and see, what happened:



To further extend monitoring and debugging capabilities I use the following:

- Cacti for monitoring resources (howto here http://forum.qnap.com/viewtopic.php?p=106885)
- Awstat for monitoring web server access (howto here http://wiki.qnap.com/wiki/AWStats)
- PhpMyAdmin for Mysql management (download from internet)
- Ntop for detailed information about traffic (usually disabled, I enable it when needed)
- Netquery for remote site testing (download from internet)
- Speedtest MINI for throughput measurement (download from internet)
- Additional console applications (iotop and lsof to know what processes are using disks, sysstat for detailed information about resources, tcpdump for sniffing traffic when needed, iptraf for online traffic statistics, ncdu to analyse disk usage)

Most of these are installed with Optware IPKG (http://wiki.qnap.com/wiki/Install_Optware_IPKG), some are directly downloaded with wget.

To save changes on reboot I use my script /etc/config/autorun.sh, which is run from flash disk /tmp/config/autorun.sh (see http://wiki.qnap.com/wiki/Autorun.sh):

ln -sf /etc/config/profile /etc
ln -sf /share/Public/Critic/System/awstats/db /var/lib/awstats
ln -sf /etc/config/awstats /etc
ln -sf /share/Public/Critic/System/awstats /usr/local
ln -s /opt/bin/perl /usr/bin/
/share/MD0_DATA/Private/Critic/Backup/QNAP/ramdisk-load.sh

To prevent Cacti disk access every 5 minutes I moved all RRA, Cacti mysql database and some scripts to ramdisk (using ln -s).

ln -s /var/cacti/mysql /share/Main/.@mysql/cacti
ln -s /var/cacti/site/include /share/Web/cacti/
ln -s /var/cacti/site/cli /share/Web/cacti/
ln -s /var/cacti/site/lib /share/Web/cacti/
ln -s /var/cacti/site/resource /share/Web/cacti/
ln -s /var/cacti/site/rra /share/Web/cacti/

[/var/cacti] # du -ch
1.1M ./mysql
7.0K ./site/resource/script_queries
5.0K ./site/resource/script_server
19K ./site/resource/snmp_queries
32K ./site/resource
132K ./site/cli
40K ./site/lib/adodb/lang
59K ./site/lib/adodb/datadict
392K ./site/lib/adodb/drivers
835K ./site/lib/adodb
1.7M ./site/lib
152K ./site/include/jscalendar/lang
212K ./site/include/jscalendar
74K ./site/include/treeview
471K ./site/include
61K ./site/scripts
7.2M ./site/rra
9.5M ./site
11M .
11M total

To avoid problems you also have to copy several files to /var/cacti/site:

cp /share/Web/cacti/cmd.php /var/cacti/site/
cp /share/Web/cacti/script_server.php /var/cacti/site/

They are backed up from ramdisk to disk every hour and loaded in autorun.sh (see above):

47 1-21 * * * /share/Main/Private/Critic/Backup/QNAP/ramdisk-save.sh

[/] # cat /share/Main/Private/Critic/Backup/QNAP/ramdisk-save.sh
cd /share/MD0_DATA/Private/Critic/Backup/QNAP/
/bin/tar -czf ramdisk-cacti.tgz /var/cacti

Storing something on ramdisk is not generally recommended, but I wanted to avoid excessive noise and disk wear. I also disabled mysql binary logs for cacti in /etc/my.cnf, which prevents mysql from constantly writing to binary log:

log-bin=mysql-bin
binlog-ignore-db=cacti

Awstats setup guide can be seen here: http://wiki.qnap.com/wiki/AWStats



Do not forget to move log files from /mnt/ext to your big disk. Also, I use logrotate for apache logs:

[/] # cat /opt/etc/logrotate.conf
compress

"/usr/local/apache/logs/access_log" /usr/local/apache/logs/error_log {
rotate 5
size=200k
sharedscripts
postrotate
/usr/local/apache/bin/apachectl restart
endscript
}

I run cacti and awstat only when I am not going to sleep, because they make a little noise, although cacti mainly uses ramdisk:

*/5 7-21 * * * /mnt/ext/opt/apache/bin/php /share/Web/cacti/poller.php >/dev/null 2>&1
57 1-21 * * * /usr/local/awstats/tools/awstats_updateall.pl now

Do not forget to copy all your crontab to /etc/config/crontab. I usually edit with "crontab -e", then copy everything to /etc/config/crontab and then update cron with "/etc/init.d/crond.sh restart".

Here are my cacti graphs. For ping I use slightly modified Advanced Ping (http://forums.cacti.net/about10049.html):






















I also collect information from my main PC into cacti (standard templates).
Also I use templates and scripts for Apache and Mysql from extremely good "mysql-cacti-templates" project.

* Other applications *

Additional applications for QNAP may be installed one of the following ways:

- From admin web interface (QPKG). Here we see some interesting packages (XMail, Dovecot, Joomla, Wordpress, Gallery, Piwik, TwonkyMedia, Magento, iStat, GLPI, vtiger CRM, Eyeos for cloud hosting and others)
- From Optware IPKG (installed from QPKG) - here you have 1315 packages today
- Manually downloading and installing

You can download my QNAP Cacti templates and scripts + icons for Autoindex + find.php (searching the storage) at http://dl.dropbox.com/u/1350128/qnap-re ... -cacti.rar

[onlyalex]

Nice HowTo mate.
Calculating for the future is a must since it will be here tomorow. Your Qnap device can do many things and you are exloring all options and doing some learning on the way

Best regards.

[XeviouS]

WOW!

Brilliant info....I can't wait to try the cacti scripts.

Many thanks for sharing.

Cheers

[rhett7660]

If only the documentation that came with the device was this good. Thank you very much! Excellent post.

[rualark]

How to decrease home storage size:

* Remove unneeded files and files that you can always find in the internet *

Some of files can be too old.
Some files can be public in the internet and torrents and you do not need to store them at home.

* Remove duplicate files *

You can just remove the duplicates or replace them with windows LNK files.
One of my favorite applications, which find files and can replace them with LNK files is CloneSpy.
I usually use only pool1 and the following settings:

clonespy.png

* Convert RAW to JPEG *

RAW files take much more space then JPEG files and are usually needed if you are planning to process them. Usually you do not need to store them for long time.
If you have RAW files mixed with your JPEG files, you can use Far to copy all RAW files out of your directory tree and them copy them back. You can convert RAWs to JPEGs with Photoshop:

First open up Photoshop and then the Image Processor via File>Scripts>Image Processor.

1) Locate & select the RAW files that you wish to batch convert.

pts2.png

2) Choose where you wish to save the outputted JPG’s. It’s easier to save them in the same location though for convenience. Photoshop will automatically create a separate folder containing these JPG’s.
3) Choose the format that you want to save the RAW files to. In my case it’s JPG’s, but you are given the option to save to TIFF or PSD as well.
4) In the final step, you are also given the option to run an Action on all of the RAW files whilst saving them to the specified format. This can be another big time saver. The other option is to include the ICC Profile (International Colour Consortium) This will be checked by default. I will always keep this checked.
Once you are happy, simply hit RUN, and the conversions will begin. You won’t get a message to tell you that the converting has completed.

(This RAW to JPEG tutorial was copied from here: http://www.techfleece.com/2011/10/28/ho ... photoshop/)

* Decrease resolution of your images *

I usually select best photos that I take and process them. Then I all the other photos and batch resize them to 1920x1080 using SoftOrbits Batch Picture Resizer.
This program can calculate image height correctly if its proportions are not like 1920x1080. What is good about this program is that it does not process images that are already small.
I usually use the following settings:

resizer.jpg

* Convert and cut your videos *

When you shoot videos using videocamera or phone, usually they are not compressed or weakly compressed. You can batch compress all your videos and then check for videos that became larger or almost did not change in size (this can happen when video was already compressed) and for videos that degraded in quality too much. Delete these bad converted videos and use source videos for these.
To batch convert all videos in a folder recursively you can use the following script (Windows BAT file). If you use disks L: and M: then do not forget to replace l: and m: in the script with two disk names that you do not use.
You will need to download ffmpeg.exe and put it in the same folder with this batch file. Start script like this: convert.bat c:\my_videos c:\converted_videos

Code: Select all

@echo off
set ffpath=N:\Private\Critic\Scripts\ffmpeg\
setlocal disableDelayedExpansion

REM map unused drive letters to source and destination
for %%F in ("%~1\.") do subst l: "%%~fF"
for %%F in ("%~2\.") do subst m: "%%~fF"

REM keep a counter for files converted
set /A nfile=0
REM do not copy empty folders or any files
@echo Copying directory structure from %1 to %2 ...
xcopy /T l:\ m:\

REM walk directory structure and convert each file in quiet mode
for /R l:\ %%F in (*.*) do (
  echo converting "%%~nxF" ...
  %ffpath%ffmpeg -v quiet -i "%%F" -preset slow -crf 20 -b:a 128k "m:%%~pnF-converted.mp4"
  set /A nfile+=1
)
echo Done! Converted %nfile% file(s)
rem exit /b

REM release mapped drive letters
subst /d l:
subst /d m:

Also, you can cut some videos, if you need to keep only part of the videos. This can also be done from command line, but often cut files are unplayable or are longer that you want. So check. The command can be something like this:

Code: Select all

ffmpeg -ss 00:01:03 -t 00:02:08 -i movie.mp4 -acodec copy -vcodec copy -async 1 cut.mp4

* Use compression or deduplication *

You have the following choices:
1. You can use Rar or 7Zip to compress files that you do not need often. Also, you can compress backups of all your data, but I found this not very effective for WinRar (7% economy at 7 MByte/s).
2. You can use native NTFS compression (2,3% economy at 40 Mbyte/s), but this increases fragmentation and reduces disk read and write speeds.
3. You can use deduplication on Windows 2012 and 8.1 (8,8% economy at 50 Mbyte/s + 30 Mbyte/s optimization). This reduces disk read speeds a little bit. Write speed is almost not reduced, because deduplication takes place at night.
4. If you do not use deduplication or compression, files can take a little more then their size (0,05% more for my 255'000 taking 1050 Gb). The more files, the more this overhead will generally be.