Fully Automated TV show Download Center

Introduce yourself to us and other members here, or share your own product reviews, suggestions, and tips and tricks of using QNAP products.
colicab
Know my way around
Posts: 105
Joined: Thu Jul 29, 2010 4:17 pm

Fully Automated TV show Download Center

Post by colicab » Wed Mar 06, 2013 2:41 am

[I've updated this how-to because of several updates (firmware, rtorrent...)]

I'm probably one out of a lot of people who puts a considerable amount of of time in collecting new TV show episodes.
Having a collection of 20+ ongoing TV shows it became a daily pass-time to check for new episodes, finding the correct subtitles, renaming and moving the episodes to my multimedia library.

Reading about a lot of cool handy programs, I decided to automate the process, merely by concatenating these programs.

What I propose here is:
- a fully automated TV show downloader
- totally free using torrents (opposed to Usenet and Sickbeard)
- everything is done on the QNAP, independent from user input

Basically what my setup does is:
1) Find new TV show torrent files using an RSS feed and FlexGet
2) Download torrent with rtorrent
3) Download English and Dutch subtitles using AutoSub (from Bierdopje.com)
4) Rename the episodes and move them to the multimedia library using FileBot
5) Rescan XBMC library on an HTPC

Some notes:
i) My QNAP setup at the time of writing is: QNAP TS-659 Pro+ running firmware 4.0.2
ii) Your folder structure will probably differ from mine so make changes accordingly: all my shares are under /share/MD0_DATA/. Therefor I advise not simply to copy-paste any code or file content before having inspected it.
iii) To do this setup, you will do some Linux command line: working with ssh,vi and browsing through the file structure. Also I working knowledge of Python basics might come in handy, although the scripts I propose for moving files etc. can be achieved with any other scripting language of your choosing.



What you need before starting

A) Install these QPKG packages from App Center:
- Optware (0.99.163)
- JRE (build 1.7.0_40-b43, headless): follow the first reply in this thread http://forum.qnap.com/viewtopic.php?f=163&t=47652
- rtorrent (0.9.3): manual installation. Download from http://pool.qnapclub.pl/projects/packages/rtorrent/releases/

B) Download these files:
- AutoSub: Download the non-Windows one https://code.google.com/p/auto-sub/downloads/list note:
I'm using the latest version: 0.5.8
- FileBot: Choose one of the Embedded Linux packages, depending on your QNAP model http://filebot.sourceforge.net/#download.
Using the latest version: 3.62

C) Install python and setup tools on your QNAP
ssh into your QNAP

Code: Select all

ipkg install python26
ipkg install py26-setuptools
ipkg install py26-cheetah


I use python2.6 because it's the latest Python version to come with Cheetah on QNAP.
Cheetah is required to get AutoSub running and setuptools are required to install FlexGet (see below)

D) Make a directory to put all your scripts and config files
ssh into your QNAP

Code: Select all

cd /share/MD0_DATA/.qpkg/
mkdir myscripts



This is practical to put all your scripts and links together so you only need to remember one directory for future adjustments

Setting up rtorrent (Bit-torrent client)

I choose rtorrent for one specific reason: multiple watch-download directory pairs. If you make a subdir in the rtorrent/watch folder, it will create the same subdir in downloads/ and complete/ so you can direct torrent traffic. By making these subdirs you can apply custom-made downstream processing for any of the folders in the rtorrent/complete folder. Very convenient IMO...

Setting it up is easy enough. After manually installing the qpkg in App Center (step A above), it's essentially done. Off course you can adjust a lot of settings which is just an added bonus for getting rtorrent. I only adjust the download speed to unlimited, the rest I keep default.
After installation rtorrent makes a folder tree for downloading in Qdownload/rtorrent. To segregate my torrent files, I made a couple of subdirs in the rtorrent/watch folder:
-tvshows: all torrent files from FlexGet go here (see later in the how-to)
-movies: movies with subtitles
-movies_getsubs: movie torrent files without subtitles
-no_action: torrent files that I will process manually

I also made a folder rtorrent/tvshows with some subdirs:
- subtitles: after episode is download file gets moved from rtorrent/complete/tvshows to here to get subtitiles
- filebot: after subtitle is download, the file gets moved from subtitles to here to get renamed by FileBot and moved to the series library

Now we have our Bittorrent set up, we have to get our torrent files. We'll use FlexGet for this



Flexget

Flexget downloads torrents from an RSS feed. These torrents file will be put in the watch folder of rtorrent (/share/MD0_DATA/Qdownload/rtorrent/watch/tvshows)
I got the installation info from: http://flexget.com/wiki/InstallWizard/QNAP

For the installation, ssh into your QNAP and give these commands one by one:

Code: Select all

easy_install-2.6 flexget


Based on the tutorial, FlexGet should be installed properly. However I got an error trying to run FlexGet

Code: Select all

ImportError: No module named pkg_resources

I you get this, give this command to fix it:

Code: Select all

curl http://python-distribute.org/distribute_setup.py | python2.6


Alright, now it should start by typing

Code: Select all

flexget
It wont do much except show a table with options but you know that it works.

Next step is to configure FlexGet for your favourite tv shows. You have to make a config.yml file for this. Make it in a text editor on your PC or with vi or something on your QNAP.
The location on your QNAP to put it is /opt/local/
On the FlexGet site, you can get a lot (really a lot) of information how to do make this file: http://flexget.com/wiki/Configuration. To make this file will take some reading into, especially if you want to additional features.
Basically, config.yml is a file that tells FlexGet where to look for new torrents (input; RSS feed), which ones to take (filter; series list) and where to put the torrent (output; watch folder). There are a lot of different ways to make this file. I'll just show you mine and explain what's is in it.

I have 2 different tasks 'seed_series_db' and 'kickass'. Kickass is my default tasks: this one is done automatically everytime flexget runs. Let's first take a loot at that one.

Code: Select all

tasks:
  seed_series_db:
    import_series:
      from:
        thetvdb_favorites:
          account_id: <your TVDB ID>
    find:
      regexp: .*(avi|mkv|mp4)$
      path: /share/MD0_DATA/Qmultimedia/Series
      recursive: yes
    manual: yes
    # manual run: flexget --task seed_series_db --disable-advancement --learn

  kickass:
    content_filter:
      require: '*.mkv'
      reject: '*.rar'
    download: /share/Qdownload/rtorrent/watch/tvshows
    import_series:
      from:
        thetvdb_favorites:
          account_id: <your TVDB ID>
      settings:
        upgrade: yes
        qualities:
          - 1080p webdl
          - 720p webdl
        series:
          - NCIS:
               exact: yes
    rss: http://kickass.to/tv/?rss=1

As you can see, it doesn't really matter if the options of a task are chronologically. My input ('rss') is the last line.
With the rss: I tell Flexget to look at the RSS feed from kickasstorrents to retrieve torrent files. Off course you can use any rss feeds for tv shows, I'm just inclined to use kickasstorrents', my favorite torrent site.
For the filter, I use import_series which accesses my TvDB account to learn which series I'm watching. If a torrent for a certain episode in the rss matches my favorite series, it gets downloaded. Also, flexget remembers your progress, so it wont download an episode more than once, normally.
With the option 'settings' I specify the preferred video quality, mine is either 720p WEB-DL or 1080p WEB-DL. The 720p version is available sooner so it gets downloaded first. With the upgrade option, however, FlexGet knows that ultimately I want the 1080p version so it will download this one too once it arrives on kickasstorrents. I have 1 extra filter because I don't want to download rar files, only mkv's to prevent the additional step of building up the zipped file.
One caveat perhaps which you have to take into account is in the case of partial matching series. I have this with NCIS, which is a favorite in my TvDB account, NCIS LA. However since NCIS partially matches NCIS LA this one will get downloaded too. This behavior might have some advantages, but in my case I have to tell Flexget I want NCIS to match exactly to the series title; this is an override for the TvDB input. Finally, with download: I specify the watch folder .

The 2nd task is not for downloading torrents but to initialize the FlexGet database, a list of which episode are already downloaded. This comes in handy If you already have a series library before using FlexGet. Or you need to reinstall for some reason. By giving the following command FlexGet will learn which episode are already in your library (off course, your path should be altered to your own media library)

Code: Select all

 flexget --task seed_series_db --disable-advancement --learn


If you put your finished config.yml file in /opt/local/ FlexGet will start downloading the torrents you asked for in the watch folder. For future purposes I would advise to make a softlink to this file in your /share/MD0_DATA/.qpkg/myscripts/ folder (from now on I will just refer to it as myscripts/). Doing this, you can later on change the file from this link, so you don't need to remember the actual path.

Code: Select all

ln -s /opt/local/config.yml /share/MD0_DATA/.qpkg/myscripts/config.yml


Off course we want to automate FlexGet, so the QNAP will periodically do this for us. We will use the Linux program cron for this at the end of the tutorial.
For now we will start by making a bash file (pipeline.sh) in the myscripts/ directory and put this in:

Code: Select all

#!/bin/sh
/opt/bin/python2.7 /opt/local/bin/flexget


Make it executable by giving the command

Code: Select all

chmod +x pipeline.sh


So, at this moment we already have our torrents being downloaded to Qdownload/rtorrent/complete/tvshows.
(If you don't care for subtitles, you can skip the next step discussing AutoSub where we will automate the downloading of subtitles)


Getting subtitles with AutoSub

To download subtitles I use AutoSub from Bierdopje.com. This website both has the English as well as the Dutch subtitles.
The English subtitles are almost immediately available when the new episode is, but the Dutch subtitle takes a couple of days longer before you can download it. Being inpatient I don't want to wait for the Dutch subtitle to watch the episode, so I wrote 2 Python scripts to move the episode once the English subtitle is available. Put these in your myscripts/ folder and make them executable


movescript.py

Code: Select all

'''
This is the top-level script to process new TV show episodes
It uses the object.py file to create Episode objects
'''

import os
from media import Episode

rtorrent = '/share/Qdownload/rtorrent/'

for root, dirs, files in os.walk(rtorrent):
   for file in files:
      file = os.path.join(root,file)
      if file.endswith('.mkv'):
         file = Episode(file)
         #print(file)
         file.run()


object.py

Code: Select all

import os
import shutil
import re


class Episode:
   def __init__(self, fullpath, sim=0):
      self.fullpath = fullpath      # the full path including, episode file
      self.name = os.path.basename(fullpath)   # name of episode file
      self.upperdir = os.path.dirname(fullpath)   # directory of episode file (may be the completed map)
      self.platform = None
      self.strip = self.fullpath[:-3]
      self.eng, self.nl = self.strip + 'English.srt', self.strip + 'Dutch.srt'         
      if sim:
         self.platform = '/share/Qdownload/rtorrent_bis/'
         print('TEST mode...')
      else:
         self.platform = '/share/Qdownload/rtorrent'
      self.values = [os.path.join(self.platform,folder) for folder in ['complete/tvshows','tvshows/subtitles', 'tvshows/filebot']]
      self.keys = ['completed', 'subtitles', 'rename']
      self.folders = dict(zip(self.keys,self.values))
      #print self.folders
      self.excluded = ['tvshows', 'rtorrent_bis', 'rtorrent', 'Qdownload', 'MD0_DATA', 'share', '/']
   def failsafe(self,check):
                for dir in self.excluded:
                        if re.search(dir, check, re.I): return None
         else: return True
   def clean(self):
      if self.failsafe(os.path.basename(self.upperdir)):
         for folder, subfolder, files in os.walk(self.upperdir):
            ext = [os.path.splitext(file)[1] for file in files]
            if '.mkv' in ext:
               pass
               #print("still mkvs present in directory, wont delete")
            else:
               #print("Deleting '%s'" % (self.upperdir,))
               shutil.rmtree(self.upperdir)
      else:
         #print("you were about to delete '%s'. Thats's not allowed!!" % (self.upperdir,))
         pass
   
   def __str__(self):
      return "-File:'%s'\n Stripped name:'%s'\n English sub:'%s'\n Dutch sub:'%s'\n Full path:'%s'\n Upper dir:'%s'\n  " \
      % (self.name, self.strip,self.eng, self.nl, self.fullpath, self.upperdir)
   def makeDummy(self):
      dummy = os.path.join(self.folders['subtitles'], self.name)
      open(dummy,"w")
   def customprint(self,file):
      print("Moving '%s' to rename/ folder" % (file))
   def run(self):
      if re.match(self.folders['completed'], self.upperdir, re.I):
         if re.search('1080p', self.name, re.I):
            m = re.search ('(.*)S\d{2}E\d{2}', self.name, re.I).group()
            for dir,subdir,files in os.walk(self.folders['subtitles']):
               for file in files:
                  if (re.search(m, file, re.I) and re.search('720p', file)):
                     #print("Delete %s" % (file,))
                     os.remove(os.path.join(dir,file))
         shutil.move(self.fullpath, self.folders['subtitles'])
         #print("Moving '%s' to subtitles folder" % (self.name,))
         self.clean()
      if self.upperdir == self.folders['subtitles']:
         if os.path.getsize(self.fullpath) == 0: ## if True: the original mkv and english subs are already at destination, see below
            if os.path.exists(self.nl):               
               shutil.move(self.nl, self.folders['rename'])
               os.unlink(self.fullpath)
               if os.path.exists(self.eng): os.unlink(self.eng)   ## Shoud normally always exist; extra failsafe
         elif os.path.exists(self.eng):
            if os.path.exists(self.nl):               
               for x in [self.fullpath, self.eng, self.nl]:
                  shutil.move(x, self.folders['rename'])
               #print("Moving mkv, english and dutch subtitle of %s to filebot folder" % (self.fullpath,))
            else:
               for x in [self.fullpath, self.eng]:
                                                shutil.move(x, self.folders['rename'])
               self.makeDummy()
               #print("Moving mkv and english subtitle of %s to filebot folder\nMake dummy mkv file" % (self.fullpath,))

if __name__ == '__main__':
   for root, dirs, files in os.walk('/share/Qdownload/rtorrent_bis'):
      for file in files:
                        file = os.path.join(root,file)
         if file.endswith('.mkv'):
            file = Episode(file,sim=1)
            #print(file)
            file.run()



I'll just say briefly what these scripts do. First it looks in rtorrent/complete/tvshows to see if a new episode (<episode>.mkv) is downloaded which it will move to rtorrent/tvshows/subtitles. It will also remove clutter files (eg .nfo files) and remaining folders.
rtorrent/tvshows/subtitles is the watch directory for AutoSub: when subtitles are available, they get put in here. If the English subtitle is downloaded (<episode>.English.srt) it will move the subtitle and the episode to rtorrent/tvshows/filebot. Waiting for the Dutch subtitle, it will make a 'dummy' empty mkv file with the same name so AutoSub is still looking for the Dutch Subtitle. Once this one is available and downloaded, it will also move this one to rtorrent/tvshows/filebot and remove the dummy file (as well as the English subtitle because AutoSub will keep on downloading this one).
Additionally, because every episode gets downloaded twice (first 720p then 1080p), it will delete the 720p data in the AutoSub directory when the 1080p version is downloaded and moved to that folder. That way all 720p gets replaced by 1080p.

Again to automate, we put another command line in the pipeline.sh file

Code: Select all

/opt/bin/python2.7 /share/MD0_DATA/.qpkg/myscripts/movescript.py 



Now we setup AutoSub. I mainly used this topic from the QNAP forum to install it http://forum.qnap.com/viewtopic.php?p=298485

Put the AutoSub folder you downloaded previously in your file directory as

Code: Select all

/share/MD0_DATA/.qpkg/AutoSub


Make an autostart script called AutoSub.sh and make it executable: put here the code you find on the AutoSub page https://code.google.com/p/auto-sub/wiki/QnapAutoStart
Put this one in the AutoSub folder. Off course this one has to be edited according to your setup
Mine looks like this:

Code: Select all

#! /bin/sh
### BEGIN INIT INFO
# Provides: Auto-sub application instance
# Required-Start: $all
# Required-Stop: $all
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: starts instance of Auto-sub
# Description: starts instance of Auto-sub using start-stop-daemon
### END INIT INFO
############### EDIT ME ##################
QPKG_NAME=AutoSub
QPKG_DIR=
DAEMON=/opt/bin/python2.6
DAEMON_OPTS=" /share/MD0_DATA/.qpkg/AutoSub/AutoSub.py -c /share/MD0_DATA/.qpkg/AutoSub/config.properties -d -l"
case "$1" in
start)
echo "Starting $QPKG_NAME"
if [ `/sbin/getcfg ${QPKG_NAME} Enable -u -d FALSE -f /etc/config/qpkg.conf` = UNKNOWN ]; then
/sbin/setcfg ${QPKG_NAME} Enable TRUE -f /etc/config/qpkg.conf
elif [ `/sbin/getcfg ${QPKG_NAME} Enable -u -d FALSE -f /etc/config/qpkg.conf` != TRUE ]; then
echo "${QPKG_NAME} is disabled."
exit 1
fi
${DAEMON} ${DAEMON_OPTS}
;;
stop)
echo "Stopping $QPKG_NAME"
for pid in $(/bin/pidof python); do
/bin/grep -q "AutoSub.py" /proc/$pid/cmdline && /bin/kill $pid
done
/bin/sleep 2
;;
restart|force-reload)
echo "Restarting $QPKG_NAME"
$0 stop
$0 start
;;
*)
N=/etc/init.d/$QPKG_NAME
echo "Usage: $N {start|stop|restart|force-reload}" >&2
exit 1
;;
esac
exit 0


The only thing you have to change the lines DAEMON and DAEMON_OPTS according to your system and file system.
Now make a config.properties file and also put this in the AutoSub folder.

Code: Select all

[webserver]
username =
webserverport = 8083
webroot =
webserverip = 0.0.0.0
password =

[logfile]
loglevel = info
lognum = 1
loglevelconsole = error
logsize = 1000000

[config]
subeng = English
notifynl = True
logfile = AutoSubService.log
checksub = 28800
scandisk = 3600
minmatchscore = 8
checkrss = 900
rootpath = /share/MD0_DATA/Qdownload/rtorrent/tvshows/subtitles
minmatchscorerss = 14
notifyen = True
fallbacktoeng = True
postprocesscmd =
path = /share/MD0_DATA/.qpkg/AutoSub
configversion = 2
subnl = Dutch
launchbrowser = True
downloadeng = True

[notify]
mailencryption = TLS
mailtoaddr = example@gmail.com
growlport = 23053
notifytwitter = False
notifyprowl = False
growlpass = mysecretpassword
twittersecret = token secret
mailusername = example@gmail.com
twitterkey = token key
mailsrv = smtp.gmail.com:587
growlhost = 127.0.0.1
notifynma = False
mailsubject = Subs info
mailfromaddr = example@gmail.com
notifygrowl = False
mailauth =
mailpassword = mysecretpassword
prowlpriority = -2
nmaapi = API key
notifymail = False
prowlapi =


AutoSub is a program which will run as a daemon after the NAS has been restarted. For that you nee to edit the /etc/config/qpkg.conf file to autostart the AutoSub.sh script.
Again, it's advisable to make a softlink for this file in your myscripts/ folder. Add this to the file:

Code: Select all

[AutoSub]
Name = AutoSub
Version = 0.5.8
Enable = TRUE
QPKG_File = Autosubdummy.qpkg
Date = 2012-12-09
shell = /share/MD0_DATA/.qpkg/AutoSub/AutoSub.sh
Install_Path = /share/MD0_DATA/.qpkg/AutoSub
WebUI = /
Author = Bas
Web_port = 8083


After refreshing the App Center, you will see the AutoSub application running. Browsing to port 8083 you will see the web page of AutoSub. You can further configure here, like refresh rates etc.



FileBot

FileBot is the last part of our pipeline. It's a program to rename and move our files to the multimedia library.
Up until now we have the mkv episode file and the subtitle files going to the rtorrent/tvshows/filebot folder. So this is the input folder for FileBot.

Browse to the folder with the FileBot install file and type

Code: Select all

ipkg install filebot_3.4_i686.ipk


To configure FileBot you have to give in a command telling FileBot what to do with files residing in the watch folder.
Check these pages to make commands to your liking:
http://filebot.sourceforge.net/naming.html
http://filebot.sourceforge.net/forums/viewtopic.php?t=2
http://filebot.sourceforge.net/cli.html
Again as for FlexGet, this takes some reading if you want to further customize this part.

Here is mine:

Code: Select all

/opt/usr/bin/filebot -script svn:amc /share/Qdownload/rtorrent/tvshows/filebot -non-strict --output /share/Qmultimedia --def "seriesFormat=Series/{n}/{n} Season {s}/{n}.{s.pad(2)}x{e.pad(2)}.{t}{'.'+lang.getDisplayName(Locale.ENGLISH)}" --def myepisodes=<user>:<password> --def xbmc=192.168.1.19 --conflict override


What this does is:
- Rename files to look like this: "Breaking Bad.01x01.Pilot.mkv" "Breaking Bad.01x01.Pilot.English.srt" "Breaking Bad.01x01.Pilot.Dutch.srt"
- Put it in a my multimedia file structure looking like Qmultimedia --> Series --> Breaking Bad --> Breaking Bad Season 1 --> Breaking Bad.01x01.Pilot.mkv ...
- Update MyEpisodes.com account (this option seems deprecated)
- Update XBMC library on my HTPC (you need to know the IP address). Of course you have to allow XBMC to do this: System > Network > Allow programs on other systems to control XBMC <enable>
- Override existing episode files (720p quality) if it is already present in the multimedia library

Once you're finished with drafting the command, copy it on a new line in the pipeline.sh file.


Crontab

As said, as a last step we need to automate our pipeline. We already tied all the pieces together by making the pipeline.sh file. AutoSub and rtorrent don't need to be automated manually, they reside in the App Center and will run after you have restarted your QNAP.

The pipeline.sh should look like this

Code: Select all

#!/bin/sh
/opt/bin/python2.7 /opt/local/bin/flexget
/opt/bin/python2.7 /share/MD0_DATA/.qpkg/myscripts/tvshows.py
/opt/usr/bin/filebot -script svn:amc /share/Qdownload/rtorrent/tvshows/filebot -non-strict --output /share/Qmultimedia --def "seriesFormat=Series/{n}/{n} Season {s}/{n}.{s.pad(2)}x{e.pad(2)}.{t}{'.'+lang.getDisplayName(Locale.ENGLISH)}" --def myepisodes=<user>:<password> --def xbmc=192.168.1.19 --conflict override


To automate this we use cron
To do that, edit the /etc/config/crontab file and add these lines

Code: Select all

0,30 * * * * /share/MD0_DATA/.qpkg/myscripts/pipeline.sh >> /share/MD0_DATA/.qpkg/myscripts/log.txt 2>&1


This command makes the pipeline.sh bash run every 30 minutes. This can be adjusted to your liking http://en.wikipedia.org/wiki/Cron
At the end I make STDOUT en STDERROR go to a log file so you can check progress and look at possible errors for debugging.

To refresh cron, execute these two lines separately

Code: Select all

crontab /etc/config/crontab
/etc/init.d/crond.sh restart




Automation for movies

Although this paragraph is unrelated to the main topic, I'd like to note that you can also use FileBot to automate your movie downloads. In my setup this is done semi-automatic meaning I still choose the torrent which I want and put them in the rtorrent/watch/movies folder (or rtorrent/watch/movies_getsubs folder)
After this FileBot takes over. I have 2 extra lines in the pipeline.sh file to process movie

Code: Select all

/share/MD0_DATA/.qpkg/Optware/usr/share/filebot/bin/filebot.sh -script svn:amc /share/Qdownload/rtorrent/complete/movies -non-strict --output /share/Qmultimedia --def "movieFormat=Movies/{n} [{y}]/{n} [{y}]{'.'+lang.getDisplayName(Locale.ENGLISH)}" --def xbmc=192.168.1.19
/share/MD0_DATA/.qpkg/Optware/usr/share/filebot/bin/filebot.sh -script svn:amc /share/Qdownload/rtorrent/complete/movies_getsubs -non-strict --output /share/Qmultimedia --def subtitles=en,nl --def "movieFormat=Movies/{n} [{y}]/{n} [{y}]{'.'+lang.getDisplayName(Locale.ENGLISH)}" --def xbmc=192.168.1.19


I won't go in to details what it does because it's quite specific to my preferences. Check documentation on FileBot for more information.



So, this is my entire setup. I know it looks quite elaborate but I can assure you, once it's set up, you will enjoy all the hours not being occupied with downloading tv shows manually.

Finally I'd like to thank the people behind all the programs I use for their great continuing work (rtorrent, FlexGet, FileBot, AutoSub). My tutorial is in fact just a how-to to get these programs working as a pipeline.

If anyone has any comments, questions or you need any help with the Python scripts, feel free to reply on this thread or send me a PM for more details.

Cheers!
Last edited by colicab on Sat Nov 28, 2015 7:56 pm, edited 19 times in total.
TS-659 Pro+ (mod 2GB RAM) | QTS 4.2.6 build 20180531 | 6 * 6TB WDC WD60EFRX RAID6

User avatar
schumaku
Guru
Posts: 43673
Joined: Mon Jan 21, 2008 4:41 pm
Location: Kloten (Zurich), Switzerland -- Skype: schumaku
Contact:

Re: Fully automate downloading TVshows using torrents

Post by schumaku » Wed Mar 06, 2013 6:48 am

colicab wrote:I'm probably one of a lot of people who puts some time in collecting new TV show episodes.


Curious - why not simply make use of the QNAP Download Station RSS feed feature, which is perfectly able to deal with RSS feeds containing torrents?

colicab
Know my way around
Posts: 105
Joined: Thu Jul 29, 2010 4:17 pm

Re: Fully automate downloading TVshows using torrents

Post by colicab » Wed Mar 06, 2013 7:54 am

I first tried using Download Station with showRSS+ but there were some problems with this setup.

First, you need something as showRSS to prefilter for your torrent files. But looking at the list of available torrents in the showRSS, there was a huge lack of 720p. Dailytorrents really has all of them. FlexGet let's you filter in the feed, something you can't do with DS.

Indeed, DS can get torrents from a RSS feed and yes, normally it can check if this one is already done so not to download the same episode twice.
However, once I moved the file and deleted the torrent, he refreshed the RSS feed and started downloading again. Don't know why. I thought that's what the status column in DS is for, but the refresh on this wasn't reliable, IMO.
FlexGet does this himself, he keeps a log of all episodes downloaded so he wont download 2 torrents from the same episode.

Also, there is no way for downstream applications to scan a watch folder with DS since incomplete and complete downloads are in the same folder.
So I guess you would manually have to move the finished ones for postprocessing, or wait for all to finish. But that's difficult to automate...
With Transmission you can at least change some settings like folder structure. Only downside is the lack of a good (any?) iPhone app for Transmission
TS-659 Pro+ (mod 2GB RAM) | QTS 4.2.6 build 20180531 | 6 * 6TB WDC WD60EFRX RAID6

bendsch
Know my way around
Posts: 229
Joined: Fri Oct 23, 2009 3:20 am

Re: Fully automate downloading TVshows using torrents

Post by bendsch » Wed Mar 06, 2013 8:39 am

why dont you use the transmission qpkg? it covers all the functionalities listed here + even more.
cheers.
NASmodesl: TS-459 Pro+ , TS-251
Firmware: 4.2.4

colicab
Know my way around
Posts: 105
Joined: Thu Jul 29, 2010 4:17 pm

Re: Fully automate downloading TVshows using torrents

Post by colicab » Wed Mar 06, 2013 9:16 am

Really? Some time ago I tried using subtitle downloader from transmission but it didn't download all my subtitles and only english ones.

Is there a renaming tool in Transmission comparable to filebot?


Nevertheless, I'm really satisfied with this setup.
True, it's some work but it's flexible at every step.
I had different experiences with the Transmission addons (beside flexget)
TS-659 Pro+ (mod 2GB RAM) | QTS 4.2.6 build 20180531 | 6 * 6TB WDC WD60EFRX RAID6

bendsch
Know my way around
Posts: 229
Joined: Fri Oct 23, 2009 3:20 am

Re: Fully automate downloading TVshows using torrents

Post by bendsch » Wed Mar 06, 2013 4:48 pm

ok, there is no renaming tool in the transmission package, but there is the possibility to add your own scripts via webinterface.

cheers, bendsch.
NASmodesl: TS-459 Pro+ , TS-251
Firmware: 4.2.4

colicab
Know my way around
Posts: 105
Joined: Thu Jul 29, 2010 4:17 pm

Re: Fully automate downloading TVshows using torrents

Post by colicab » Thu Mar 07, 2013 5:09 am

FileBot is so easy to set up, I wouldn't even start writing something of my own for the renaming.

It's really a very powerful and easy tool to use.
TS-659 Pro+ (mod 2GB RAM) | QTS 4.2.6 build 20180531 | 6 * 6TB WDC WD60EFRX RAID6

mctid82
Starting out
Posts: 39
Joined: Tue Nov 13, 2012 1:13 pm

Re: Fully automate downloading TVshows using torrents

Post by mctid82 » Thu Mar 07, 2013 8:19 am

I'm using transmission and torrentwatch-x... moves all my shows to the right folders for which XBMC then picks them up.

colicab
Know my way around
Posts: 105
Joined: Thu Jul 29, 2010 4:17 pm

Re: Fully Automated Download Center

Post by colicab » Fri Mar 08, 2013 9:44 pm

That sounds easier to set-up than my system :D

I looked at TorrentWatch-X but I was hesitant to use it because they will stop developing it unless someone takes over. So it's doubtful if any updates will come for TWX...
Also, I had some specific requirements about file structure, downloading subtitles. That's why I set it up like this.
TS-659 Pro+ (mod 2GB RAM) | QTS 4.2.6 build 20180531 | 6 * 6TB WDC WD60EFRX RAID6

jeroenqui
New here
Posts: 4
Joined: Tue Feb 26, 2013 8:11 pm

Re: Fully Automated Download Center

Post by jeroenqui » Tue Mar 19, 2013 11:47 pm

Thanks for this awesome guide.

Little question, can you post the contents of your second (dutch) config.privateproperties (AutoSub2) file?
I'm a bit confused which values to alter since it contains both references to dutch and english.
Even if it's just your setup, i'll know which values to alter then.
I'm dutch aswell, and really glad you wrote this to cover up multiple languages.

Thanks in advance for replying.

colicab
Know my way around
Posts: 105
Joined: Thu Jul 29, 2010 4:17 pm

Re: Fully Automated Download Center

Post by colicab » Wed Mar 20, 2013 9:17 pm

Code: Select all

[webserver]
username =
webserverport = 8084
webroot =
webserverip = 0.0.0.0
password =

[logfile]
loglevel = info
lognum = 1
loglevelconsole = error
logsize = 1000000

[config]
subeng = English
notifynl = True
logfile = AutoSubService.log
checksub = 21600
scandisk = 3600
minmatchscore = 8
checkrss = 60
rootpath = /share/MD0_DATA/Qmultimedia/transmission/dutch_subs
minmatchscorerss = 14
notifyen = True
fallbacktoeng = False
postprocesscmd =
path = /share/MD0_DATA/.qpkg/AutoSub2
configversion = 2
subnl = Dutch
launchbrowser = True
downloadeng = False

[notify]
notifyprowl = False
growlport = 23053
prowlpriority = -2
prowlapi =
growlpass = mysecretpassword
mailauth =
nmaapi = API key
mailusername = example@gmail.com
growlhost = 127.0.0.1
mailfromaddr = example@gmail.com
notifygrowl = False
mailsubject = Subs info
mailsrv = smtp.gmail.com:587
notifytwitter = False
twittersecret = token secret
notifynma = False
mailpassword = mysecretpassword
notifymail = False
mailencryption = TLS
mailtoaddr = example@gmail.com
twitterkey = token key




It's the same as the other config file, but with 'fallbacktoeng' and 'downloadeng' set to False
And the path set to the second AutoSub folder


Cheers
TS-659 Pro+ (mod 2GB RAM) | QTS 4.2.6 build 20180531 | 6 * 6TB WDC WD60EFRX RAID6

jeroenqui
New here
Posts: 4
Joined: Tue Feb 26, 2013 8:11 pm

Re: Fully Automated Download Center

Post by jeroenqui » Thu Mar 21, 2013 5:16 pm

Thanks for your reply.
Still something weird, it the dutch_subs dir it downloads both english and dutch subs. Is that normal?

2013-03-21 10:17:21,735 INFO downloadSubs: DOWNLOADED: /share/MD0_DATA/Download/transmission/dutch_subs/How.I.Met.Your.Mother.S08E19.720p.HDTV.X264-DIMENSION.English.srt
2013-03-21 10:17:21,104 INFO checkSub: The episode How I Met Your Mother - Season 08 Episode 19 has a matching subtitle on bierdopje, downloading it!
2013-03-21 10:17:20,154 INFO downloadSubs: DOWNLOADED: /share/MD0_DATA/Download/transmission/dutch_subs/How.I.Met.Your.Mother.S08E19.720p.HDTV.X264-DIMENSION.Dutch.srt
2013-03-21 10:17:19,528 INFO checkSub: The episode How I Met Your Mother - Season 08 Episode 19 has a matching subtitle on bierdopje, downloading it!
2013-03-21 10:17:14,795 INFO getShowid: u'How I Met Your Mother' added to cache with 4287

even though I set the properties 'fallbacktoeng' and 'downloadeng' to False in config.privateproperies and restarted the autosub2 daemon.

[EDIT]
I rebooted the NAS and tried again, this time it worked! Might want to add that to your guide.
[/EDIT]

Razorax
New here
Posts: 8
Joined: Sun Sep 14, 2008 5:33 pm

Re: Fully Automated TV show Download Center

Post by Razorax » Sat Apr 20, 2013 1:14 am

Awesome post!
Thanks a bunch for all the work

colicab
Know my way around
Posts: 105
Joined: Thu Jul 29, 2010 4:17 pm

Re: Fully Automated TV show Download Center

Post by colicab » Sat May 18, 2013 8:51 pm

As you might have noticed, http://www.dailytvtorrents.org/ is down.

For the moment I'm using the RSS feed from the BT-Chat.com site. I only started using it today so I'll check the next week if it works as good as dailytvtorrents.

[EDIT] changed my tutorial using the rss feed from kat.ph
TS-659 Pro+ (mod 2GB RAM) | QTS 4.2.6 build 20180531 | 6 * 6TB WDC WD60EFRX RAID6

colicab
Know my way around
Posts: 105
Joined: Thu Jul 29, 2010 4:17 pm

Re: Fully Automated TV show Download Center

Post by colicab » Wed Jan 15, 2014 1:26 am

I completely changed my setup.

For TV shows: SickBeard-TPB edition + Transmission + AutoSub
For Movies: CouchPotato2 + Transmission
TS-659 Pro+ (mod 2GB RAM) | QTS 4.2.6 build 20180531 | 6 * 6TB WDC WD60EFRX RAID6

Post Reply

Return to “Users' Corner”