I'm probably one out of a lot of people who puts a considerable amount of of time in collecting new TV show episodes.
Having a collection of 20+ ongoing TV shows it became a daily pass-time to check for new episodes, finding the correct subtitles, renaming and moving the episodes to my multimedia library.
Reading about a lot of cool handy programs, I decided to automate the process, merely by concatenating these programs.
What I propose here is:
- a fully automated TV show downloader
- totally free using torrents (opposed to Usenet and Sickbeard)
- everything is done on the QNAP, independent from user input
Basically what my setup does is:
1) Find new TV show torrent files using an RSS feed and FlexGet
2) Download torrent with rtorrent
3) Download English and Dutch subtitles using AutoSub (from Bierdopje.com)
4) Rename the episodes and move them to the multimedia library using FileBot
5) Rescan XBMC library on an HTPC
Some notes:
i) My QNAP setup at the time of writing is: QNAP TS-659 Pro+ running firmware 4.0.2
ii) Your folder structure will probably differ from mine so make changes accordingly: all my shares are under /share/MD0_DATA/. Therefor I advise not simply to copy-paste any code or file content before having inspected it.
iii) To do this setup, you will do some Linux command line: working with ssh,vi and browsing through the file structure. Also I working knowledge of Python basics might come in handy, although the scripts I propose for moving files etc. can be achieved with any other scripting language of your choosing.
What you need before starting
A) Install these QPKG packages from App Center:
- Optware (0.99.163)
- JRE (build 1.7.0_40-b43, headless): follow the first reply in this thread http://forum.qnap.com/viewtopic.php?f=163&t=47652
- rtorrent (0.9.3): manual installation. Download from http://pool.qnapclub.pl/projects/packag ... /releases/
B) Download these files:
- AutoSub: Download the non-Windows one https://code.google.com/p/auto-sub/downloads/list note:
I'm using the latest version: 0.5.8
- FileBot: Choose one of the Embedded Linux packages, depending on your QNAP model http://filebot.sourceforge.net/#download.
Using the latest version: 3.62
C) Install python and setup tools on your QNAP
ssh into your QNAP
Code: Select all
ipkg install python26
ipkg install py26-setuptools
ipkg install py26-cheetah
Cheetah is required to get AutoSub running and setuptools are required to install FlexGet (see below)
D) Make a directory to put all your scripts and config files
ssh into your QNAP
Code: Select all
cd /share/MD0_DATA/.qpkg/
mkdir myscripts
Setting up rtorrent (Bit-torrent client)
I choose rtorrent for one specific reason: multiple watch-download directory pairs. If you make a subdir in the rtorrent/watch folder, it will create the same subdir in downloads/ and complete/ so you can direct torrent traffic. By making these subdirs you can apply custom-made downstream processing for any of the folders in the rtorrent/complete folder. Very convenient IMO...
Setting it up is easy enough. After manually installing the qpkg in App Center (step A above), it's essentially done. Off course you can adjust a lot of settings which is just an added bonus for getting rtorrent. I only adjust the download speed to unlimited, the rest I keep default.
After installation rtorrent makes a folder tree for downloading in Qdownload/rtorrent. To segregate my torrent files, I made a couple of subdirs in the rtorrent/watch folder:
-tvshows: all torrent files from FlexGet go here (see later in the how-to)
-movies: movies with subtitles
-movies_getsubs: movie torrent files without subtitles
-no_action: torrent files that I will process manually
I also made a folder rtorrent/tvshows with some subdirs:
- subtitles: after episode is download file gets moved from rtorrent/complete/tvshows to here to get subtitiles
- filebot: after subtitle is download, the file gets moved from subtitles to here to get renamed by FileBot and moved to the series library
Now we have our Bittorrent set up, we have to get our torrent files. We'll use FlexGet for this
Flexget
Flexget downloads torrents from an RSS feed. These torrents file will be put in the watch folder of rtorrent (/share/MD0_DATA/Qdownload/rtorrent/watch/tvshows)
I got the installation info from: http://flexget.com/wiki/InstallWizard/QNAP
For the installation, ssh into your QNAP and give these commands one by one:
Code: Select all
easy_install-2.6 flexget
Code: Select all
ImportError: No module named pkg_resources
Code: Select all
curl http://python-distribute.org/distribute_setup.py | python2.6
Code: Select all
flexget
Next step is to configure FlexGet for your favourite tv shows. You have to make a config.yml file for this. Make it in a text editor on your PC or with vi or something on your QNAP.
The location on your QNAP to put it is /opt/local/
On the FlexGet site, you can get a lot (really a lot) of information how to do make this file: http://flexget.com/wiki/Configuration. To make this file will take some reading into, especially if you want to additional features.
Basically, config.yml is a file that tells FlexGet where to look for new torrents (input; RSS feed), which ones to take (filter; series list) and where to put the torrent (output; watch folder). There are a lot of different ways to make this file. I'll just show you mine and explain what's is in it.
I have 2 different tasks 'seed_series_db' and 'kickass'. Kickass is my default tasks: this one is done automatically everytime flexget runs. Let's first take a loot at that one.
Code: Select all
tasks:
seed_series_db:
import_series:
from:
thetvdb_favorites:
account_id: <your TVDB ID>
find:
regexp: .*(avi|mkv|mp4)$
path: /share/MD0_DATA/Qmultimedia/Series
recursive: yes
manual: yes
# manual run: flexget --task seed_series_db --disable-advancement --learn
kickass:
content_filter:
require: '*.mkv'
reject: '*.rar'
download: /share/Qdownload/rtorrent/watch/tvshows
import_series:
from:
thetvdb_favorites:
account_id: <your TVDB ID>
settings:
upgrade: yes
qualities:
- 1080p webdl
- 720p webdl
series:
- NCIS:
exact: yes
rss: http://kickass.to/tv/?rss=1
With the rss: I tell Flexget to look at the RSS feed from kickasstorrents to retrieve torrent files. Off course you can use any rss feeds for tv shows, I'm just inclined to use kickasstorrents', my favorite torrent site.
For the filter, I use import_series which accesses my TvDB account to learn which series I'm watching. If a torrent for a certain episode in the rss matches my favorite series, it gets downloaded. Also, flexget remembers your progress, so it wont download an episode more than once, normally.
With the option 'settings' I specify the preferred video quality, mine is either 720p WEB-DL or 1080p WEB-DL. The 720p version is available sooner so it gets downloaded first. With the upgrade option, however, FlexGet knows that ultimately I want the 1080p version so it will download this one too once it arrives on kickasstorrents. I have 1 extra filter because I don't want to download rar files, only mkv's to prevent the additional step of building up the zipped file.
One caveat perhaps which you have to take into account is in the case of partial matching series. I have this with NCIS, which is a favorite in my TvDB account, NCIS LA. However since NCIS partially matches NCIS LA this one will get downloaded too. This behavior might have some advantages, but in my case I have to tell Flexget I want NCIS to match exactly to the series title; this is an override for the TvDB input. Finally, with download: I specify the watch folder .
The 2nd task is not for downloading torrents but to initialize the FlexGet database, a list of which episode are already downloaded. This comes in handy If you already have a series library before using FlexGet. Or you need to reinstall for some reason. By giving the following command FlexGet will learn which episode are already in your library (off course, your path should be altered to your own media library)
Code: Select all
flexget --task seed_series_db --disable-advancement --learn
Code: Select all
ln -s /opt/local/config.yml /share/MD0_DATA/.qpkg/myscripts/config.yml
For now we will start by making a bash file (pipeline.sh) in the myscripts/ directory and put this in:
Code: Select all
#!/bin/sh
/opt/bin/python2.7 /opt/local/bin/flexget
Code: Select all
chmod +x pipeline.sh
(If you don't care for subtitles, you can skip the next step discussing AutoSub where we will automate the downloading of subtitles)
Getting subtitles with AutoSub
To download subtitles I use AutoSub from Bierdopje.com. This website both has the English as well as the Dutch subtitles.
The English subtitles are almost immediately available when the new episode is, but the Dutch subtitle takes a couple of days longer before you can download it. Being inpatient I don't want to wait for the Dutch subtitle to watch the episode, so I wrote 2 Python scripts to move the episode once the English subtitle is available. Put these in your myscripts/ folder and make them executable
movescript.py
Code: Select all
'''
This is the top-level script to process new TV show episodes
It uses the object.py file to create Episode objects
'''
import os
from media import Episode
rtorrent = '/share/Qdownload/rtorrent/'
for root, dirs, files in os.walk(rtorrent):
for file in files:
file = os.path.join(root,file)
if file.endswith('.mkv'):
file = Episode(file)
#print(file)
file.run()
Code: Select all
import os
import shutil
import re
class Episode:
def __init__(self, fullpath, sim=0):
self.fullpath = fullpath # the full path including, episode file
self.name = os.path.basename(fullpath) # name of episode file
self.upperdir = os.path.dirname(fullpath) # directory of episode file (may be the completed map)
self.platform = None
self.strip = self.fullpath[:-3]
self.eng, self.nl = self.strip + 'English.srt', self.strip + 'Dutch.srt'
if sim:
self.platform = '/share/Qdownload/rtorrent_bis/'
print('TEST mode...')
else:
self.platform = '/share/Qdownload/rtorrent'
self.values = [os.path.join(self.platform,folder) for folder in ['complete/tvshows','tvshows/subtitles', 'tvshows/filebot']]
self.keys = ['completed', 'subtitles', 'rename']
self.folders = dict(zip(self.keys,self.values))
#print self.folders
self.excluded = ['tvshows', 'rtorrent_bis', 'rtorrent', 'Qdownload', 'MD0_DATA', 'share', '/']
def failsafe(self,check):
for dir in self.excluded:
if re.search(dir, check, re.I): return None
else: return True
def clean(self):
if self.failsafe(os.path.basename(self.upperdir)):
for folder, subfolder, files in os.walk(self.upperdir):
ext = [os.path.splitext(file)[1] for file in files]
if '.mkv' in ext:
pass
#print("still mkvs present in directory, wont delete")
else:
#print("Deleting '%s'" % (self.upperdir,))
shutil.rmtree(self.upperdir)
else:
#print("you were about to delete '%s'. Thats's not allowed!!" % (self.upperdir,))
pass
def __str__(self):
return "-File:'%s'\n Stripped name:'%s'\n English sub:'%s'\n Dutch sub:'%s'\n Full path:'%s'\n Upper dir:'%s'\n " \
% (self.name, self.strip,self.eng, self.nl, self.fullpath, self.upperdir)
def makeDummy(self):
dummy = os.path.join(self.folders['subtitles'], self.name)
open(dummy,"w")
def customprint(self,file):
print("Moving '%s' to rename/ folder" % (file))
def run(self):
if re.match(self.folders['completed'], self.upperdir, re.I):
if re.search('1080p', self.name, re.I):
m = re.search ('(.*)S\d{2}E\d{2}', self.name, re.I).group()
for dir,subdir,files in os.walk(self.folders['subtitles']):
for file in files:
if (re.search(m, file, re.I) and re.search('720p', file)):
#print("Delete %s" % (file,))
os.remove(os.path.join(dir,file))
shutil.move(self.fullpath, self.folders['subtitles'])
#print("Moving '%s' to subtitles folder" % (self.name,))
self.clean()
if self.upperdir == self.folders['subtitles']:
if os.path.getsize(self.fullpath) == 0: ## if True: the original mkv and english subs are already at destination, see below
if os.path.exists(self.nl):
shutil.move(self.nl, self.folders['rename'])
os.unlink(self.fullpath)
if os.path.exists(self.eng): os.unlink(self.eng) ## Shoud normally always exist; extra failsafe
elif os.path.exists(self.eng):
if os.path.exists(self.nl):
for x in [self.fullpath, self.eng, self.nl]:
shutil.move(x, self.folders['rename'])
#print("Moving mkv, english and dutch subtitle of %s to filebot folder" % (self.fullpath,))
else:
for x in [self.fullpath, self.eng]:
shutil.move(x, self.folders['rename'])
self.makeDummy()
#print("Moving mkv and english subtitle of %s to filebot folder\nMake dummy mkv file" % (self.fullpath,))
if __name__ == '__main__':
for root, dirs, files in os.walk('/share/Qdownload/rtorrent_bis'):
for file in files:
file = os.path.join(root,file)
if file.endswith('.mkv'):
file = Episode(file,sim=1)
#print(file)
file.run()
I'll just say briefly what these scripts do. First it looks in rtorrent/complete/tvshows to see if a new episode (<episode>.mkv) is downloaded which it will move to rtorrent/tvshows/subtitles. It will also remove clutter files (eg .nfo files) and remaining folders.
rtorrent/tvshows/subtitles is the watch directory for AutoSub: when subtitles are available, they get put in here. If the English subtitle is downloaded (<episode>.English.srt) it will move the subtitle and the episode to rtorrent/tvshows/filebot. Waiting for the Dutch subtitle, it will make a 'dummy' empty mkv file with the same name so AutoSub is still looking for the Dutch Subtitle. Once this one is available and downloaded, it will also move this one to rtorrent/tvshows/filebot and remove the dummy file (as well as the English subtitle because AutoSub will keep on downloading this one).
Additionally, because every episode gets downloaded twice (first 720p then 1080p), it will delete the 720p data in the AutoSub directory when the 1080p version is downloaded and moved to that folder. That way all 720p gets replaced by 1080p.
Again to automate, we put another command line in the pipeline.sh file
Code: Select all
/opt/bin/python2.7 /share/MD0_DATA/.qpkg/myscripts/movescript.py
Now we setup AutoSub. I mainly used this topic from the QNAP forum to install it http://forum.qnap.com/viewtopic.php?p=298485
Put the AutoSub folder you downloaded previously in your file directory as
Code: Select all
/share/MD0_DATA/.qpkg/AutoSub
Put this one in the AutoSub folder. Off course this one has to be edited according to your setup
Mine looks like this:
Code: Select all
#! /bin/sh
### BEGIN INIT INFO
# Provides: Auto-sub application instance
# Required-Start: $all
# Required-Stop: $all
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: starts instance of Auto-sub
# Description: starts instance of Auto-sub using start-stop-daemon
### END INIT INFO
############### EDIT ME ##################
QPKG_NAME=AutoSub
QPKG_DIR=
DAEMON=/opt/bin/python2.6
DAEMON_OPTS=" /share/MD0_DATA/.qpkg/AutoSub/AutoSub.py -c /share/MD0_DATA/.qpkg/AutoSub/config.properties -d -l"
case "$1" in
start)
echo "Starting $QPKG_NAME"
if [ `/sbin/getcfg ${QPKG_NAME} Enable -u -d FALSE -f /etc/config/qpkg.conf` = UNKNOWN ]; then
/sbin/setcfg ${QPKG_NAME} Enable TRUE -f /etc/config/qpkg.conf
elif [ `/sbin/getcfg ${QPKG_NAME} Enable -u -d FALSE -f /etc/config/qpkg.conf` != TRUE ]; then
echo "${QPKG_NAME} is disabled."
exit 1
fi
${DAEMON} ${DAEMON_OPTS}
;;
stop)
echo "Stopping $QPKG_NAME"
for pid in $(/bin/pidof python); do
/bin/grep -q "AutoSub.py" /proc/$pid/cmdline && /bin/kill $pid
done
/bin/sleep 2
;;
restart|force-reload)
echo "Restarting $QPKG_NAME"
$0 stop
$0 start
;;
*)
N=/etc/init.d/$QPKG_NAME
echo "Usage: $N {start|stop|restart|force-reload}" >&2
exit 1
;;
esac
exit 0
Now make a config.properties file and also put this in the AutoSub folder.
Code: Select all
[webserver]
username =
webserverport = 8083
webroot =
webserverip = 0.0.0.0
password =
[logfile]
loglevel = info
lognum = 1
loglevelconsole = error
logsize = 1000000
[config]
subeng = English
notifynl = True
logfile = AutoSubService.log
checksub = 28800
scandisk = 3600
minmatchscore = 8
checkrss = 900
rootpath = /share/MD0_DATA/Qdownload/rtorrent/tvshows/subtitles
minmatchscorerss = 14
notifyen = True
fallbacktoeng = True
postprocesscmd =
path = /share/MD0_DATA/.qpkg/AutoSub
configversion = 2
subnl = Dutch
launchbrowser = True
downloadeng = True
[notify]
mailencryption = TLS
mailtoaddr = example@gmail.com
growlport = 23053
notifytwitter = False
notifyprowl = False
growlpass = mysecretpassword
twittersecret = token secret
mailusername = example@gmail.com
twitterkey = token key
mailsrv = smtp.gmail.com:587
growlhost = 127.0.0.1
notifynma = False
mailsubject = Subs info
mailfromaddr = example@gmail.com
notifygrowl = False
mailauth =
mailpassword = mysecretpassword
prowlpriority = -2
nmaapi = API key
notifymail = False
prowlapi =
Again, it's advisable to make a softlink for this file in your myscripts/ folder. Add this to the file:
Code: Select all
[AutoSub]
Name = AutoSub
Version = 0.5.8
Enable = TRUE
QPKG_File = Autosubdummy.qpkg
Date = 2012-12-09
shell = /share/MD0_DATA/.qpkg/AutoSub/AutoSub.sh
Install_Path = /share/MD0_DATA/.qpkg/AutoSub
WebUI = /
Author = Bas
Web_port = 8083
FileBot
FileBot is the last part of our pipeline. It's a program to rename and move our files to the multimedia library.
Up until now we have the mkv episode file and the subtitle files going to the rtorrent/tvshows/filebot folder. So this is the input folder for FileBot.
Browse to the folder with the FileBot install file and type
Code: Select all
ipkg install filebot_3.4_i686.ipk
Check these pages to make commands to your liking:
http://filebot.sourceforge.net/naming.html
http://filebot.sourceforge.net/forums/viewtopic.php?t=2
http://filebot.sourceforge.net/cli.html
Again as for FlexGet, this takes some reading if you want to further customize this part.
Here is mine:
Code: Select all
/opt/usr/bin/filebot -script svn:amc /share/Qdownload/rtorrent/tvshows/filebot -non-strict --output /share/Qmultimedia --def "seriesFormat=Series/{n}/{n} Season {s}/{n}.{s.pad(2)}x{e.pad(2)}.{t}{'.'+lang.getDisplayName(Locale.ENGLISH)}" --def myepisodes=<user>:<password> --def xbmc=192.168.1.19 --conflict override
- Rename files to look like this: "Breaking Bad.01x01.Pilot.mkv" "Breaking Bad.01x01.Pilot.English.srt" "Breaking Bad.01x01.Pilot.Dutch.srt"
- Put it in a my multimedia file structure looking like Qmultimedia --> Series --> Breaking Bad --> Breaking Bad Season 1 --> Breaking Bad.01x01.Pilot.mkv ...
- Update MyEpisodes.com account (this option seems deprecated)
- Update XBMC library on my HTPC (you need to know the IP address). Of course you have to allow XBMC to do this: System > Network > Allow programs on other systems to control XBMC <enable>
- Override existing episode files (720p quality) if it is already present in the multimedia library
Once you're finished with drafting the command, copy it on a new line in the pipeline.sh file.
Crontab
As said, as a last step we need to automate our pipeline. We already tied all the pieces together by making the pipeline.sh file. AutoSub and rtorrent don't need to be automated manually, they reside in the App Center and will run after you have restarted your QNAP.
The pipeline.sh should look like this
Code: Select all
#!/bin/sh
/opt/bin/python2.7 /opt/local/bin/flexget
/opt/bin/python2.7 /share/MD0_DATA/.qpkg/myscripts/tvshows.py
/opt/usr/bin/filebot -script svn:amc /share/Qdownload/rtorrent/tvshows/filebot -non-strict --output /share/Qmultimedia --def "seriesFormat=Series/{n}/{n} Season {s}/{n}.{s.pad(2)}x{e.pad(2)}.{t}{'.'+lang.getDisplayName(Locale.ENGLISH)}" --def myepisodes=<user>:<password> --def xbmc=192.168.1.19 --conflict override
To do that, edit the /etc/config/crontab file and add these lines
Code: Select all
0,30 * * * * /share/MD0_DATA/.qpkg/myscripts/pipeline.sh >> /share/MD0_DATA/.qpkg/myscripts/log.txt 2>&1
At the end I make STDOUT en STDERROR go to a log file so you can check progress and look at possible errors for debugging.
To refresh cron, execute these two lines separately
Code: Select all
crontab /etc/config/crontab
/etc/init.d/crond.sh restart
Automation for movies
Although this paragraph is unrelated to the main topic, I'd like to note that you can also use FileBot to automate your movie downloads. In my setup this is done semi-automatic meaning I still choose the torrent which I want and put them in the rtorrent/watch/movies folder (or rtorrent/watch/movies_getsubs folder)
After this FileBot takes over. I have 2 extra lines in the pipeline.sh file to process movie
Code: Select all
/share/MD0_DATA/.qpkg/Optware/usr/share/filebot/bin/filebot.sh -script svn:amc /share/Qdownload/rtorrent/complete/movies -non-strict --output /share/Qmultimedia --def "movieFormat=Movies/{n} [{y}]/{n} [{y}]{'.'+lang.getDisplayName(Locale.ENGLISH)}" --def xbmc=192.168.1.19
/share/MD0_DATA/.qpkg/Optware/usr/share/filebot/bin/filebot.sh -script svn:amc /share/Qdownload/rtorrent/complete/movies_getsubs -non-strict --output /share/Qmultimedia --def subtitles=en,nl --def "movieFormat=Movies/{n} [{y}]/{n} [{y}]{'.'+lang.getDisplayName(Locale.ENGLISH)}" --def xbmc=192.168.1.19
So, this is my entire setup. I know it looks quite elaborate but I can assure you, once it's set up, you will enjoy all the hours not being occupied with downloading tv shows manually.
Finally I'd like to thank the people behind all the programs I use for their great continuing work (rtorrent, FlexGet, FileBot, AutoSub). My tutorial is in fact just a how-to to get these programs working as a pipeline.
If anyone has any comments, questions or you need any help with the Python scripts, feel free to reply on this thread or send me a PM for more details.
Cheers!