WeeWX on QNAP

[Sar6e]

I wanted to run Weewx on my QNAP TS119 to connect to my Fine Offset WH1080 weather station. This is how I achieved that.

Hardware

I purchased a 'Wireless Weather Sation' from Jaycar (http://www.jaycar.com.au/productView.asp?ID=XC0348). This station is sold under a variety of names but is generally referred to in most cases as a Fine offset WH1080. The 1080 has a USB port that allows it to be connected to a computer to download the data from the station. It is supplied with some Windows software but I don't have a PC.

The best solution for me was to connect it to my QNAP NAS. This acts as my home 'server' and is always on, making it the best place to connect the weather station to.

Dead Ends

The QNAP uses a highly customised version of Linux with a graphical user interface to configure and operate it. This is the main selling point for me for whom the command line is a foreign country. The QNAP system is extensible with its own package management system (QPKGs) which makes adding additional software literally as simple as a single click. This is my kind of Linux!

So first port of call was to find weather station software that was available as a QPKG. I downloaded and installed wfrog which installed OK. It relied on another package called pywws to actually get the data with wfrog primarily acting as a presentation layer (from what I understand). Despite great support from the author of the packages I never could get pywws working.

Software

The user community for both wfrog and pywws is quite small. A number of people pointed me at weewx which has a comparatively much larger support base and I knew people running this. I decided weewx was the best bet for the software part of my solution.

And the Bad News...

The downside of this decision was there is no QPKG. The worse news was that a number of posts on the weewx user forums said that because the QNAP was such a specialised Linux environment it would simply not be possible to get weewx working on it.

The 'smart money' seemed to be to grab a Raspberry Pi or a Beaglebone Black and run up a standard Linux system to act as a dedicated controller for the weather station. I've been itching for an excuse to buy one of these mini marvels for a while so I wasn't averse to that.

But I really like the idea of getting some more value from the QNAP system. I had it, so what was the harm in having a go at getting it running?

The Journey

As Linux novice fearful of the command line, I found the weewx documentation to be really quite good. Rather than the usual instructions laden with assumed knowledge, the document contained clear commentary on what needed to be achieved in each step, along with the actual commands required with variants for a range of Linux distributions - but sadly not for the QNAP. But it did give me a really good idea of what I needed to achieve to get this going.

QNAP and QPKGS

The first steps were to turn on embedded capabilities in the QNAP and to grab the things that were available as QPKGs. I opted to use the MySQL database for my weather data. The default is SQLite which doesn't require any setup. I kinda liked the idea of being able to use PHPMyAdmin to look at the data directly so opted for MySQL. The weewx system is a Python application so naturally enough Python is required. All these things are able to be installed and setup from the QNAP graphical interface. Yay!

MySQL
Enabling MySQL is as simple as logging into the QNAP QTS Desktop, opening the Control Panel, navigating to Applications and selecting MySQL Server. Click the checkbox to enable MySQL Server.

PHPMyAdmin
PHPMyAdmin gives you a graphical UI to manage your databases, permissions and you can execute queries if you can manage the odd “select * from...” In the QTS Desktop, open the App Centre and find the PHPMyAdmin QPKG and click “Add to QTS'. The package will download and install.

Python
Weewx requires Python. As at the time of writing, Python 3 is not supported. Similarly, the Python QPKG installs version 2.7 so this aligns nicely. Find the Python QPKG and click 'Add to QTS'. The package will download and install.

Pre-PreRequisites

The generally helpful weewx documentation lists the prerequisites including a number of Python modules. In practice I found a couple of these modules would not install without a compiler. So I first needed to install the gcc compiler before these would install successfully.

GCC Module
A compiler is needed to install the Cheetah and PIL modules (from memory). The way I installed he GCC compiler was from an IPKG. There is a PQKG that allows you to use IPKGs. The first step is to install the Optware QPKG. Open the App Centre on the QTS Desktop and find the Optware package and click 'Add to QTS'. It will download and install.

Click the Optware IPKG icon on the QTS Desktop. This will open the IPKG system. In the “Filter' box type 'gcc' and click Submit. To the left of the gcc IPKG you will see the option to 'Install' Click this to install the compiler.

We're Not in Kansas Anymore Dorothy

So here ends the part where I was in my comfort zone. The next steps are executed from the command line. I used a terminal window on OS X to connect to the QNAP. When your terminal window opens, connect to the QNAP using the following command

Code: Select all

ssh admin@nnn.nnn.nnn.nnn 

where the last part is the IP address of the QNAP. If you don't know this you can find the IP address from the Network page in System Settings of the control Panel.

You will be prompted for the admin password and then you will see the unnerving, vacuous blinking of the command line prompt “#”; telling you nothing, taking you nowhere.

When you are done you can type 'exit' to disconnect.

PreRequisites

To PIP or not to PIP
Python has a number of package management system designed to make it simple to download and install Python modules. The most current method from what I read is PIP. I used these nice instructions to get PIP setup: http://rdkevinchien.blogspot.com.au/201 ... oping.html

As it turned out I was never able to get modules installed using PIP. PIP enforces an SSL connection for downloading the package which is a good thing. The self issued certificate on the QNAP meant the download always failed. It is beyond me to sort out the certificate thing and I couldn't find a pip option or switch to override the secure download.

There is a nice set of instructions for purchasing and installing a certificate for your QNAP. See my references at the end. This would probably make the PIP installs works and will be bit simper if you want to go down that path.

My solution was to use the commands to download and install pip as a template to download the modules manually and build and install them manually.
I used the admin home folder to store the downloads into.

Code: Select all

cd /share/homes/admin

I used the Python module pages to grab the URL to download each module. The Python Package Index allows you to search by module name: https://pypi.python.org/pypi

For example the configobj module page is: https://pypi.python.org/pypi/configobj/5.0.5 From this page you can find the download link and copy the link address. Paste the link address into the download commands below to make it easier and avoid messing about retyping the url without making an error.

I downloaded, unzipped, built and installed each of the module dependencies in turn.

Configobj

First download

Code: Select all

wget https://pypi.python.org/packages/source/c/configobj/configobj-5.0.5.tar.gz#md5=00af246754b4bb41a4e9c625c60dca5c --no-check-certificate

Extract the archive

Code: Select all

tar xvfz configobj-5.0.5.tar.gz

Change to the folder where the file was extracted.

Code: Select all

cd configobj-5.0.5

Build the module

Code: Select all

python setup.py build

Install the module

Code: Select all

python setup.py install

Repeat for the remaining dependencies listed in the weewx documentation: http://www.weewx.com/docs/setup.htm

I installed the following modules:

* configobj
* cheetah
* pil
* pyusb
* pyephem

Nearly There

I needed just a couple of last steps to finally get things running.

The MySQL Gotcha
Walking the path less travelled had one last little bump in the road that needed to be ironed out. I was following the pip install instructions in the weewx documentation. When I tried to run weewx for the first time it failed with an error message saying mysqldb module not found.

No dramas I thought - feeling quite confident have smashed all the other module installs - once I finally sorted my workaround method out. After downloading and expanding the module as expected, the build command failed with an error saying a config was not found. I had horrible premonitions about needing to manually configure a build file of some sort. Fortunately MySQL Boy had solved this particular problem. Just a path problem: http://blog.mysqlboy.com/2010/08/instal ... odule.html

This command let me build and install the mysqldb module successfully

Code: Select all

export PATH=$PATH:/usr/local/mysql/bin

Download and Install Weewx

Now I was able to download and install weewx as per the standard documentation.
http://www.weewx.com/docs/setup.htm

I found the weewx documentation explaining the weewx.conf file clear and straight forward. Similarly the documentation covering configuring the weewx outputs is generally clearly explained.

When I first saw the lines of data from the weather station scrolling up the terminal window I thought it was a problem and weewx was writing to screen instead of the database. A quick peek at the tables confirmed in MySQL using the PHPMyAdmin confirmed the data was writing there. So I concluded the writing to screen was normal behaviour.

It will take a few minutes but you will be able to point your browser at http://nnn.nnn.nnn.nnn/weewx and see your weather data. The nnn.nnn.nnn.nnn part is the IP address of the QNAP. It was great to see this page populated with data from my weather station!

I kept the terminal window with weewx running in it open on the Mac for a few days as I wasn't game to just close it in case it stopped the process. It eventually disconnected and the web pages keep updating so it seems OK to just close the terminal window once weewx is up and running.

Apache Config and REstart
The Weewx documentation provides instructions to get weewx talking to the web server (Apache) in the case it was not installed from a package. This was certainly the case for me. These instructions needed some slight modifications to reflect the location of apache on the QNAP and a different commend to restart apache.

The command in the document:

Code: Select all

cp util/apache/conf.d/weewx.conf /etc/apache2/conf.d

Needs to be:

Code: Select all

cp util/apache/conf.d/weewx.conf /etc/config/apache/conf.d

The equivalent commas to restart apache on the QNAP is:

Code: Select all

/etc/init.d/Qthttpd.sh restart

Outstanding Tasks

I have the data coming off the weather station onto the QNAP and an elementary web page displaying our weather data. There are still quite a few tasks still to sort out.

Autorun
So far I have just started weewx from the command line. If the QNAP is restarted I will manually need to restart it. Adding things to start on boot on the QNAP is not straightforward. It looks like I'll need to declare a dummy QPKG as the recommended solution form QNAP but that looks like another journey of discovery...
http://wiki.qnap.com/wiki/Running_Your_ ... at_Startup

Graphs
None of the graphs on the web page are showing. I thought this may be to do with a font specified in the skin.conf and not on the QNAP. I have downloaded the required font but still not getting any plots.

Publish to Web
So far the web page is generated to the QNAP. Weewx provides a report to publish to FTP to make our weather data available outside our network. Need to configure the ftp report options to make this happen.

Publish to Weather Underground and ...
Weewx also allows data from my weather station to be published to crowd sourced weather data systems like Weather Underground and WOW. I've setup a Weather Underground account and station. Need to configure to publish to that account.

References and Recognition

I am nowhere near enough knowledgeable enough to have solved any of this myself. All I have done is piece together a lot of disparate threads leveraging the knowledge and contributions of people who actually know what they are doing. The following sources were absolutely vital in me nutting this out. Thanks to all the authors!

PIP Setup on QNAP
http://rdkevinchien.blogspot.com.au/201 ... oping.html

Workaround for PIP Download Problem
http://gleenders.blogspot.com.au/2014/0 ... p-nas.html

MySQLDB Build Fail
http://blog.mysqlboy.com/2010/08/instal ... odule.html

QNAP Autorun
http://wiki.qnap.com/wiki/Running_Your_ ... at_Startup

[Toxic17]

Thanks for the Info!

just one issue, you may also need Pyton module pyserial for Davis Weatherlink-USB adaptor.

also if your not on a Mac, then use Putty and/or Winscp.

[halol]

It was very useful to know that somebody else managed to get weewx working, thanks!

The installation steps for me have been much simpler (on a TS 219P II with firmware 4.1.1):
- first I used the SQLite database. There are plenty of good admin tools to open SQLite database if you want to look at the data, and it's easier to backup or move to another PC if you want to play with it.
- second, I used pip to install the python modules. I didn't run into any problem, but it could be because I am using my own version of python, compiled with SSL support.

But the main problem I faced was accessing the weather station: there was no /dev/ttyUSB0 on my system, and connecting to /dev/usb/ttyUSB0 didn't work. Turns out that I first needed to load the following kernel modules:
- usbserial
- cp210x
Using insmod /mnt/ext/usr/local/modules/cp210x.ko. Then it worked like a charm.

Oh, something else: when installing weewx, I also thought that it would be better to change the default installation directory (/home/weewx).

PS: by the way, I am using a Vantage Pro 2 weather station.

[aardvark]

Great instructions but right at the last hurdle -
--------------------------------------------------------------------------------------
[/share/homes/admin/weewx-2.6.4] # python setup.py install
Traceback (most recent call last):
File "setup.py", line 56, in <module>
import configobj
File "/share/MD0_DATA/.qpkg/Python/lib/python2.7/site-packages/configobj.py", line 22, in <module>
import six
ImportError: No module named six
-----------------------------------------------------------------------------------------
Any ideas how to fix gratefully received

[halol]

The message is fairly explicit: python doesn't find the module 'six'

You will have to install six as you did for the other required modules (either using pip or setup.py).

[aardvark]

Thanks halol. Everything OK now. Senior moments are terrible things......

[halol]

You're welcome, glad to hear that you got it working.

My turn to have small problem: weewx is running fine, it is filling the database exactly the way I configured it. So far so good. But I also try to upload my data to Wunderground. This seems to be working OK as well, but on a regular basis I get the message: "PWS is not emitting" and I have to restart weewx.

I would like to troubleshoot this rather than just restart weewx every time, but on QNAP I have no clue where the logs are going! /var/log/messages is always empty, there is no syslog (where we can see weewx traces on a Ubuntu system for example).

Does anybody knows where I can find the system logs, or how I could tweak weewx to send the traces to a custom logs file?

Thanks.

[visuel]

You have to start the syslogd first to get messages.

You can do the following

/sbin/syslogd -m0

[visuel]

Trying to get weewx running on my Qnap 419P with the Bootstrap templates.

Everthing seems to work fine, except for the images of the gauges.

Code: Select all

raise ImportError("The _imagingft C module is not installed")

Code: Select all

  
Oct  5 19:41:15 VISUELNAS user.crit weewx[29587]:         ****  The _imagingft C module is not installed
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****  Traceback (most recent call last):
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****    File "/mnt/ext/home/weewx/bin/weewx/reportengine.py", line 133, in run
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****      obj.start()
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****    File "/mnt/ext/home/weewx/bin/weewx/reportengine.py", line 298, in start
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****      self.run()
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****    File "/mnt/ext/home/weewx/bin/user/gaugeengine.py", line 126, in run
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****      self.gen_gauges()
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****    File "/mnt/ext/home/weewx/bin/user/gaugeengine.py", line 157, in gen_gauges
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****      self.gen_gauge(gauge, plot_options, img_file)
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****    File "/mnt/ext/home/weewx/bin/user/gaugeengine.py", line 254, in gen_gauge
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****      gauge.add_text(label_text, text_font_size=label_font_size, text_font=font_path, text_color=text_color)
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****    File "/mnt/ext/home/weewx/bin/user/gauges.py", line 151, in add_text
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****      self.text_font = ImageFont.truetype(text_font, text_font_size)
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****    File "/share/MD0_DATA/.qpkg/Python/lib/python2.7/site-packages/PIL/ImageFont.py", line 218, in truetype
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****      return FreeTypeFont(filename, size, index, encoding)
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****    File "/share/MD0_DATA/.qpkg/Python/lib/python2.7/site-packages/PIL/ImageFont.py", line 134, in __init__
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****      self.font = core.getfont(file, size, index, encoding)
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****    File "/share/MD0_DATA/.qpkg/Python/lib/python2.7/site-packages/PIL/ImageFont.py", line 34, in __getattr__
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****      raise ImportError("The _imagingft C module is not installed")
Oct  5 19:41:15 VISUELNAS user.info weewx[29587]:         ****  ImportError: The _imagingft C module is not installed
Oct  5 19:41:15 VISUELNAS user.crit weewx[29587]:         ****  Generator terminated...

I think this is because PIL misses some libraries.

Does somebody know how to get these for the QNAP....

[halol]

You have to start the syslogd first to get messages.
You can do the following
/sbin/syslogd -m0

Very good thing to know! Thanks a lot for the tip.

[halol]

I think this is because PIL misses some libraries.

Indeed, it looks like you are missing truetype support in PIL (according to Google at least).

Strangely I didn't run into this error... How did you install PIL? Using pip or setup.py?

Some suggestions (all untested unfortunately):
- re-install PIL after enabling truetype support.
- if you used the setup.py method, try pip => that's what I did and I don't have this error.
- change the skin configuration to use .pil fonts instead of .ttf (see skins/Images/skin.conf, there are some explanations).
- you can apparently replace PIL with Pillow (supposed to be a drop-in replacement). Might be worth a try.

PS: although I didn't get the same error than you do, I am still a long way from having a working Bootstrap skin and I had several other issues so far:
- first, there is no truetype fond on the QNAP. I had to copy some from my Linux box, and make some changes both in the skin.ini and in some .py files.
- then I am still getting an error somewhere in the code (didn't have time to look at it so far).
- and only some of the images are displayed, I am missing some.

[halol]

I finally got the Bootstrap skin working. Here is what worked for me:
- get version 2 of the skin. I initially got version 1 and it didn't like the fact that there are empty values in my archive database (for wind direction).
- I did the manual install of the skin, I haven't tested installation using setup.py
- I copied truetype fonts to /opt/share/fonts/truetype/freefont/
- I modified the path to the font in all the .conf and .py files provided with the skin (skin.conf in Bootstrap and Images, and gaugeengine.py)

I still have a problem with the history tab which is displaying incorrect values, but everything else is working fine.

[Toxic17]

And how easy is it to upgrade Weewx with setup.py to 2.7.0?

Sent from my A0001 using Tapatalk

[halol]

This shouldn't be too difficult:
http://www.weewx.com/docs/upgrading.htm

I never did it so far though... But anyway, once weewx is installed with all the appropriate dependencies all the difficulty is gone, all that must be done for an upgrade is to replace the files in the 'bin' folder and merge potential changes in the configuration file (if any). It could probably easily be done manually I guess, unless there are also updates to the database schema.

[halol]

I just upgraded to 2.7.0 following the instructions here:
http://www.weewx.com/docs/upgrading.htm
and I can confirm that it worked without any problem.