In the last few months I've been exploring the use of the Web Thing API using the Python language. After lots of trial and error, I've made some abstractions to simplify interacting with my smart home devices. While I've been blogging about my explorations for quite a while, I've not made a concerted effort to make it easy for anyone else to follow in my foot steps. I'm correcting that today, though I fear I may fail on the "easy" part.
This is a guide to help you accept my invitation to explore with me. However, I need to be clear, this is a journey for programmers familiar with the Python programming language and Linux development practices.
Everyone has different needs and a different programming environment. It is easiest to work with my Python module, pywot if you have a second Linux machine on which to run the pywot scripts. However, if you only have the Things Gateway Raspberry Pi itself, you can still participate, it'll just be a longer task to setup.
The instructions below will walk you through setting up the Things Gateway Raspberry Pi with the requirements to run pywot scripts. If you already have a machine available with Python3.6 you can run pywot scripts there instead of on the RaspberryPi, you can save a lot of hassle by skipping all the way down to Step 5 . If you have the second machine and still want to run the scripts on the Raspberry Pi, I suggest enabling SSH on the Things Gateway and doing command line work with ssh .
One of the unfortunate things about the Raspbian Stretch Linux distribution on which earlier versions of the Things Gateway were distributed, is a rather outdated version of Python 3. Version 3.6 of Python was released in 2016, yet Raspbian Stretch still includes only 3.5. There were a number of important changes to the language between those two versions, especially in the realm of asynchronous programming. Interacting with the Web of Things is all about asynchronous programming.
When I started experimenting with the Web Thing API, I did so using my Ubuntu based Linux workstation that already had Python 3.6 installed as native. I blithely used the newer asynchronous constructs in creating my External Rules Framework. It was a nasty surprise when I moved my code over to the Raspberry Pi running the Things Gateway and found the code wouldn't run.
To get Python 3.6 running on Raspbian Stretch, it must be configured and compiled from source and then installed as an alternate Python. Some folks blanched at that last sentence, I certainly did when I realized what I would have to do. As it turns out, it isn't as onerous as I thought. Searching on the Web for a HOW-TO I found this great page on github . My use of these instructions went flawlessly - there were neither mysterious failures nor unexpected complications requiring research.
Here's exactly what I did to get a version of Python 3.6 running on Raspbian Stretch:
1) Connect a keyboard and monitor to your Raspberry Pi. You'll get a login prompt. The default user is "pi" and the password is "raspberry". You really ought to change the default password to something more secure. See Change Your Default Password for details. (Alternatively, you can enable ssh and login from another machine. From your browser, go to http://gateway.local/settings , select "Developer" and then click the checkbox for "Enable SSH")
2) Once logged in, you want to make sure the RPi is fully updated and install some additional packages. For me, this took about 10 minutes .
pi@gateway:~ $ sudo apt-get update pi@gateway:~ $ sudo apt-get install build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev libffi-dev
3) Now you've got to download Python 3.6 and build it.
The last command " ./configure " took nearly 4 minutes on my RPi:
pi@gateway:~ $ wget https://www.python.org/ftp/python/3.6.6/Python-3.6.6.tar.xz pi@gateway:~ $ tar xf Python-3.6.6.tar.xz pi@gateway:~ $ cd Python-3.6.6 pi@gateway:~/Python-3.6.6 $ ./configureThe next step is to compile it. This takes a lot of time. Mine ran for just under 30 minutes :
pi@gateway:~/Python-3.6.6 $ make
Now you've got to tag this version of Python as an alternative to the default Python. This command took just over
4 minutes
on my RPi:
pi@gateway:~/Python-3.6.6 $ sudo make altinstall
Finally, a bit of clean up:
pi@gateway:~/Python-3.6.6 $ cd .. pi@gateway:~ $ rm Python-3.6.6.tar.xz pi@gateway:~ $ sudo rm -r Python-3.6.6 pi@gateway:~ $
4) Python 3.6 is now installed along side the native 3.5 version. Invoking the command python3 will give you the native 3.5 version, where python3.6 will give you our newly installed 3.6 version. It would be nice to have version 3.6 be the default for our work. You can do that with a virtual environment:
pi@gateway:~ $ python3.6 -m venv py36
This has given you a private version of Python3.6 that you can customize at will without interfering with any other Python applications that may be tied to specific versions. Each time you want to run programs with this Python3.6 virtual environment, you need to activate it:
pi@gateway:~ $ . ~/py36/bin/activate (py36) pi@gateway:~ $It would be wise to edit your .bashrc or other initialization file to make an alias for that command. If you do not, you'll have to remember that somewhat cryptic invocation.
5 ) It's time to install my pywot system. This is the Python source code that implements my experiments with the Things Gateway using the Web Thing API.
I've not uploaded pywot to PyPI. I've chosen not to productize this code because it's what I call stream of consciousness programming. The code is the result of me hacking and experimenting. I'm exploring the problem space looking for interesting and pleasing implementations. Maybe someday it'll be the basis for a product, but until then, no warranty is expressed or implied .
Even though pywot isn't on PyPI, you still get to use the pip command to install it. You're going to get a full git clone of my public pywot repo. Since it's pip , all the dependencies will automatically download and install into the virtual Python3.6 environment. On my Raspberry Pi, this command took more than five minutes to execute.
(py36) pi@gateway:~ $ mkdir dev (py36) pi@gateway:~ $ cd dev (py36) pi@gateway:~/dev $ git clone https://github.com/twobraids/pywot.git (py36) pi@gateway:~/dev $ pip install -e pywot (py36) pi@gateway:~/dev $Did you get a message saying, " You should consider upgrading via the 'pip install --upgrade pip' command. " ? I suggest that you do not do that. It made a mess when I tried it and I'm not too inclined to figure out why. Things will work fine if you skip that no-so-helpful suggestion. <sigh>
6) Before you can run any of the pywot demos or write your own apps, you need to get the Things Gateway to grant you permission to talk to it. Normally, one gets the Authorization Token by accessing the Gateway using a browser. That could be difficult if the RasberryPi that has your Gateway on it is your only computer other than a mobile device. Manually typing a 235 character code from your phone screen would be vexing to say the least.
Instead, run this script adding the url to your instance of the Things Gateway, and your Things Gateway login and password. Take note: if you're running this on the Raspberry Pi that runs the Gateway, the url should have ":8080" appended to it. If you are using some other machine, you do not need that.
(py36) pi@gateway:~/dev $ . ./pywot/demo/auth_key.sh Enter URL: http://gateway.local:8080 Enter email: your.email@somewhere.com Enter password: your_password (py36) pi@gateway:~/dev $ ls sample_auth.ini sample_auth.ini (py36) pi@gateway:~/dev $This command created a file called ~/dev/sample_auth.ini You will use the data authorization key within that file in the configuration files used by the demo apps and the apps you create.
7) Finally, it's time to start playing with the demos. Since everyone has a different set of devices, all of the demos in the ~/dev/pywot/demo and ~/dev/pywot/demo/rule_system will require some modification. To me, the most interesting demos are those in the latter directory.
All of the demo files use configman to control configuration. This gives each script command line switches and the ability to use environment variables and configuration files. All configuration parameters can acquire their values using any of those three methods. Conflicts are resolved with this hierarchy:
- command line switches,
- configuration file,
- environment variables,
- program defaults.
--help will always show you what configuration options are available
--admin.config=<somefilename.ini> will specify a configuration file from which to load values
--admin.dump_config=<somefilename.ini> will create a configuration file for you that you can customize with an editor.
Start with the simplest rule example: ~/dev/pywot/demo/rule_system/example_if_rule.py . Run it to produce a blank configuration file.
(py36) pi@gateway:~/dev $ cd ./demo/rule_system (py36) pi@gateway:~/dev/demo/rule_system $ ./example_if_rule.py --admin.dump_conf=example.ini (py36) pi@gateway:~/dev $ cat example.ini # a URL for fetching all things data #http_things_gateway_host=http://gateway.local # the name of the timezone where the Things are ('US/Pacific, UTC, ...') local_timezone=US/Pacific # format string for logging #logging_format=%(asctime)s %(filename)s:%(lineno)s %(levelname)s %(message)s # log level (DEBUG, INFO, WARNING, ERROR, CRITICAL) #logging_level=DEBUG # the fully qualified name of the RuleSystem class #rule_system_class=pywot.rules.RuleSystem # the number of seconds to allow for fetching data #seconds_for_timeout=10 # the name of the default timezone running on the system ('US/Pacific, UTC, ...') system_timezone=UTC # the api key to access the Things Gateway #things_gateway_auth_key=THINGS GATEWAY AUTH KEY (py36) pi@gateway:~/dev/demo/rule_system $Open example.ini in a text editor of your choice. Using the Authorization key you generated in the file ~/dev/sample_auth.ini , uncomment and set the value of things_gateway_auth_key . Set your local timezone on the local_timezone line. Finally set the value of system_timezone . If you're using the Gateway's Raspberry Pi, you can leave it as UTC Otherwise set it whatever timezone your system is using.
Nota bene : it is unfortunate that the url for connecting to the Things Gateway differs depending on what machine runs the examples. If you're using the same Raspberry Pi that is running the Things Gateway, uncomment the " http_things_gateway_host " line and add ": 8080 " to the end of the line. If, instead, you're running from another machine on the network, you need not make that change.
Edit the source file ~/dev/pywot/demo/rule_system/example_if_rule.py and change the names of the devices to reflect the names of the devices you have in your smart home setup.
If you've not yet expired of old age after all these things you've had to do, it is finally time to actually run the example:
(py36) pi@gateway:~/dev $ ./example_if_rule.py --admin.config=example.ini
The script will echo its configuration to the log and then start listening to the Thing Gateway. As soon as your target light bulb is turned on, the other bulb(s) in the action will also turn on. You can explore the rest of the demos using the same method of creating configuration files.
While not a polished product, my pywot Python module is useful for demonstrating the power of the Web Thing API. Fortunately, the Web Thing API is an open standard that could be implemented by anyone. The Python based Home Assistant (HASS) has plans to integrate it . With a faithful implementation of the standard, pywot could be used as a scripting or rule engine for HASS, or any compliant platform. Cross compatibility and letting everyone join in the fun is our goal.
Some thanks this week goes to Things Gateway developer Dave Hylands for cluing me in to how to get the Gateway Auth Key without having to use a browser.