The raspberry pi is a cheap linux box you can have plugged in all the time, and they’ll just crank away until the flash wears out… which is likely to be never.
Start stuff on boot
One of the few things that can get in your way: power dropping out and causing these little guys to reboot, thereby shutting down all of your projects.
To ensure that your stuff comes up every time the power goes out – the Linux gurus have created something called Systemd. Systemd is a system daemon that manages long running processes on modern linux. It’s a relatively new manager so it has lots of nice features for inspecting processes, keeping an eye on them, and being able to reboot your whole box if anything goes sideways.
Adafruit Systemd Tutorial
The best place to get a quick start is over at Adafruit. Check that out then come back and I’ll round out what they taught you:
Welcome back. By now you should have a systemd service description file. Something like this:
[Unit] Description=Mouse Logging Service [Service] ExecStart=/root/mouse.py StandardOutput=null [Install] WantedBy=multi-user.target Alias=mouselogger.service
After Adafruit… When!
For most things we do here, you’ll want to add a little more to that. Specifically a bit about when the target should be started. To control that you need to add a line called “after”. Systemd will make sure that whatever is in the line “after” will be running before your stuff is called – this is incredibly handy for internet controlled programs, you’ll want internet to be running first!
[Unit] Description=Mouse Logging Service After=network-online.target [Service] ExecStart=/root/mouse.py StandardOutput=null [Install] WantedBy=multi-user.target Alias=mouselogger.service
Logging!
The next change you might want to make is to ensure you have someplace to see the log output from your program. The line:
StandardOutput=null
Controls your logging – and in this case deletes it. Instead, send the log data to syslog found in /var/log/syslog by deleting that line or being explicit about sending the data there:
[Unit] Description=Mouse Logging Service After=network-online.target [Service] ExecStart=/root/mouse.py StandardOutput=syslog [Install] WantedBy=multi-user.target Alias=mouselogger.service
Restart your stuff if it crashes!
One other thing you’ll want to do to make sure your stuff is running forever: watchdog it!
There are a huge number of things that might cause your program to crash. Internet connectivity, services responses your code didn’t expect, overrunning memory… really a huge number of things.
Watchdogs can work in a couple of different ways in different environments. For systemd, it expects your process to call a special notification service. If your process doesn’t call that often enough…. boom, restarted!
Enable Watchdog
To enable that service first update your configuration file:
[Unit] Description=Mouse Logging Service After=network-online.target [Service] ExecStart=/root/mouse.py StandardOutput=syslog WatchdogSec=30s Restart=on-failure StartLimitInterval=5min StartLimitBurst=4 StartLimitAction=reboot-force NotifyAccess=all [Install] WantedBy=multi-user.target Alias=mouselogger.service
There are lots of new directives there:
WatchdogSec=30s
That specifies how often the process should call back to the system service. If it does phone home within 30 seconds – then its restarted. Which is what is specified by:
Retart=on-failure
The other values are specifying the conditions that dictate a full system reboot. Meaning the raspberry-pi completely shutsdown and restarts. In this case it will be 4 reboots in a 5 minute window. As directed by these lines:
StartLimitInterval=5min StartLimitBurst=4 StartLimitAction=reboot-force
And there you have it. Your raspberry pi will now make sure all of your stuff is running… forever! But wait… your process hasn’t been notifying.
Notifying Systemd your process is alive!!!
There is a special program that needs to be called on a regular interval to ensure that watchdog doesn’t reset. Its called systemd-notify. Your program needs to call:
/bin/systemd-notify
In Node a way to do it is like this:
var args = [
'--pid=' + process.pid,
'WATCHDOG=1'
]
child_process.execFile('/bin/systemd-notify', args);
In Python – do this:
import subprocess # Simple command subprocess.call(['/bin/systemd-notify','--pid=' + str(os.getpid()),'WATCHDOG=1], shell=True)
Then do that at intervals that are smaller than the value you entered in:
WatchdogSec
Now you should be good to go forever!
If you get into a pinch! A.K.A The watchdog from hell!
Using watchdog scripts its possible to make a mistake that causes the watchdog script to put your raspberry pi into a reboot loop!
It works like this: Watchdog check fails… reboot… service starts up… watchdog check fails… reboot… service starts up… etc. etc.
If you used PiBakery to setup your disk – you are in luck! Check out this article:
PiBakery – Tool with Side benefits for setting up your Raspberry Pi
If you didn’t use PiBakery, the easiest way to fix this is probably to plug your SD card into a linux box, and edit the service file directly. For pi users that is as simple as getting a USB SD card reader, then using a new Raspbian Pi to mount and edit the disk that is in Watchdog hell.
Blogger Brothers 
This is a great article – thanks for sharing! However, I found that it was absolutely necessary to add the “TYPE=simple” option to the systemd unit file [Service] section, otherwise getpid() would return a value that systemd would reject and not reset the watchdog.
LikeLike