Electronics · raspberry pi

Raspberry PI – Run on boot, and run forever! Systemd/Systemctl

The raspberry pi is a cheap linux box you can have plugged in all the time, and they’ll just crank away until the flash wears out… which is likely to be never.

case2-500x334

Start stuff on boot

One of the few things that can get in your way:  power dropping out and causing these little guys to reboot, thereby shutting down all of your projects.

To ensure that your stuff comes up every time the power goes out – the Linux gurus have created something called Systemd.  Systemd is a system daemon that manages long running processes on modern linux.  It’s a relatively new manager so it has lots of nice features for inspecting processes, keeping an eye on them, and being able to reboot your whole box if anything goes sideways.

Adafruit Systemd Tutorial

The best place to get a quick start is over at Adafruit.  Check that out then come back and I’ll round out what they taught you:

https://learn.adafruit.com/running-programs-automatically-on-your-tiny-computer/systemd-writing-and-enabling-a-service

Welcome back.  By now you should have a systemd service description file.  Something like this:

[Unit]
Description=Mouse Logging Service
 
[Service]
ExecStart=/root/mouse.py
StandardOutput=null
 
[Install]
WantedBy=multi-user.target
Alias=mouselogger.service

After Adafruit… When!

For most things we do here, you’ll want to add a little more to that.  Specifically a bit about when the target should be started.  To control that you need to add a line called after”.  Systemd will make sure that whatever is in the line “after” will be running before your stuff is called – this is incredibly handy for internet controlled programs, you’ll want internet to be running first!

[Unit]
Description=Mouse Logging Service
After=network-online.target
 
[Service]
ExecStart=/root/mouse.py
StandardOutput=null
 
[Install]
WantedBy=multi-user.target
Alias=mouselogger.service

Logging!

The next change you might want to make is to ensure you have someplace to see the log output from your program.  The line:

 StandardOutput=null

Controls your logging – and in this case deletes it.  Instead, send the log data to syslog found in /var/log/syslog by deleting that line or being explicit about sending the data there:

[Unit]
Description=Mouse Logging Service
After=network-online.target
 
[Service]
ExecStart=/root/mouse.py
StandardOutput=syslog
 
[Install]
WantedBy=multi-user.target
Alias=mouselogger.service

Restart your stuff if it crashes!

One other thing you’ll want to do to make sure your stuff is running forever: watchdog it!

There are a huge number of things that might cause your program to crash.  Internet connectivity, services responses your code didn’t expect, overrunning memory… really a huge number of things.

Watchdogs can work in a couple of different ways in different environments.  For systemd, it expects your process to call a special notification service.  If your process doesn’t call that often enough…. boom, restarted!

Enable Watchdog

To enable that service first update your configuration file:

[Unit]
Description=Mouse Logging Service
After=network-online.target
 
[Service]
ExecStart=/root/mouse.py
StandardOutput=syslog
WatchdogSec=30s
Restart=on-failure
StartLimitInterval=5min
StartLimitBurst=4
StartLimitAction=reboot-force
NotifyAccess=all

[Install]
WantedBy=multi-user.target
Alias=mouselogger.service

There are lots of new directives there:

WatchdogSec=30s

That specifies how often the process should call back to the system service.  If it does phone home within 30 seconds – then its restarted.  Which is what is specified by:

Retart=on-failure

The other values are specifying the conditions that dictate a full system reboot.  Meaning the raspberry-pi completely shutsdown and restarts.  In this case it will be 4 reboots in a 5 minute window.  As directed by these lines:

StartLimitInterval=5min
StartLimitBurst=4
StartLimitAction=reboot-force

And there you have it.  Your raspberry pi will now make sure all of your stuff is running… forever!  But wait… your process hasn’t been notifying.

Notifying Systemd your process is alive!!!

There is a special program that needs to be called on a regular interval to ensure that watchdog doesn’t reset.  Its called systemd-notify.  Your program needs to call:

/bin/systemd-notify

In Node a way to do it is like this:

  var args = [
 '--pid=' + process.pid,
 'WATCHDOG=1'
 ]
 child_process.execFile('/bin/systemd-notify', args);

In Python – do this:

import subprocess

# Simple command
subprocess.call(['/bin/systemd-notify','--pid=' + str(os.getpid()),'WATCHDOG=1], shell=True)

Then do that at intervals that are smaller than the value you entered in:

WatchdogSec

Now you should be good to go forever!

If you get into a pinch! A.K.A The watchdog from hell!

Using watchdog scripts its possible to make a mistake that causes the watchdog script to put your raspberry pi into a reboot loop!

It works like this: Watchdog check fails… reboot… service starts up… watchdog check fails… reboot… service starts up… etc. etc.

If you used PiBakery to setup your disk – you are in luck!  Check out this article:

PiBakery – Tool with Side benefits for setting up your Raspberry Pi

If you didn’t use PiBakery, the easiest way to fix this is probably to plug your SD card into a linux box, and edit the service file directly.  For pi users that is as simple as getting a USB SD card reader, then using a new Raspbian Pi to mount and edit the disk that is in Watchdog hell.

 

Advertisements

4 thoughts on “Raspberry PI – Run on boot, and run forever! Systemd/Systemctl

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s