Pmtr is one of my many C projects. Back to the pmtr Github page.

About

I wrote pmtr as a simple supervisory daemon for my own needs. Pmtr runs under systemd as well as other init mechanisms like sysvinit, etc. It can also run as process 1 inside a container. When installing on many flavors of Linux, it can detect the host init system and set itself up to start at boot appropriately. I wrote pmtr with a few goals in mind:

  • to have one configuration file listing all processes to run

  • to exist under (not to replace) the host init system

  • to insulate myself from the host init system

  • to consume few resources

  • to have few features

  • no dependencies

It is written in C, supports Linux only, and is MIT licensed.

Example /etc/pmtr.conf
job {
  name tunnel
  cmd /usr/bin/ssh -i key -TNL 5901:127.0.0.1:5901 192.168.0.1
}

job {
  name bitcoin-daemon
  dir /home/bitcoin
  cmd /usr/bin/bitcoind -conf=/etc/bitcoin/bitcoin.conf
  user bitcoin
  nice 5
}

job {
  name capture
  dir /data
  cmd /usr/sbin/tcpdump -i eth0 -s 0 -G 10 -w %Y%m%d%H%M%S.pcap
  env LD_LIBRARY_PATH=/usr/local/lib
}

When pmtr is started, it starts all the jobs in pmtr.conf, likewise, pmtr terminates them when it is stopped.

In practice I let the host init system- often systemd- manage all the standard OS services. I use pmtr to run the local software- the parts that make up an appliance, often comprising a dozen or two dozen jobs that function together.

There are many options to use inside the job, listed later below.

Processes that run under pmtr should stay in the foreground, exit on SIGTERM or SIGKILL, and clean up after their own sub-processes when exiting.

If a job exits, pmtr restarts it. If it exits too quickly- less than 10 seconds after it started- pmtr delays its restart 10 seconds to avoid rapid cycling.

If the operator edits pmtr.conf and saves the file, the changes take effect immediately. There is no need to tell pmtr to reload its configuration file.

Platforms

Pmtr runs on these flavors of Linux, and probably others:

  • Ubuntu 12-17

  • CentOS/RHEL 5-7

  • Amazon Linux AMI

  • Debian

  • Arch

  • Raspberry Pi Raspbian

  • Beaglebone Black Debian

There is a Yocto layer to bake pmtr into an embedded image.

Build & Install

Option 1: RPM

A RHEL/CentOS 7 x86_64 RPM package for pmtr can be found here.

Option 2: Build from source

To build pmtr from source follow these steps.

Install the prerequisite tools:

Ubuntu
sudo apt install git build-essential autoconf automake
RedHat/CentOS
sudo yum install git gcc autoconf automake
Clone pmtr
git clone https://github.com/troydhanson/pmtr.git
Build
cd pmtr
./autogen.sh
./configure --bindir=/usr/bin --sysconfdir=/etc
make
Install
sudo make install
sudo touch /etc/pmtr.conf
Set up initscript

Do this to start pmtr automatically at boot.

cd initscripts
sudo ./setup-initscript --auto

Using setup-initscript --auto detects common init managers like systemd and sets up pmtr to start at boot. Run ./setup-initscript --help to see further options. Here’s an example of running it on Ubuntu based on systemd:

sudo ./setup-initscript --auto
initsystem          : systemd
pmtr                : /usr/bin/pmtr
services            : /etc/systemd/system
initscript          : /etc/systemd/system/pmtr.service
enabling            : systemctl enable pmtr
starting service    : systemctl start pmtr

You can look at the syslog to confirm it is now running.

grep pmtr /var/log/syslog   # ubuntu
grep pmtr /var/log/messages # redhat/centos

You should see something like:

Nov 13 17:13:10 ubuntu systemd[1]: Started pmtr process manager.
Nov 13 17:13:10 ubuntu pmtr[10888]: pmtr: starting

It is always a good idea to look in the syslog after making changes to the configuration file. This is where pmtr reports on starting jobs, or on any errors in parsing the configuration file. Any output generated by the jobs also appears in the syslog, by default.

Config file

The configuration file /etc/pmtr.conf drives pmtr.

The config file can be changed using a command line option (-c) when running pmtr from the command line.

As shown previously it consists of zero or more job definitions. A job is just a process definition. It looks like:

job {
  name bitcoin-daemon
  dir /home/bitcoin
  cmd /usr/bin/bitcoind -conf /etc/bitcoin/bitcoin.conf
  user bitcoin
}

A job must have a name (used for logging only) and a cmd. Everything else in the curly braces optional.

  • Order does not matter.

  • Indentation is optional.

  • Blank lines are ok.

  • Comments start with #.

Options

The full list of options that may appear inside a job are listed here.

Table 1. Job options
option argument

name

descriptive job name used for logging - must be unique

cmd

executable (fully-qualified) and any arguments

dir

working directory (fully qualified) to run the process in

out

send stdout to this file

err

send stderr to this file

in

take stdin from this file

env

environment variable to set (repeatable)

user

unix username under whose id to run the process

nice

unix priority between -19 (highest) and 20 (lowest)

ulimit

process ulimits

bounce every

a time interval to restart the process

depends

files to watch, any changes induce the job to restart

disable

disable the job

wait

(special) wait for the job to finish before going on

once

(special) do not restart the job

More details on each option follows.

cmd

  • Specifies the absolute path to the executable (there is no $PATH searching!)

  • It may have arguments after the executable (e.g. cmd /usr/bin/python rest.py)

  • Use double-quotes to make one argument from the quoted string.

  • No shell expansion: no wildcards, backticks, variables, etc.

If you need shell features in your command, wrap it with a shell script. You can also use a quoted string as a shell script like this:

job {
  name api
  cmd /bin/bash -c "ifconfig eth1 up; sleep 5; exec /api/rest.py"
}

env

  • Use to push an environment variable into a job, e.g. env DEBUG=1.

  • Use repeatedly on separate lines to set multiple environment variables.

disable

  • Use disable on a line by itself to make the job disabled.

  • This is sometimes quicker than commenting out the whole job.

out, err, in

  • Use out and err to send stdout or stderr to a file.

  • stdout and stderr go to syslog by default.

  • stdin defaults to /dev/null; use in to override

nice

  • This changes the process priority

  • Takes a number in the range -19 (highest priority) to 20 (lowest)

user

  • Specifies the unix username to run the process as.

  • That user’s uid/gid becomes those of the process.

  • Defaults to root (when pmtr is running as root)

depends

  • Specifies a block of one or more files that the job depends on.

  • pmtr watches the dependencies for changes to their content.

  • Pmtr restarts the job if a change is detected.

    job {
      name bitcoin-daemon
      dir /home/bitcoin
      cmd /usr/bin/bitcoind -conf=/etc/bitcoin/bitcoin.conf
      user bitcoin
      depends {
        /etc/bitcoin/bitcoin.conf
      }
    }

bounce every

  • Use bounce every to restart a job on a periodic interval.

  • It takes a number and unit [smhd] e.g. bounce every 1d.

  • Units [smhd] are seconds, minutes, hours or days.

  • The exact timing of the restart is approximate.

ulimit

  • Use to modify the system resource limits for the job.

  • Takes a flag and value, e.g., ulimit -n 30.

  • Values are numeric or the keyword infinity.

To see the current limits on a process by its PID:

cat /proc/<pid>/limits

Pmtr sets both the "hard" and "soft" limit to the same value. Any error in setting the limit is logged to syslog.

See man prlimit for technical descriptions of each limit. In the bash shell, ulimit -a and help ulimit display the limits and a list of flags and descriptions respectively.

Table 2. ulimit flags

-c

the maximum size of core files created

-d

the maximum size of a process’s data segment

-e

the maximum scheduling priority (taken as 20-limit)

-f

the maximum size of files the process may create

-i

the maximum number of pending signals

-l

the maximum bytes a process may lock into memory

-m

the maximum resident set size

-n

the maximum number of open file descriptors

-q

the maximum number of bytes in POSIX message queues

-r

the maximum real-time scheduling priority

-s

the maximum stack size

-t

the maximum amount of cpu time in seconds

-u

the maximum number processes or threads

-v

the maximum size of process virtual memory

The units are discussed in the prlimit(2) man page.

wait/once

  • Used for setup jobs that need not be restarted.

  • wait pauses the startup of subsequent jobs.

  • once tells pmtr not to restart this job.

    job {
      name initial-setup
      cmd /bin/mkdir /dev/shm/go
      wait
      once
    }

Operator notes

Pmtr detects writes to pmtr.conf and applies the changes immediately.

  • A newly-added job gets started.

  • A deleted job is terminated.

  • A changed job is restarted with its new configuration.

You can run pmtr -t to check the config file syntax and report any errors.

Status and control

To start, stop or get status of pmtr, use the host system management commands.

On a systemd-based host:
systemctl start pmtr
systemctl stop pmtr
systemctl status pmtr
On most other hosts:
service pmtr status
service pmtr start
service pmtr stop

Logs

Logging from pmtr goes to syslog. Typically these logs go to /var/log/syslog or /var/log/messages. Check the syslog after any change to pmtr.conf. Errors in parsing pmtr.conf are logged there, as well as job start or exit events. On systemd-based hosts, you can also use journalctl -u pmtr to see pmtr logs.

Job termination and restart

Jobs that exit on their own

If a job terminates by itself, when pmtr did not signal it to exit, (and the job does not have the once option), pmtr restarts it. However, if the job exited within 10 seconds of when it started, pmtr waits 10 seconds to restart it.

Restart management

This has two benefits: a job that is waiting for some resource- say, to connect to a remote server that is still booting up- can just exit instead of retrying. It can expect pmtr to restart it 10 seconds later when it can try again. The 10-second wait also prevents rapid process cycling.

How pmtr kills a job

Pmtr terminates a job when it is deleted, disabled, or altered in pmtr.conf, or is being bounced due to the bounce every option; or because pmtr itself is being shut down. To terminate a job, pmtr sends SIGTERM to it, then SIGKILL shortly afterward, if it’s still running.

Command line options

The host init system normally runs pmtr at system boot. You can instead run pmtr manually. Or, it can be the init process inside a container. In these use cases, the following command line options may be useful.

Table 3. pmtr command-line options

-h

show help

-F

stay in foreground

-I

echo syslog to stderr

-c <file>

specify configuration file

-t

test syntax (parse config file and exit)

-v

verbose logging (repeatable), -vv shows parsing

-p <file>

make pidfile

In a container

To use pmtr as the init entrypoint (process 1) in a container, invoke it with -F to stay in the foreground. Also -I is convenient to make the log messages from pmtr and its jobs echo to stderr for greater visibility. If the container has the pmtr.conf in a custom location, use the -c flag.

# container entrypoint
/usr/bin/pmtr -IFc /path/to/pmtr.conf

UDP control

Note
This feature is disabled by default.

These options may appear in pmtr.conf at the global scope.

report to udp://127.0.0.1:9999
listen on udp://0.0.0.0:10000

The report to option designates a remote address and port to which pmtr should send a a UDP packet every ten seconds. The packet payload lists the job names, enabled or disabled status, and elapsed runtimes in simple text. If the report to address falls in the multicast UDP range (e.g. 239.0.0.1, etc), the specification may include a trailing interface, e.g., report to udp://239.0.0.1:9999@eth2 to designate the interface from which the multicast UDP datagrams should egress.

The listen on option allows jobs to be remotely enabled or disabled. It specifies a UDP address and port that pmtr should listen on for datagrams of form enable abc or disable abc, where abc is a job name. The address 0.0.0.0 can be used as a shortcut to denote "any address" on this system. The effect is temporary; the settings in pmtr.conf resume precedence when it’s edited or pmtr is restarted.

These options are considered experimental and may be replaced or removed.