Guide to pmtr

Pmtr is one of my many C projects. Back to the pmtr Github page.

About

I wrote pmtr as a simple supervisory daemon for my own needs. Pmtr runs under systemd as well as other init mechanisms like sysvinit, etc. It can also run as process 1 inside a container. When installing on many flavors of Linux, it can detect the host init system and set itself up to start at boot appropriately. I wrote pmtr with a few goals in mind:

to have one configuration file listing all processes to run
to exist under (not to replace) the host init system
to run under various host init systems
to consume few resources
to have few features
no dependencies

It is written in C, supports Linux only, and is MIT licensed.

Example /etc/pmtr.conf

job {
  name tunnel
  cmd /usr/bin/ssh -i key -TNL 5901:127.0.0.1:5901 192.168.0.1
}

job {
  name bitcoin-daemon
  dir /home/bitcoin
  cmd /usr/local/bin/bitcoind
  user bitcoin
  env HOME=/home/bitcoin
  cpu 0-8
}

job {
  name capture
  dir /data
  cmd /usr/sbin/tcpdump -i eth0 -s 0 -G 10 -w %Y%m%d%H%M%S.pcap
}

When pmtr is started, it starts all the jobs in pmtr.conf, likewise, pmtr terminates them when it is stopped.

In practice I let the host init system- often systemd- manage all the standard OS services. I use pmtr to run the local software- the parts that make up an appliance, often comprising a dozen or two dozen jobs that function together.

There are many options to use inside the job, listed later below.

Processes that run under pmtr should stay in the foreground, exit on SIGTERM or SIGKILL, and clean up after their own sub-processes when exiting.

If a job exits, pmtr restarts it. If it exits too quickly- less than 10 seconds after it started- pmtr delays its restart 10 seconds to avoid rapid cycling.

If the operator edits pmtr.conf and saves the file, the changes take effect immediately. There is no need to tell pmtr to reload its configuration file.

Platforms

Pmtr runs on these flavors of Linux, and probably others:

Arch
Alpine
Debian
Ubuntu 12-18
CentOS/RHEL 5-8
Amazon Linux AMI 1/2
Raspberry Pi Raspbian
Beaglebone Black Debian
Yocto layer

Build & Install

Option 1: RPM

A RHEL/CentOS 7 x86_64 RPM package for pmtr can be found here.

Option 2: Build from source

To build pmtr from source follow these steps.

Install the prerequisite tools:

Ubuntu

sudo apt install git build-essential autoconf automake

RedHat/CentOS

sudo yum install git gcc autoconf automake make

Clone pmtr

git clone https://github.com/troydhanson/pmtr.git

Build

cd pmtr
./autogen.sh
./configure --bindir=/usr/bin --sysconfdir=/etc
make

Install

sudo make install
sudo touch /etc/pmtr.conf

Set up initscript

Do this to start pmtr automatically at boot.

cd initscripts
sudo ./setup-initscript --auto

Using setup-initscript --auto detects common init managers like systemd and sets up pmtr to start at boot. Run ./setup-initscript --help to see further options. Here’s an example of running it on Ubuntu based on systemd:

sudo ./setup-initscript --auto
initsystem          : systemd
pmtr                : /usr/bin/pmtr
services            : /etc/systemd/system
initscript          : /etc/systemd/system/pmtr.service
enabling            : systemctl enable pmtr
starting service    : systemctl start pmtr

You can look at the syslog to confirm it is now running.

grep pmtr /var/log/syslog   # ubuntu
grep pmtr /var/log/messages # redhat/centos

You should see something like:

Nov 13 17:13:10 ubuntu systemd[1]: Started pmtr process manager.
Nov 13 17:13:10 ubuntu pmtr[10888]: pmtr: starting

It is always a good idea to look in the syslog after making changes to the configuration file. This is where pmtr reports on starting jobs, or on any errors in parsing the configuration file. Any output generated by the jobs also appears in the syslog, by default.

Config file

Overview

The configuration file /etc/pmtr.conf drives pmtr.

The config file can be changed using a command line option (-c) when running pmtr from the command line.

As shown previously it consists of zero or more job definitions. A job is just a process definition. It looks like:

job {
  name bitcoin-daemon
  dir /home/bitcoin
  cmd /usr/local/bin/bitcoind
  user bitcoin
  env HOME=/home/bitcoin
}

A job must have a name (used for logging only) and a cmd. Everything else in the curly braces optional.

Order does not matter.
Indentation is optional.
Blank lines are ok.
Comments start with #.

Options

The full list of options that may appear inside a job are listed here.

Table 1. Job options
option	argument
`name`	descriptive job name used for logging - must be unique
`cmd`	executable (fully-qualified) and any arguments
`dir`	working directory (fully qualified) to run the process in
`out`	send stdout to this file
`err`	send stderr to this file
`in`	take stdin from this file
`env`	environment variable to set (repeatable)
`user`	unix username under whose id to run the process
`nice`	unix priority between -19 (highest) and 20 (lowest)
`cpu`	CPU affinity as hex mask (0xABCD) or number/ranges (0,2-4)
`ulimit`	process ulimits
`bounce every`	a time interval to restart the process
`depends`	files to watch, any changes induce the job to restart
`disable`	disable the job
`wait`	(special) wait for the job to finish before going on
`once`	(special) do not restart the job

More details on each option follows.

cmd

Specifies the absolute path to the executable (there is no $PATH searching!)
It may have arguments after the executable (e.g. cmd /usr/bin/python rest.py)
Use double-quotes to make one argument from the quoted string.
No shell expansion: no wildcards, backticks, variables, etc.

If you need shell features in your command, wrap it with a shell script. You can also use a quoted string as a shell script like this:

job {
  name api
  cmd /bin/bash -c "ifconfig eth1 up; sleep 5; exec /api/rest.py"
}

env

Use to push an environment variable into a job, e.g. env DEBUG=1.
Use repeatedly on separate lines to set multiple environment variables.

disable

Use disable on a line by itself to make the job disabled.
This is sometimes quicker than commenting out the whole job.

out, err, in

Use out and err to send stdout or stderr to a file.
stdout and stderr go to syslog by default.
stdin defaults to /dev/null; use in to override

nice

This changes the process priority
Takes a number in the range -19 (highest priority) to 20 (lowest)

cpu

This sets the CPU affinity- the list of CPU’s the task can run on
Takes a CPU number (e.g. 0) or range (e.g. 2-4) or a mix (e.g. 0,2-4)
Alternatively, can take a 0x-prefixed hex mask (e.g. 0x8f)
Any CPUs in the set that are physically absent are ignored

user

Specifies the unix username to run the process as.
That user’s uid/gid becomes those of the process.
Defaults to root (when pmtr is running as root)

depends

Specifies a block of one or more files that the job depends on.
pmtr watches the dependencies for changes to their content.

Pmtr restarts the job if a change is detected.

job {
  name bitcoin-daemon
  dir /home/bitcoin
  cmd /usr/bin/bitcoind
  user bitcoin
  env HOME=/home/bitcoin
  depends {
    /home/bitcoin/.bitcoin/bitcoin.conf
  }
}

bounce every

Use bounce every to restart a job on a periodic interval.
It takes a number and unit [smhd] e.g. bounce every 1d.
Units [smhd] are seconds, minutes, hours or days.
The exact timing of the restart is approximate.

ulimit

Use to modify the system resource limits for the job.
Takes a flag and value, e.g., ulimit -n 30.
Values are numeric or the keyword infinity.

To see the current limits on a process by its PID:

cat /proc/<pid>/limits

Pmtr sets both the "hard" and "soft" limit to the same value. Any error in setting the limit is logged to syslog.

See man prlimit for technical descriptions of each limit. In the bash shell, ulimit -a and help ulimit display the limits and a list of flags and descriptions respectively.

Table 2. ulimit flags
`-c`	the maximum size of core files created
`-d`	the maximum size of a process’s data segment
`-e`	the maximum scheduling priority (taken as 20-limit)
`-f`	the maximum size of files the process may create
`-i`	the maximum number of pending signals
`-l`	the maximum bytes a process may lock into memory
`-m`	the maximum resident set size
`-n`	the maximum number of open file descriptors
`-q`	the maximum number of bytes in POSIX message queues
`-r`	the maximum real-time scheduling priority
`-s`	the maximum stack size
`-t`	the maximum amount of cpu time in seconds
`-u`	the maximum number processes or threads
`-v`	the maximum size of process virtual memory

The units are discussed in the prlimit(2) man page.

wait/once

Used for setup jobs that need not be restarted.
wait pauses the startup of subsequent jobs.

once tells pmtr not to restart this job.

job {
  name initial-setup
  cmd /bin/mkdir /dev/shm/go
  wait
  once
}

Operator notes

Pmtr detects writes to pmtr.conf and applies the changes immediately.

A newly-added job gets started.
A deleted job is terminated.
A changed job is restarted with its new configuration.

You can run pmtr -t to check the config file syntax and report any errors.

Logs

Logging from pmtr goes to syslog. Typically these logs go to /var/log/syslog or /var/log/messages. Check the syslog after any change to pmtr.conf. Errors in parsing pmtr.conf are logged there, as well as job start or exit events. On systemd-based hosts, you can also use journalctl -u pmtr to see pmtr logs.

Status and control

To start, stop or get status of pmtr, use the host system management commands.

# under systemd
systemctl start pmtr
systemctl stop pmtr
systemctl status pmtr

# other initsystems
service pmtr status
service pmtr start
service pmtr stop

Job exits

Jobs that exit on their own

If a job terminates by itself, when pmtr did not signal it to exit, (and the job does not have the once option), pmtr restarts it. However, if it exited within 10 seconds of when it started, pmtr waits 10 seconds to restart it. The 10-second wait prevents rapid process cycling. Also, a job that’s waiting for something (like a network resource to come up) can be designed to exit instead of retry, relying on pmtr to restart it periodically to try again.

How pmtr kills a job

Pmtr terminates a job when it is deleted, disabled, or altered in pmtr.conf, or is being bounced due to the bounce every option; or because pmtr itself is being shut down. To terminate a job, pmtr sends SIGTERM to it, then SIGKILL shortly afterward, if it’s still running.

Command line options

The host init system normally runs pmtr at system boot. You can instead run pmtr manually. Or, it can be the init process inside a container. In these use cases, the following command line options may be useful.

Table 3. pmtr command-line options
`-h`	show help
`-F`	stay in foreground (enabled by default when PID 1)
`-I`	echo syslog to stderr (enabled by default)
`-c <file>`	specify configuration file
`-t`	test syntax (parse config file and exit)
`-v`	verbose logging (repeatable), -vv shows parsing
`-p <file>`	make pidfile

In a container

When pmtr is the init entrypoint (process 1) in a container, it stays in the foreground by default. An example invocation command is shown.

# container entrypoint
/usr/bin/pmtr -IFc /path/to/pmtr.conf

UDP control

Note	This feature is disabled by default.

These options may appear in pmtr.conf at the global scope.

report to udp://127.0.0.1:9999
listen on udp://0.0.0.0:10000

The report to option designates a remote address and port to which pmtr should send a a UDP packet every ten seconds. The packet payload lists the job names, enabled or disabled status, and elapsed runtimes in simple text. If the report to address falls in the multicast UDP range (e.g. 239.0.0.1, etc), the specification may include a trailing interface, e.g., report to udp://239.0.0.1:9999@eth2 to designate the interface from which the multicast UDP datagrams should egress.

The listen on option allows jobs to be remotely enabled or disabled. It specifies a UDP address and port that pmtr should listen on for datagrams of form enable abc or disable abc, where abc is a job name. The address 0.0.0.0 can be used as a shortcut to denote "any address" on this system. The effect is temporary; the settings in pmtr.conf resume precedence when it’s edited or pmtr is restarted.

These options are considered experimental and may be replaced or removed.