Guide to pmtr

Pmtr is one of my many C projects. Back to the pmtr Github page.

About

I wrote pmtr as a simple supervisory daemon for my own needs. Pmtr runs under systemd as well as other init mechanisms (sysvinit, upstart, etc), or it can run as process 1 inside a container. When installing on many flavors of Linux, it can auto-detect the underlying init system and set itself up to start at boot appropriately. I wrote pmtr with a few goals in mind:

to have one configuration file listing all processes to run
to exist under (not to replace) the underlying init system
to insulate myself from the underlying init system
to be a tiny C program
to consume few resources
to have few features
no dependencies

I use pmtr to abstract away the underlying init system, listing my processes inside the pmtr.conf configuration file. Only pmtr itself is managed as a system service. Pmtr is for Linux only, and is MIT licensed.

Example /etc/pmtr.conf

job {
  name tunnel
  cmd /usr/bin/ssh -i key -TNL 5901:127.0.0.1:5901 192.168.0.1
}

job {
  name capture
  dir /data
  cmd /usr/sbin/tcpdump -i eth0 -s 0 -G 10 -w %Y%m%d%H%M%S.pcap
}

There are options to customize each job (process) including where to send its stdout and stderr streams (default is to syslog them), its ulimits, user id, environment variables, etc, as documented below. A job that exits gets restarted and if it exits too quickly (less than 10 seconds after it started), pmtr delays its restart slightly to avoid rapid process cycling. Processes can often simply die on error conditions (such as failure to connect to a network server) and rely on pmtr to restart them, rather than embedding retry logic in each process. If the operator edits pmtr.conf and saves the file, the changes take effect immediately. There is no need to tell pmtr to reload its configuration file.

pmtr

Platforms

Pmtr runs on these flavors of Linux and probably more:

Ubuntu 12-16
CentOS/RHEL 5-7
Amazon Linux (AMI)
Debian
Arch
Raspberry Pi (Raspbian)
Beaglebone Black (Debian)
Yocto layer

Build & Install

Download

git clone https://github.com/troydhanson/pmtr.git

Build

cd pmtr
./autogen.sh
./configure --bindir=/usr/bin --sysconfdir=/etc
make

Install

sudo make install
sudo touch /etc/pmtr.conf
cd initscripts
sudo ./setup-initscript --auto

pmtr.conf

The pmtr configuration is, by default, /etc/pmtr.conf. Note that /etc comes from the --sysconfidir option to configure. If you omit that option to configure, it may default to /usr/local/etc instead.

General job requirements

Processes that run under pmtr should stay in the foreground, exit on SIGTERM or SIGKILL, and clean up after their own sub-processes when exiting.

The minimal requirement is that each job specification has a name and a cmd. The order of options inside does not matter. Indentation is optional. Blank lines are ok. Comment lines start with #.

These options may appear in a job definition.

Table 1. Job options
option	argument
`name`	descriptive job name used for logging - must be unique
`cmd`	executable (fully-qualified) and any arguments
`dir`	working directory (fully qualified) to run the process in
`out`	send stdout to this file
`err`	send stderr to this file
`in`	hook stdin to this file
`env`	environment variable to set (repeatable)
`user`	unix username under whose id to run the process
`ulimit`	process ulimits
`bounce every`	a time interval to restart the process
`depends`	files to watch, any changes induce the job to restart
`disable`	disable the job
`wait`	(special) wait for the job to finish before going on
`once`	(special) do not restart the job

More details on each option follows.

cmd

Specifies the absolute path to the executable (there is no $PATH searching!)
cmd may have arguments (like cmd /usr/bin/python rest.py)
You can use double-quotes in arguments (see example below)
cmd does no shell expansion: no wildcards, backticks, variables, etc.
If you need shell features in your command, wrap it with a shell script.

Tip: you can invoke shell features right inside a command in this way:

job {
  name api-server
  dir /home/py
  cmd /bin/bash -c "ifconfig eth1 up; sleep 5; exec python server.py"
}

env

Use env to push an environment variable into a job, e.g. env DEBUG=1.
Use env repeatedly to set multiple environment variables.

disable

disable on a line by itself is like commenting the job out

out, err, in

Use out, err, and in to attach stdout, stderr or stdin to a file.
stdin defaults to /dev/null.
attach stdout or stderr to /dev/null to discard it.
jobs' stdout and stderr go to syslog by default
the keyword syslog is reserved (out syslog, err syslog are default)

user

user is the unix username under whose uid/gid the process runs
it defaults to root (if pmtr is running as root, as it normally is)

depends

Use a depends block to specify files, one per line, that the job depends on.
pmtr watches the dependencies, and restarts the job if they change
```
depends {
  /etc/ssh/sshd_config
  /root/.ssh/id_rsa.pub
}
```

bounce every

The bounce every option restarts the job every so often.
It’s for jobs that don’t behave well as long-running processes.
It takes a number and unit, like 5m to restart a job every five minutes.
The units are smhd- (s)econds, (m)inutes, (h)ours or (d)ays.
The exact timing of the restart is approximate.

ulimit

Use ulimit to modify the kernel-enforced resource limits for the job.
It takes a flag denoting which limit to set, and a value, e.g., ulimit -n 30.
The values are all numeric or the keyword infinity.
To see the current kernel limits on a process, run cat /proc/<pid>/limits.
pmtr sets both the "hard" and "soft" ulimit to the same value.
If there is an error setting the ulimits, it’s logged to syslog.

For info on what ulimits are available, you can run these commands in bash:

ulimit -a to list flags, and the values in your shell
help ulimit for more information about the ulimits
man prlimit for more technical information and the units

wait/once

Pmtr has rudimentary support for "one-time" setup jobs using wait and once.
wait pauses startup of subsequent jobs until this one finishes

once tells pmtr not to restart this job upon exit

job {
  name initial-setup
  cmd /bin/mkdir -p /dev/shm/work
  wait
  once
}

Operational notes

Edits take effect immediately

When you edit and save pmtr.conf, pmtr applies the changes immediately. (It watches the file using inotify).

A newly-added job gets started.
A deleted job is terminated.
A changed job is restarted with its new configuration.

Testing the configuration

You can run pmtr -t to check the config file syntax and report any errors. This checks the file without running any jobs.

Status and control

To start, stop or get status of pmtr, use the host system management commands.

Management on a systemd-based host

systemctl start pmtr
systemctl stop pmtr
systemctl status pmtr

Management on most other systems

service pmtr status
service pmtr start
service pmtr stop

When pmtr is started, it starts all the jobs in pmtr.conf, likewise, pmtr terminates them when it is stopped.

Logs

Pmtr logs typically go to /var/log/syslog or /var/log/messages, according to the host syslog configuration. It is recommended to check the logs after any pmtr.conf changes. Any errors in parsing pmtr.conf are logged. Job starts, exits and restarts are logged. On systemd-based hosts, you can use journalctl -u pmtr to see pmtr logs.

Job termination and restart

Jobs that exit on their own

If a job terminates by itself, when pmtr did not signal it to exit, (and the job does not have the once option), pmtr restarts it. However, if the job exited within 10 seconds of when it started, pmtr waits 10 seconds to restart it.

Restart management

This has two benefits: a job that is waiting for some resource- say, to connect to a remote server that is still booting up- can just exit instead of retrying. It can expect pmtr to restart it 10 seconds later when it can try again. The 10-second wait also prevents fork bombs (rapid process cycling).

How pmtr kills a job

Pmtr terminates a job when it is deleted, disabled, or altered in pmtr.conf, or is being bounced due to the bounce every option; or because pmtr itself is being shut down. To terminate a job, pmtr sends SIGTERM to it, then SIGKILL shortly afterward, if it’s still running.

Command line options

The host init system normally runs pmtr at system boot. However you can run pmtr manually. These are the command line options that pmtr accepts.

Table 2. pmtr options
`-h`	show help
`-F`	stay in foreground
`-I`	make jobs' stdout/err inherit pmtr’s instead of syslog
`-c <file>`	specify configuration file
`-t`	test syntax (parse config file and exit)
`-v`	verbose logging (repeatable), -vv shows parsing
`-p <file>`	make pidfile

UDP control

Note	This feature is disabled by default.

These options may appear in pmtr.conf at the global scope.

report to udp://127.0.0.1:9999
listen on udp://0.0.0.0:10000

The report to option designates a remote address and port to which pmtr should send a a UDP packet every ten seconds. The packet payload lists the job names, enabled or disabled status, and elapsed runtimes in simple text. If the report to address falls in the multicast UDP range (e.g. 239.0.0.1, etc), the specification may include a trailing interface, e.g., report to udp://239.0.0.1:9999@eth2 to designate the interface from which the multicast UDP datagrams should egress.

The listen on option allows jobs to be remotely enabled or disabled. It specifies a UDP address and port that pmtr should listen on for datagrams of form enable abc or disable abc, where abc is a job name. The address 0.0.0.0 can be used as a shortcut to denote "any address" on this system. The effect is temporary; the settings in pmtr.conf resume precedence when it’s edited or pmtr is restarted.

These options are considered experimental and may be replaced or removed.