pmtr: process monitor

pmtr: a Linux process supervisor that can start your executables or scripts at system boot, and restart them as needed

pmtr: process monitor

Pmtr is a supervisor daemon that starts your programs and then babysits them. Most "standard" system services (httpd, sshd, etc) have initscripts, which are the typical way to run daemons on Linux. For programs you wrote yourself, or your local application servers that don’t have initscripts, you might find pmtr easier to use or its features beneficial. Your jobs go in pmtr.conf:

Example /etc/pmtr.conf

  job {
    name rest_server
    cmd /usr/bin/python rest.py
  }

  # set working directory, keep errors
  job {
    name ws-server
    dir /home/maps
    cmd /usr/bin/node maps.js
    err /tmp/maps.err
  }

  # restart if dependencies change
  job {
    name proxy
    cmd /usr/bin/proxy -c /etc/proxy.cfg
    depends {
      /etc/proxy.cfg
    }
  }

  # bounce this job every hour, raise ulimits
  job {
    name thermostat
    dir /home/thermo
    cmd /usr/bin/python therm.py
    bounce every 1h
    ulimit {
      -n infinity
      -l infinity
    }
    user nobody
  }

See the full options below, and the requirements for pmtr jobs. Pmtr applies changes immediately when the config file is edited. The pmtr daemon itself is small, written in C, open-source and for Linux only. It runs on Ubuntu, Debian, RHEL/CentOS, in Docker containers, and others.

Why did I write it?

I wrote pmtr because I wanted one simple configuration file where I could list all my programs, knowing that they’d be run at boot, get restarted if they exit, and they’d react immediately to configuration files changes.

Does it work?

The author has used pmtr for several years in production on hundreds of systems having dozens of jobs each. If you find a bug or have a new feature to suggest- please use the pmtr Github page.

Download & Install

You can use the Github page to:

download a zipfile.
or, git clone https://github.com/troydhanson/pmtr.git

To build and install:

cd pmtr
make
sudo make install

The result of installation is:

the executable: /usr/bin/pmtr
the configuration file: /etc/pmtr.conf
an OS-specific initscript to run pmtr at boot

Running

After installation, pmtr should already be running. Verify using service pmtr status.

service start/stop

You can use the system’s service command to start or stop pmtr itself. If you stop pmtr, it first stops any jobs that it’s running, then exits.

service pmtr stop
service pmtr start

You can also run pmtr on the command-line. By default, it forks into the background. You can tell it to stay in the foreground using -F.

Command line options

While pmtr is normally run as a background service, you can run it manually on the command line, in which case you can specify these flags:

`-c <file>`	config file	specify alternate configuration file
`-t`	test only	parse config file and exit, implies -F
`-F`	Foreground	stay in foreground instead of detaching
`-v`	verbose	extra logging. repeatable. -vv shows parsing

Logging

Look in syslog (typically /var/log/messages or /var/log/syslog) to see pmtr log messages relating to configuration file changes and job activity.

Nov  7 1:38:09 ubuntu pmtr[477]: rescanning job configuration
Nov  7 1:38:10 ubuntu pmtr[477]: started job proxy [478]
Nov  7 1:38:20 ubuntu pmtr[477]: job proxy [478] exited after 10 sec: exit status 0

If your job isn’t running, this is the first place to look. Pmtr is verbose about restarting jobs, or indicating if they are exiting for some reason.

Job requirements

In pmtr the job is just some executable that you want to keep running. It must:

stay in the foreground
exit on SIGTERM or SIGKILL
clean up any sub-processes when it exits

Restarts

Since pmtr is designed to keep things running, if a job exits, pmtr restarts it.

If a job exits quickly (within ten seconds of when it started) pmtr waits ten seconds then restarts it. This avoids rapid restart cycles. The job probably has some sort of configuration problem if it exits quickly.
Exit status 33 can be returned from a process to tell pmtr not to restart it.

Configuration file

The default pmtr configuration file is /etc/pmtr.conf.

Syntax check

To manually check the syntax of a configuration file, use the -t (test) flag:

pmtr -tc /etc/pmtr.conf

This displays any syntax errors. In normal background operation pmtr would log syntax and other errors via syslog; see logging.

Immediate effect

When you edit and save pmtr.conf, the changes get applied immediately.

A newly-added job gets started.
A deleted job is terminated.
A changed job is restarted with its new configuration.

To terminate a job, pmtr sends SIGTERM to it, then SIGKILL shortly afterward.

Syntax summary

The configuration file contains any number of jobs.

Each job is in a curly-brace delimited block.
The order of options inside does not matter.
Only the name and cmd are required.
Indentation is optional.
Blank lines are ok.
Comments start with # and must be on their own line.

A job with many options

  # trending service
  # restarted daily

  job {
    name trends
    dir /home/trends
    cmd /usr/bin/python trends.py
    bounce every 1d
    err trends.err
    ulimit -n infinity
    depends {
      trends.conf
      db.conf
    }
    user pyuser
    env DEBUG=1
  }

option	argument
`name`	descriptive job name used for logging - must be unique
`cmd`	executable (fully-qualified) and any arguments
`dir`	working directory (fully qualified) to run the process in
`out`	send stdout to this file
`err`	send stderr to this file
`in`	hook stdin to this file
`env`	environment variable setting `VAR=VALUE` (repeatable)
`user`	unix username under whose id to run the process
`ulimit`	a flag and value, or a curly-brace delimited list of them
`bounce every`	a time interval like 5m (five minutes) - units are `smhd`
`depends`	a curly-brace delimited list of files, one per line
`disable`	this keyword disables the job

Further information about these options appears below.

cmd

The executable named in the cmd must have its executable-bit set.
The cmd may include trailing arguments e.g. cmd /bin/node server.js.
The cmd may contain double-quoted arguments e.g. cmd /bin/echo "hello world"
The cmd does no shell expansion: no wildcards, backticks, variables, etc.

Tip	If you need shell features in your command, wrap it with a shell script. You can even do this "inline" by passing your commands as a string to bash:

env DIR=/home/ws
cmd /bin/bash -c "/bin/python ws.py -d $DIR/`date +%s`"

env

Use env to push an environment variable into a job, e.g. env TMPDIR=/tmp.
Use env repeatedly to set multiple environment variables.

disable

Use disable to stop a job, without having to comment it out.
This is often helpful in debugging a job since it makes it easy to re-enable the job later (by commenting out the disable line, or removing it).

out, err, in

Use out, err, and in to attach stdout, stderr or stdin to a file.
The stdout, stderr and stdin default to /dev/null.

user

The user option is the unix username that should be the process owner.
The user defaults to root usually, when pmtr is running as a service.
The user defaults to whoever started pmtr, if pmtr was run manually.

depends

Use a depends block to specify files, one per line, that the job depends on.
If the content of any of the dependencies changes, pmtr restarts the job.

bounce every

The bounce every option restarts the job every so often.
It’s intended for jobs that don’t behave well as long-running processes.
It takes a number and unit, like 5m to restart a job every five minutes.
The units are smhd- (s)econds, (m)inutes, (h)ours or (d)ays.
The interval should be more than 10 seconds.
The exact timing of the restart is approximate.

ulimit

Use ulimit to modify the kernel-enforced resource limits for the job.
It takes a flag denoting which limit to set, and a value, e.g., ulimit -n 30.
Alternatively, you can list a series of ulimit settings in a curly brace block:
```
ulimit {
  -n infinity
  -l 10000
}
```
The values are all numeric or the keyword infinity.
To see the current kernel limits on a process, run cat /proc/<pid>/limits.
pmtr sets both the "hard" and "soft" ulimit to the same value.
If there is an error setting the ulimits, it’s logged to syslog.
The meaning of the flags like -n are explained in the bash shell itself:
In bash, type ulimit -a in bash to list flags, and the values in your shell;
In bash, type help ulimit for more information about the ulimits;
In bash, type man prlimit to see the more technical description and the units.

One-time jobs

For "initial setup" situations, pmtr has basic support for one-time jobs. The wait option makes pmtr wait for the job to exit before starting subsequent jobs. To keep pmtr from restarting it, use the once option.

job {
      name create-ramdisk
      cmd /bin/mkdir -p /dev/shm/ws/data
      wait
      once
}

Experimental

These features are disabled by default.

These configuration options may appear in pmtr.conf at the global scope.

report to udp://127.0.0.1:9999
listen on udp://0.0.0.0:10000

The report to option designates a remote address and port to which pmtr should send a a UDP packet every ten seconds. The packet payload lists the job names, enabled or disabled status, and elapsed runtimes in simple text. (An included script pmtr/utils/pmtr-rptserver can receive these reports and write updates to a text file summarizing the received status messages. This script is invoked like pmtr-rptserver -p 9999 -f status.txt). If the report to address falls in the multicast UDP range (e.g. 239.0.0.1, etc), the specification may include a trailing interface, e.g., report to udp://239.0.0.1:9999@eth2 to explicitly designate the interface from which the multicast UDP datagrams should egress.

The listen on option allows jobs to be remotely enabled or disabled. It specifies a UDP address and port that pmtr should listen on for datagrams of form enable abc or disable abc, where abc is a job name. The address 0.0.0.0 can be used as a shortcut to denote "any address" on this system. (A script in pmtr/utils/pmtr-ctl may be used to send the datagrams, for example pmtr-ctl -s 192.168.0.22 -p 10000 enable abc). The effect is temporary; the settings in pmtr.conf resume precedence when it’s edited or pmtr is restarted.

Tips

pstree

To get a nice visual confirmation of the jobs you have running under pmtr, you can use the pstree utility, if you have it installed. Here is the output of pstree -p.

Example pstree output

        ├─pmtr(2830)─┬─pmtr(2834)
        │            ├─proxy(2831)
        │            ├─sshd(2832)
        │            └─python(2833)

The other system processes have been omitted. Note that pmtr has its own child process. This is the dependency monitor that watches the config file.

Using pmtr with Docker

Docker is a Linux container technology. If you’re using Docker, you can use pmtr as the initial process of a Docker container to run a process hierarchy inside the container. Make sure pmtr runs in the foreground using the -F flag, since the container exits if the initial process detaches.

/usr/bin/pmtr -F -c config.conf

If you’re writing a Dockerfile to build your Docker images, you can use CMD to designate pmtr as default initial process for the image.

Dockerfile using pmtr

CMD [ "/usr/bin/pmtr", "-F", "-c", "/etc/container-pmtr.conf" ]

Resources

GitHub

Please use the pmtr Github page for bug reports and pull requests.

Author

The author’s GitHub page lists my other software.

License

See the LICENSE file.