Pmtr is one of my many C projects. Back to the pmtr Github page.
I wrote pmtr as a simple supervisory daemon for my own needs. Pmtr runs under systemd as well as other init mechanisms like sysvinit, etc. It can also run as process 1 inside a container. When installing on many flavors of Linux, it can detect the host init system and set itself up to start at boot appropriately. I wrote pmtr with a few goals in mind:
-
to have one configuration file listing all processes to run
-
to exist under (not to replace) the host init system
-
to run under various host init systems
-
to consume few resources
-
to have few features
-
no dependencies
It is written in C, supports Linux only, and is MIT licensed.
job {
name tunnel
cmd /usr/bin/ssh -i key -TNL 5901:127.0.0.1:5901 192.168.0.1
}
job {
name bitcoin-daemon
dir /home/bitcoin
cmd /usr/local/bin/bitcoind
user bitcoin
env HOME=/home/bitcoin
cpu 0-8
}
job {
name capture
dir /data
cmd /usr/sbin/tcpdump -i eth0 -s 0 -G 10 -w %Y%m%d%H%M%S.pcap
}
When pmtr is started, it starts all the jobs in pmtr.conf
, likewise, pmtr
terminates them when it is stopped.
In practice I let the host init system- often systemd- manage all the standard OS services. I use pmtr to run the local software- the parts that make up an appliance, often comprising a dozen or two dozen jobs that function together.
There are many options to use inside the job, listed later below.
Processes that run under pmtr should stay in the foreground, exit on SIGTERM or SIGKILL, and clean up after their own sub-processes when exiting.
If a job exits, pmtr restarts it. If it exits too quickly- less than 10 seconds after it started- pmtr delays its restart 10 seconds to avoid rapid cycling.
If the operator edits pmtr.conf
and saves the file, the changes take effect
immediately. There is no need to tell pmtr to reload its configuration file.
Platforms
Pmtr runs on these flavors of Linux, and probably others:
-
Arch
-
Alpine
-
Debian
-
Ubuntu 12-18
-
CentOS/RHEL 5-8
-
Amazon Linux AMI 1/2
-
Raspberry Pi Raspbian
-
Beaglebone Black Debian
-
Yocto layer
Build & Install
Option 1: RPM
A RHEL/CentOS 7 x86_64 RPM package for pmtr can be found here.
Option 2: Build from source
To build pmtr from source follow these steps.
Install the prerequisite tools:
sudo apt install git build-essential autoconf automake
sudo yum install git gcc autoconf automake make
git clone https://github.com/troydhanson/pmtr.git
cd pmtr
./autogen.sh
./configure --bindir=/usr/bin --sysconfdir=/etc
make
sudo make install
sudo touch /etc/pmtr.conf
Do this to start pmtr automatically at boot.
cd initscripts
sudo ./setup-initscript --auto
Using setup-initscript --auto
detects common init managers like systemd and
sets up pmtr to start at boot. Run ./setup-initscript --help
to see further
options. Here’s an example of running it on Ubuntu based on systemd:
sudo ./setup-initscript --auto
initsystem : systemd
pmtr : /usr/bin/pmtr
services : /etc/systemd/system
initscript : /etc/systemd/system/pmtr.service
enabling : systemctl enable pmtr
starting service : systemctl start pmtr
You can look at the syslog to confirm it is now running.
grep pmtr /var/log/syslog # ubuntu
grep pmtr /var/log/messages # redhat/centos
You should see something like:
Nov 13 17:13:10 ubuntu systemd[1]: Started pmtr process manager.
Nov 13 17:13:10 ubuntu pmtr[10888]: pmtr: starting
It is always a good idea to look in the syslog after making changes to the configuration file. This is where pmtr reports on starting jobs, or on any errors in parsing the configuration file. Any output generated by the jobs also appears in the syslog, by default.
Config file
Overview
The configuration file /etc/pmtr.conf
drives pmtr.
The config file can be changed using a command line option (-c
) when running
pmtr from the command line.
As shown previously it consists of zero or more job definitions. A job is just a process definition. It looks like:
job {
name bitcoin-daemon
dir /home/bitcoin
cmd /usr/local/bin/bitcoind
user bitcoin
env HOME=/home/bitcoin
}
A job must have a name
(used for logging only) and a cmd
. Everything else
in the curly braces optional.
-
Order does not matter.
-
Indentation is optional.
-
Blank lines are ok.
-
Comments start with
#
.
Options
The full list of options that may appear inside a job are listed here.
option | argument |
---|---|
|
descriptive job name used for logging - must be unique |
|
executable (fully-qualified) and any arguments |
|
working directory (fully qualified) to run the process in |
|
send stdout to this file |
|
send stderr to this file |
|
take stdin from this file |
|
environment variable to set (repeatable) |
|
unix username under whose id to run the process |
|
unix priority between -19 (highest) and 20 (lowest) |
|
CPU affinity as hex mask (0xABCD) or number/ranges (0,2-4) |
|
process ulimits |
|
a time interval to restart the process |
|
files to watch, any changes induce the job to restart |
|
disable the job |
|
(special) wait for the job to finish before going on |
|
(special) do not restart the job |
More details on each option follows.
cmd
-
Specifies the absolute path to the executable (there is no $PATH searching!)
-
It may have arguments after the executable (e.g.
cmd /usr/bin/python rest.py
) -
Use double-quotes to make one argument from the quoted string.
-
No shell expansion: no wildcards, backticks, variables, etc.
If you need shell features in your command, wrap it with a shell script. You can also use a quoted string as a shell script like this:
job {
name api
cmd /bin/bash -c "ifconfig eth1 up; sleep 5; exec /api/rest.py"
}
env
-
Use to push an environment variable into a job, e.g.
env DEBUG=1
. -
Use repeatedly on separate lines to set multiple environment variables.
disable
-
Use
disable
on a line by itself to make the job disabled. -
This is sometimes quicker than commenting out the whole job.
out, err, in
-
Use
out
anderr
to send stdout or stderr to a file. -
stdout and stderr go to syslog by default.
-
stdin defaults to
/dev/null
; usein
to override
nice
-
This changes the process priority
-
Takes a number in the range -19 (highest priority) to 20 (lowest)
cpu
-
This sets the CPU affinity- the list of CPU’s the task can run on
-
Takes a CPU number (e.g. 0) or range (e.g. 2-4) or a mix (e.g. 0,2-4)
-
Alternatively, can take a 0x-prefixed hex mask (e.g. 0x8f)
-
Any CPUs in the set that are physically absent are ignored
user
-
Specifies the unix username to run the process as.
-
That user’s uid/gid becomes those of the process.
-
Defaults to root (when pmtr is running as root)
depends
-
Specifies a block of one or more files that the job depends on.
-
pmtr watches the dependencies for changes to their content.
-
Pmtr restarts the job if a change is detected.
job { name bitcoin-daemon dir /home/bitcoin cmd /usr/bin/bitcoind user bitcoin env HOME=/home/bitcoin depends { /home/bitcoin/.bitcoin/bitcoin.conf } }
bounce every
-
Use
bounce every
to restart a job on a periodic interval. -
It takes a number and unit [smhd] e.g.
bounce every 1d
. -
Units [smhd] are seconds, minutes, hours or days.
-
The exact timing of the restart is approximate.
ulimit
-
Use to modify the system resource limits for the job.
-
Takes a flag and value, e.g.,
ulimit -n 30
. -
Values are numeric or the keyword
infinity
.
To see the current limits on a process by its PID:
cat /proc/<pid>/limits
Pmtr sets both the "hard" and "soft" limit to the same value. Any error in setting the limit is logged to syslog.
See man prlimit
for technical descriptions of each limit.
In the bash
shell, ulimit -a
and help ulimit
display the
limits and a list of flags and descriptions respectively.
|
the maximum size of core files created |
|
the maximum size of a process’s data segment |
|
the maximum scheduling priority (taken as 20-limit) |
|
the maximum size of files the process may create |
|
the maximum number of pending signals |
|
the maximum bytes a process may lock into memory |
|
the maximum resident set size |
|
the maximum number of open file descriptors |
|
the maximum number of bytes in POSIX message queues |
|
the maximum real-time scheduling priority |
|
the maximum stack size |
|
the maximum amount of cpu time in seconds |
|
the maximum number processes or threads |
|
the maximum size of process virtual memory |
The units are discussed in the prlimit(2) man page.
wait/once
-
Used for setup jobs that need not be restarted.
-
wait
pauses the startup of subsequent jobs. -
once
tells pmtr not to restart this job.job { name initial-setup cmd /bin/mkdir /dev/shm/go wait once }
Operator notes
Pmtr detects writes to pmtr.conf
and applies the changes immediately.
-
A newly-added job gets started.
-
A deleted job is terminated.
-
A changed job is restarted with its new configuration.
You can run pmtr -t
to check the config file syntax and report any errors.
Logs
Logging from pmtr goes to syslog. Typically these logs go to /var/log/syslog
or
/var/log/messages
. Check the syslog after any change to pmtr.conf
. Errors in
parsing pmtr.conf
are logged there, as well as job start or exit events. On
systemd-based hosts, you can also use journalctl -u pmtr
to see pmtr logs.
Status and control
To start, stop or get status of pmtr, use the host system management commands.
# under systemd
systemctl start pmtr
systemctl stop pmtr
systemctl status pmtr
or
# other initsystems
service pmtr status
service pmtr start
service pmtr stop
Job exits
If a job terminates by itself, when pmtr did not signal it to exit, (and the
job does not have the once
option), pmtr restarts it. However, if it exited
within 10 seconds of when it started, pmtr waits 10 seconds to restart it. The
10-second wait prevents rapid process cycling. Also, a job that’s waiting for
something (like a network resource to come up) can be designed to exit instead
of retry, relying on pmtr to restart it periodically to try again.
Pmtr terminates a job when it is deleted, disabled, or altered in pmtr.conf
,
or is being bounced due to the bounce every
option; or because pmtr itself is
being shut down. To terminate a job, pmtr sends SIGTERM to it, then SIGKILL
shortly afterward, if it’s still running.
Command line options
The host init system normally runs pmtr at system boot. You can instead run pmtr manually. Or, it can be the init process inside a container. In these use cases, the following command line options may be useful.
|
show help |
|
stay in foreground (enabled by default when PID 1) |
|
echo syslog to stderr (enabled by default) |
|
specify configuration file |
|
test syntax (parse config file and exit) |
|
verbose logging (repeatable), -vv shows parsing |
|
make pidfile |
In a container
When pmtr is the init entrypoint (process 1) in a container, it stays in the foreground by default. An example invocation command is shown.
# container entrypoint
/usr/bin/pmtr -IFc /path/to/pmtr.conf
UDP control
Note
|
This feature is disabled by default. |
These options may appear in pmtr.conf
at the global scope.
report to udp://127.0.0.1:9999
listen on udp://0.0.0.0:10000
The report to
option designates a remote address and port to which pmtr
should send a a UDP packet every ten seconds. The packet payload lists the job
names, enabled or disabled status, and elapsed runtimes in simple text. If the
report to
address falls in the multicast UDP range (e.g. 239.0.0.1, etc), the
specification may include a trailing interface, e.g., report to
udp://239.0.0.1:9999@eth2
to designate the interface from which the multicast
UDP datagrams should egress.
The listen on
option allows jobs to be remotely enabled or disabled. It
specifies a UDP address and port that pmtr should listen on for datagrams of
form enable abc
or disable abc
, where abc is a job name. The address
0.0.0.0 can be used as a shortcut to denote "any address" on this system. The
effect is temporary; the settings in pmtr.conf resume precedence when it’s
edited or pmtr is restarted.
These options are considered experimental and may be replaced or removed.