Determine if your system is using systemd

Systemd is a suite of basic building blocks for a Linux system that provides a system and service manager that runs as PID 1 and starts the rest of the system. It works as a replacement for sysvinit.

Almost all popular distros are using systemd, but if you need to use old version of the popular distros, or simply some non popular distros, here is how you can determine whether the linux your are having is using systemd, or not.

First, check which process is running PID 1.

$ ps 1

The output, most of the time will be /sbin/init. 

Now we need to determine if /sbin/init is a symlink to something else. There are 2 commands we can use to achieve that:

1. use stat

$ stat /sbin/init

2. use readlink

$ readlink /sbin/init

In a systemd based system, you will be getting output showing that /sbin/init is actually a symlink to /lib/systemd/systemd, where as in other system, you will not get that output.

Reducing disk usage by journalctl logs

Systemd is using journalctl as the log keeper for its services. The log location is at /var/log/journal.

To check the disk usage of journalctl log

$ sudo journalctl --disk-usage

To check the logs date, just run 

$ sudo journalctl 

To flush active logs to file, so it can be cleared (vacuum)

$ sudo journalctl --flush

To clear the log, we can use "journalctl --vacuum-size". This below command will reduce the total size used by journalctl log to 1G. This command however, will only work on archived logs, and not active logs.

$ sudo journalctl --vacuum-size=1G

We can also vacuum by date, this command will remove data older than a month. Same with vacuum-size, this will work only to the archived logs.

$ sudo journalctl --vacuum-time=1month 

We can also ask journalctl to rotate its log files, using --rotate. The rotation criteria can be set in /etc/systemd/journald.conf, by editing MaxFileSec and MaxRetentionSec configuration.

$ sudo journalctl --rotate

Autostart podman containers on host boot/reboot

Podman is an excellent alternative to docker, but it would not survive host reboot since it does not have any daemon running. Is all hope lost? Nope, podman has one trick up its sleeve that can save the day.

Introducing "podman generate" command. This command can generate yaml, json or systemd file for any containers. In this case, we are going to generate a systemd script for our container, to make sure it survives host reboot.

How to use this tool? First we need a running container.

Once we have a running container, simply run below command to create systemd service file for your container (in this example, mycontainer is the name of the container), and save it to a file called container-mycontainer.service.

$ podman generate systemd --new --name mycontainer -f

Then, move the conatainer-mycontainer.service file into /etc/systemd/system, for systemd to start recognizing your service (we changed the name to just mycontainer.service, to shorten the name, and make it easier to type). 

$ sudo mv container-mycontainer.service /etc/systemd/system/mycontainer.service

After that, we need to reload the systemd for the root user, to make systemd aware of the new service.

$ sudo systemctl daemon-reload

Now, start the new service, and enable it on every boot

$ sudo systemctl enable --now mycontainer.service

Check whether your new service is running

$ sudo systemctl status mycontainer.service

Scheduling Tasks With systemd.timer

Systemd.timer is a way to scheduling jobs and tasks in systemd based linux system. The systemd timer units are identified by ".timer" (dot timer) file name extension, compared to ".service" for service units. 

Each timer file requires a service file for it to work. In other word, systemd timers only can schedule systemd services.  

Some of systemd.timer features:

1. timer is managed by systemd similar to other units, using systemctl command
2. timers can be triggered by calendar event, or triggered by specific time elapsed from a certain starting point
3. time units are logged to journal, for easier troubleshooting and monitoring
4. if the system is off during the expected execution time, the timer will be executed when the system is running again

The usage of timer is best explained using an example. Let's say we want to create a timer to run a script called hello.sh.

First, we need to create the script, and get the location of the script. Let's say we created the script inside /usr/local/bin/hello.sh

Next, we need to create a systemd service unit for the above script. Just insert below settings into /etc/systemd/system/hello.service

[Unit]

Description="Hello app"

[Service]

ExecStart=/usr/local/bin/hello.sh

Then, create a systemd timer unit for hello.service. Just add below setting into /etc/systemd/system/hello.timer

[Unit]

Description="Run hello.service 5min after boot, every 24 hours relative to activation time, and everyday at 10:00 am"

[Timer]

OnBootSec=5min

OnUnitActiveSec=24h

OnCalendar=Mon..Fri *-*-* 10:00:*

Unit=helloworld.service

[Install]

WantedBy=multi-user.target

The above example will run the script after 5 minutes of system boot, 24 hours after the timer activation time, and everyday at 10:00am. The format for OnCalendar is as below

Day of Week = Sun to Sat

Date = in yyyy-mm-dd format

Time = in hour:minute:seconds format

For example, if we want to execute the script at 3:00 pm every Monday and Friday

OnCalendar=Mon,Fri *-*-* 15:00:00

Or execute the script at 4:00 pm 20 February 2023

OnCalendar=2023-02-20 16:00:00

For more example on setting the timer, we can type this command to see the manual page of systemd.time

# man 7 systemd.time

Before we use the new systemd unit files for hello.sh, we can verify that they are error free

# systemd-analyze verify /etc/systemd/system/hello.*

If no error detected, then we can start the timer

# systemctl start hello.timer

We can also enable the timer on boot

# systemctl enable hello.timer

To check the status of timer

# systemctl status hello.timer

To list all active timers

# systemctl list-timers 

To list all timers, regardless whether they are active or note

# systemctl list-timers --all

To stop the timer

# systemctl stop hello.timer

Adding custom nameserver in systemd-resolve

The old /etc/resolve is now being managed by systemd-resolve service, which is part of systemd. In order to add new nameserver, please follow below steps

1. Create a directory named /etc/systemd/resolved.conf.d/

# mkdir /etc/systemd/resolved.conf.d

2. Add a new configuration file for your new dns server. Let's say we want to add google's dns ip address, which are 8.8.8.8 and 8.8.4.4 

# cat >> /etc/systemd/resolved.conf.d/mynameserver.conf <<EOF

[Resolve]

DNS=8.8.8.8 8.8.4.4

EOF

3. Restart the service

# systemctl restart systemd-resolved

4. Verify that your dns is now being used by the system

# systemd-resolve --status

Global

       LLMNR setting: no                  

MulticastDNS setting: no                  

  DNSOverTLS setting: no                  

      DNSSEC setting: no                  

    DNSSEC supported: no                  

         DNS Servers: 8.8.8.8             

                                8.8.4.4             

...

For more information about what option can be included in the configuration file, please refer to resolved.conf man page.

Getting Access Denied Error when Using systemctl as root

 I got this error in one of our server, when trying to restart nginx

# systemctl status nginx

Failed to get properties: Access denied

Does not make sense, I am a root user. After some searching, a few suggestions came. 

The first suggestion was to restart systemctl daemon:

# systemctl daemon-reexec

That did not work for me. Another solution is to disable selinux temporarily, but this also did not work for me:

# setenforce 0 

The last thing that I tried (that actually worked) was to sending sigterm to systemd, and it will restart by itself:

# kill -TERM 1

If you guys happened to encounter this sort of error, you can try all the above. Some might suit you better than the other.

Set Bitnami Wordpress Application to Start on Boot on Ubuntu 20.04

To set a bitnami wordpress application to start on boot, you just need to make sure that systemctl can use the ctlscript. To do that:

1. Locate the installation directory of your bitnami wordpress. If you are following the default setting, the location should be in /opt

2. Locate the ctlscript.sh file, which is the script to start, stop and restart bitnami-application.

$ sudo find / -iname "ctlscript.sh" -type f

/opt/bitnami-wordpress-5.6-0/ctlscript.sh

3. Copy the script to /etc/init.d, and rename it to bitnami-wordpress

$ sudo cp /opt/bitnami-wordpress-5.6-0/ctlscript.sh /etc/init.t/bitnami-wordpress

4. Edit the script, and add "INIT INFO" settings below #!/bin/sh to ensure systemd can understand the script

#!/bin/sh

### BEGIN INIT INFO

# Provides:          bitnami-wordpress

# Required-Start:    $remote_fs $syslog

# Required-Stop:     $remote_fs $syslog

# Default-Start:     2 3 4 5

# Default-Stop:      0 1 6

# Short-Description: Start daemon at boot time

# Description:       Enable services provided by daemon.

### END INIT INFO

...

5. Save the file, and make systemd aware of the new startup script

$ sudo systemctl daemon-reload

6. Test the new bitnami-wordpress script

$ sudo systemctl status bitnami-wordpress

7. Enable bitnami-wordpress on boot

$ sudo systemctl enable bitnami-wordpress

8. Test it out by rebooting your machine...

$ sudo reboot

9. ...and check the status of the bitnami-wordpress service after reboot

$ sudo systemctl status bitnami-wordpress

Sleep / suspend linux from command line

To suspend a linux machine using command (useful if you want to suspend a machine that is connected via ssh), the command is pretty simple

$ sudo systemctl suspend

That's all.