Auteur: jeroen

Introduction

Like many people I use Zabbix for monitoring. I love the web GUI to configure stuff and the API to automate its configuration when I need to.

And although Zabbix comes packed with a lot of usable templates, they are more a starting point for your own infrastructure than a 100% ready solution.

Recently I had the need to start monitoring filesystem usage growth so I would get warned in time when a system was nearing its boundaries. It turns that for a really long time Zabbix has a timeleft function just for this occasion. But how, and where to use it?

Well, usually there is already a template available that does filesystem usage numbers. Those keep track on used space and used inodes. Within the template is a LLD, a Low Level Discovery rule. The result of that rule is a list of stuff found. And, together with ‘prototype’ items and triggers it can automatically add items and triggers to your host.

My setup

If you search in the available templates for ‘Linux filesystems’ you will easily find it. One is called ‘linux filesystems by Zabbix agent’ and the other one is ‘linux filesystems by Zabbix agent active’ (for if you are using active instead of passive checks). In the ‘Discovery’ column you can see that it has (in my case) 1 LLD.

If you click on the ‘Discovery’ you will see the list of LLD’s (in my case a single rule) with the following info:

Usually you will see four (4) item and trigger prototypes but my list shows 5 of each. Let’s start with the list if item prototypes:

As you can probably guess, the first one is the subject of this blog post. Let’s have a close look at it:

My item definition

If we analyse this item we see the following settings:

• Name: Since this item will expand (because it is part of a LLD!) it is important to add a macro (#FSNAME) to the name. This allows you to distinguish what filesystem this item is talking about later. Also, without it you would try to create multiple items with the same name and Zabbix would raise an error.
• Type: We are going to perform a calculation, so the item type is ‘calculated’
• Key: Since we are performing a calculation, this means there is an input variable. That is what is meant here. We are using the input value of vfs.fs.size.timeleft[{#FSNAME},pused]. This is one of the items already gathered in this LLD.
• Type of information: Since we are performing a calculation the result will be a number. That is why we select ‘Numeric (float) here.
• Formula: this is where it’s all about. ((((timeleft(//vfs.fs.size[{#FSNAME},pused],7d,95)/60)/60)/24)/30). This means: Calculate how much time is left for this filesystem to become 95% full, based on the last seven days of data. Since the result is in seconds we have to do some divisions to get to a number of month.

My trigger definition

Nice, but does this give us an alert when it becomes time to have a look at it? No, it doesn’t. For that we have to define a trigger. So we create a trigger prototype in the trigger section of the LLD. Mine looks like this:

Again, let’s take a closer look at the individual settings:

• Name: Like with the item name we need to add the #FSNAME macro in our descriptive text
• Severity: For me, I set the severity to high because an alert like this definitely deserves attention!
• Expression. This is the expression for when to trigger an alert: last(/Linux filesystems by Zabbix agent/vfs.fs.size.timeleft[{#FSNAME},pused])<3. This means: As soon as the last measured value of the item for this host with key vfs.fs.size.timeleft[{#FSNAME},pused] becomes below three (month) I get an alert.

After having configured this all correctly I head over to the ‘lastes data’ section of Zabbix to see how I’m doing:

Few! As you can see I am in the clear for now but one system will need disc size usage growth attention in one and a half (21-3=18 month) time.

If you found this helpfull please reward my work of researching and writing this. Please go to GitHub or Patreon and show your appreciation.

I have been working in the field of open source software since 1997-ish. Being a tech geek myself and loving computers I never understood back then why people would choose to use inferior buggy software.

I do now, and it’s called marketing. Still, in the early two thousands the term ‘open source’ caught on and popularity started to grow. Sure, at some points it was an uphill battle but after some twenty years were passed we could all look each other in the eye and say that we did it!

Look around you today. Just about every embedded system runs Linux, from TV’s to media centers. Heck, even the top 500 supercomputers in the world run mostly Linux.

In the realm of databases PostgreSQL rules big time. Every payment done through Facebook is stored on a PostgreSQL database on the site of the payment provider Adyen.

Have you taken a look at open source ERP lately? The open source Odoo system, programmed in Python, is really gaining momentum against the usual suspects. Today’s speed of doing business and optimizing business processes asks for better, faster and more agile software tools. Hence the choice for the Python programming language.

In December of 2021 a blog post caught my eye. “Open source is broken”. In it, the writer makes some valid points, pointing at underfunding of sometimes vital infrastructure. But is it all doom and gloom around the future of open source as we know it? Of course not! It is just a logical evolutionary step in its growth. Actually there was another major event surrounding this problem back in 2014.

On 7 April 2014, the “Heartbleed” bug in the OpenSSL software library was publicly disclosed and fixed. At the same time if became apparent that OpenSSL (the library that makes the difference between “http” and “https“) was extremely underfunded. Quickly the Linux Foundation stepped in and started the Core Infrastructure Initiative (CII) to fund crucial Internet infrastructure projects. But what about all these other projects? The very nice applications that are not vital to a working Internet?

Well, the response to these problems are starting to appear. Let me give you some examples:

The very popular Ardour software (to run a recording studio) only allows payed downloads of ready to install packages. Mind you, the software is still a 100% open source, but if you want to download an installer package, you have to pay. Or download and compile it yourself, that is also still an option. The developer does not ask for a lot. You can already subscribe for $1 per month. But with the popularity of the product, the numbers do add up. For him, that is. It is still nowhere near the numbers that larger companies are getting for their software.

The open source network analysis tool Ntop, uses a similar strategy. Still open source, but after installation you see a very prominent “Upgrade to Pro/Enterprise version” at the top of the screen. And some add-ons only work like demo software after which you need to buy a license (or compile stuff from scratch of course). Their source code is also still 100% open source.

The financial struggle in the open source world is no different than in the rest of the job world. People like to get paid a decent wage for their work. And it seems that offering paid packages as a convenience to the intended user is a promising direction for the future sustainability of open source projects.

Let’s start with a description of my needs. I have two remote systems and I want to be able to connect to them both. Both systems are behind a standard NAT firewall (like a home router). And I want to be able to copy files between them easily. I am not a vpn or network whizz but know my way around IP addresses. I know that besides WireGuard there are more options, like OpenVPN, but I prefer an easy setup with enough security. So I got hacking the other day and found a few small pitfalls. To help others setting this up I decided to write a small “The complete guide to setting up a multi-peer WireGuard VPN network”

Getting started is the easy part. There are enough guides on the Internet by now on how to get some initial setup. The thing is, after following those directions you are probably only half way there.

So, let’s get started.

First, take a piece of paper and draw the network you want to setup. Draw all hosts, and assign them all a unique IP-address in a new network that you are not already using. So, in my case, I choose 10.10.1.0/24. This means that my network is 10.10.1 and the last digit is for each systems IP address.

Since both systems are behind a firewall this means I can not access them from the outside world. This also means I need to have at least one system in my network that is (accessible from the outside world). For this I choose to instantiate a very cheap cloud virtual machine at some supplier. This will be my vpn-router-vm system. All it will do is route all traffic within my 10.10.1.0/24 vpn network. Of course it has a public IP address that is visible to the outside world.

Now that I have decided on all the above I can assign IP addresses to my two systems. It makes common sense to assign 10.10.1.1 to my vpn-router-vm. This also means my other nodes will be 10.10.1.2 and 10.10.1.3.

I instantiate the vpn-router-vm, and choose Ubuntu 20.04 for the OS. I do an apt update and apt upgrade to make sure I am using the latest patches. I install the UFW firewall tool and make it only accessible over SSH from my home server. No need for script kiddies to do dictionary attacks, right?

# apt install ufw
# ufw default deny incoming
# ufw default allow outgoing
# ufw allow from <my.home.ip.address> to any port 22 proto tcp comment 'ssh access from home'

Since this vpn-router-vm needs to be accessible from the outside world, the default port for WireGuard is 41194 and one of my systems does not have a fixed IP address I need to allow all WireGuard traffic:

# ufw allow 41194/udp

Now, on all our systems we are going to do exactly the same commands to install WireGuard, make a configuration directory, generate private key, and, based on this private key, a public key. So any system that connects with a public keys is checked whether or not this public key is based on our (hidden!) private key before it is allowed access.

# apt install wireguard
# mkdir -m 0700 /etc/wireguard/
# cd /etc/wireguard
# umask 077; wg genkey | tee privatekey | wg pubkey > publickey
# cat privatekey
# cat publickey

Okay, now, all our systems will get a new network interface with the name ‘wg0’. So we need to create a wg0.conf file in every /etc/wireguard directory. We will start with a skeleton configuration (yes, you will add stuff to this later, and for good reason) for our vpn-router-vm system. Note that in our interface definition we use /24 because we define our vpn network here.

## Set Up WireGuard VPN on Ubuntu By Editing/Creating wg0.conf File ##
[Interface]
## My VPN server private IP address ##
Address = 10.10.1.1/24
 
## My VPN server port ##
ListenPort = 41194
 
## VPN server's private key i.e. /etc/wireguard/privatekey ##
PrivateKey = private-key-of-von-router-vm

[Peer]
## Desktop/client VPN public key ##
PublicKey = public-key-of-my-first-peer-system
 
## client VPN IP address (note  the /32 subnet) ##
AllowedIPs = 10.10.1.2/32

On my first remote node I also create a wg0.conf file, but with slightly different contents:

[Interface]
## This Desktop/client's private key ##
PrivateKey = my-systems-private-key
 
## Client ip address ##
Address = 10.10.1.2/24
 
[Peer]
## Ubuntu 20.04 server public key ##
PublicKey = the-public-key-of-my-vpn-router-vm
 
## set ACL ##
AllowedIPs = 10.10.1.0/24
 
## Your Ubuntu 20.04 LTS server's public IPv4/IPv6 address and port ##
Endpoint = the-public-ip-address-of-my-vpn-router-vm:41194
 
##  Key connection alive ##
# This is needed because we are behind NAT firewall
PersistentKeepalive = 15

One thing to note in the text above is the last line. Since the system is behind a NAT firewall it is not accessible from the outside world. I like it that way. But it also means that this node has to ‘ping’ the vpn server from time to time.

Okay. All that is left now is to start WireGuard on the vpn-router-vm and on my first peer:

# systemctl enable wg-quick@wg0
# systemctl start wg-quick@wg0
# systemctl status wg-quick@wg0

The status should show something like/similar to this:

wg-quick@wg0.service - WireGuard via wg-quick(8) for wg0
     Loaded: loaded (/lib/systemd/system/wg-quick@.service; enabled; vendor preset: enabled)
     Active: active (exited) since Sat 2022-03-12 12:35:01 CET; 23h ago
       Docs: man:wg-quick(8)
             man:wg(8)
             https://www.wireguard.com/
             https://www.wireguard.com/quickstart/
             https://git.zx2c4.com/wireguard-tools/about/src/man/wg-quick.8
             https://git.zx2c4.com/wireguard-tools/about/src/man/wg.8
   Main PID: 1316620 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 38309)
     Memory: 0B
     CGroup: /system.slice/system-wg\x2dquick.slice/wg-quick@wg0.service

mrt 12 12:35:01 inzicht systemd[1]: Starting WireGuard via wg-quick(8) for wg0...
mrt 12 12:35:01 inzicht wg-quick[1316620]: [#] ip link add wg0 type wireguard
mrt 12 12:35:01 inzicht wg-quick[1316620]: [#] wg setconf wg0 /dev/fd/63
mrt 12 12:35:01 inzicht wg-quick[1316620]: [#] ip -4 address add 10.10.1.2/24 dev wg0
mrt 12 12:35:01 inzicht wg-quick[1316620]: [#] ip link set mtu 1420 up dev wg0
mrt 12 12:35:01 inzicht systemd[1]: Finished WireGuard via wg-quick(8) for wg0.

Assuming they are active and working properly on both systems you should now be able to ping one another:

# ping 10.10.1.1
PING 10.10.1.1 (10.10.1.1) 56(84) bytes of data.
64 bytes from 10.10.1.1: icmp_seq=1 ttl=64 time=13.3 ms
64 bytes from 10.10.1.1: icmp_seq=2 ttl=64 time=12.4 ms
^C
--- 10.10.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.441/12.875/13.309/0.434 ms

All rejoice for you have a working VPN!! \0/. Unfortunately your not done yet. Sorry.

So add a second remote system to your vpn setup by adding another “peer” section to the wg0.conf file on the vpn-router-vm and configure the second remote system like you did before, taking care of course of the other peer’s private key.

If all goes according to plan then that server is capable of pinging your vpn-router-vm. Again, we all rejoice \o/.

Now try to ping one of the remote systems from the other remote system. I am guessing it doesn’t work. That’s a bummer but can easily be fixed. The thing is that usually a default Linux system does not automatically do forwarding of IP packets. To enable IP forwarding you need two command:

# cat /proc/sys/net/ipv4/ip_forward  # <- probably this is zero/0
# sysctl -w net.ipv4.ip_forward=1    # store setting in startup config file
# echo 1 > /proc/sys/net/ipv4/ip_forward  # enable IP forwarding on running system

So, can you now ping the remote system from the other remote system? Yes, you can! Again, we all rejoice \o/. Surely you can now also ssh into a remote system from the other remote system? And, again, bummer, you can’t. Something is prohibiting access to the ssh port from the remote system. What can it be? Yes, the firewall on the vpn-router-vm system of course! So, you add a few lines to the wg0.conf on the vpn-router-vm to enable traffic to all ports in the vpn network (All credits to user ‘dddma’ on Reddit for this). Your wg0.conf file on the vpn-router-vm will now look like this (Both PostUp and PostDown are very long single lines!):

## Set Up WireGuard VPN on Ubuntu By Editing/Creating wg0.conf File ##
[Interface]
## My VPN server private IP address ##
AddreThe complete guide to setting up a multi peer WireGuard VPN networkss = 10.10.1.1/24
 
## My VPN server port ##
ListenPort = 41194
 
## VPN server's private key i.e. /etc/wireguard/privatekey ##
PrivateKey = private-key-of-von-router-vm

#Allow forwarding of ports
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
## Desktop/client VPN public key ##
PublicKey = public-key-of-my-first-peer-system
 
## client VPN IP address (note  the /32 subnet) ##
AllowedIPs = 10.10.1.2/32

[Peer]
## Desktop/client VPN public key ##
# dskdesk
PublicKey = public-key-of-my-second-peer-system
 
## client VPN IP address (note  the /32 subnet) ##
AllowedIPs = 10.10.1.3/32

Don’t forget to do a “systemctl restart wg-quick@wg0” when you change a config file. Anyway, that’s it. Your done. Enjoy! I hope you enjoyed this “Complete guide to setting up a multi-peer WireGuard VPN network”. No likes needed. Have a nice day.

If you found this helpfull please reward my work of researching and writing this. Please go to GitHub or Patreon to show your appreciation.

En toen kwam STAP in het nieuws. Natuurlijk een mooi acroniem voor een regeling die voluit STimulering ArbeidsmarktPositie heet. Ik laat de nogal geforceerde kronkel van regeling naar afko voor wat het is. We hebben grotere problemen in dit land. Zin in het volgen van een cursus? Kijk dan even op https://www.stap-budget.nl/opleidingen. Prima. Maar hoe zit het met de opleiders?

Op de url https://www.stapvooropleiders.nl/ is meer info te vinden voor opleiders. Een opleider kan pas mee doen met STAP als hij is opgenomen in het STAP-scholingsregister. En daarvoor moet je natuurlijk an criteria voldoen:

Een scholingsactiviteit kan pas in het STAP-scholingsregister worden geregistreerd, als de opleider is opgenomen in het register. Hiervoor heeft hij 1 van deze 4 erkenningen nodig:

Kijken we naar deze criteria dan lezen we:

• De opleider is erkend door het ministerie van OCW.
• De opleider beschikt over het NRTO-keurmerk.
• De opleider biedt opleidingen aan die leiden tot een door het NLQF ingeschaalde kwalificatie.
• De opleider is erkend door een sector- en brancheorganisatie (pdf, versie 1.9).

Prachtig! Er is nagedacht over een vorm van kwaliteitsborging en er is een drempel ter voorkoming van wildgroei. Zodat er geen geld uit een goed bedoelde regeling bijvoorbeeld gaat naar een cursus Boomknuffelen (don’t ask…). Natuurlijk zijn er toch ook weer journalisten die als azijnpissers op zoek gaan naar waar het mis zou kunnen gaan maar bij nadere bestudering lijkt het wat mij betreft om een mooie regeling te gaan. Mits…..

De ICT

Laten we eens kijken naar één van de opleidingsgebieden, de ICT.

Algemeen wordt erkend dat er een schreeuwend tekort is aan ICT-ers. Niet alleen bij de Belastingdienst, maar elk bedrijf dat ik ken “bends over backwards” om aan goede ICT-ers te komen.

In deze markt ken ik een heel groot aantal eenpitters of kleinere opleiders die fantastische docenten zijn en vanuit hun kleine bedrijfje opleidingen verzorgen. En die kunnen zelf, ook al zijn ze ondernemer, hun kansen om in het STAP-scholingsregister te worden opgenomen wel op hun buik schrijven.

Waarom? Wel, laten we eerder genoemde criteria nog eens naast elkaar zetten:

Criterium	Argument
De opleider is erkend door het ministerie van OCW.	Alleen maar realistisch voor bestaande scholen en universiteiten
De opleider beschikt over het NRTO-keurmerk.	Ja, mits je omzet minimaal €150.000 is en je minimaal 51% van je omzet uit opleiden haalt.
De opleider biedt opleidingen aan die leiden tot een door het NLQF ingeschaalde kwalificatie.	Als je niet beschikt over een accreditatie/keurmerk/toezicht van de Inspectie van het Onderwijs of NVAO ITK, Stichting Examenkamer, SNRO, Vembo, of als je geen NRTO, NIAZ of ISO 9001 kwaliteitslabel hebt, dan kun je voor €2500,- een audit doorlopen om te kijken of je geschikt bent voor het verkrijgen van een ingeschaalde kwalificatie.
De opleider is erkend door een sector- en brancheorganisatie (pdf, versie 1.9).	Top! Nou, dat moet dan toch wel lukken? Eens kijken of er een ICT sector- of brancheorganisatie tussen staat. Goed, niet dus. Zelfs van de gedoodverfde kandidaat hiervoor, NLdigital, ontbreekt ten ene male elk spoor.

Kortom, als briljante eenpitter of kleine opleider kun je fluiten naar het in aanmerking komen voor opname in het STAP-scholingsregister. Goede docenten komen niet in STAP-scholingsregister. Jammer, pech, helaas.

Zijn er alternatieven?

Natuurlijk zijn er best wel een aantal hele goede (grotere) opleidingsinstituten zoals LOI en ICTtrainingen.nl die in spelen op de STAP-regeling. En daar kunnen mensen natuurlijk prima terecht.

Maar die hele goede eenpitter of kleine opleider die al 40 jaar in de ICT zit, diverse ICT boeken heeft geschreven en ook af en toe zelf een cursus organiseert, die kan fluiten naar deze regeling.

Toch wel een beetje een stap omlaag (sorry, kon het niet laten).

En toen was het ineens februari 2021. De tijd vliegt. Te snel om een persoonlijke blog bij te houden met periodieke mentale boeren. Ergo, ik schrijf nog steeds. Veel en vaak. Boeken en software. En doe nog veel meer dingen. Druk baasje van nature maar zonder een testosteron-kanon te zijn (denk ik dan). Anyway, er komt vanzelf wel weer een volgende post. Of niet. En anders is er altijd www.i2rs.nl nog, of Twitter. Ciao!