Some friends and I have been toying with setting up our own little mesh VPN dark net of sorts. Below are the first baby steps I’ve been able to take in setting up ZeroTier on OPNSense. A few things this configuration enables are as follows.
- A self hosted ZeroTier controller through ZTNCUI
- Bridging the ZeroTier subnet to locally provisioned devices without the need to have the ZeroTier client installed.
- NATing of other local subnets which are not natively in the ZeroTier network to be able to mostly transparently route to hosts in the ZeroTier network.
- Afformentioned NATing allows port forwarding to these local hosts not natively in the ZeroTier network through the network bridge.
- Easy to remember IP allocation bounds.
01 Set up a VM to host ZTNCUI.
I set up a VM with Debian10.
01 A Install OS.
The OS install is outside the scope of this guide, but below are some items to ensure are configured.
Give the VM at least 32 GB storage for scratch space and future growth.
Set up an administrative user other than root.
If you have a local domain set the domain name. The domain I use in my internal LAN is just “lan”.
Set strong passwords.
01 B If you used Debian then set up sudo access for the admin user.root@Deb10:~# sed -i "s/$(grep sudo /etc/group)/$(grep sudo /etc/group)/g" /etc/group
01 C Again if you used Debian edit apt sources to exclude the install cd/dvd.root@Deb10:~# grep -i "s/deb cdrom/#deb cdrom/g" /etc/apt/sources.list
01 D Update OS, install some requisites and quality of life packages then reboot.root@Deb10:~# apt update && apt full-upgrade -y
root@Deb10:~# apt install vim curl wget dnsutils
root@Deb10:~# /sbin/reboot
01 E Install ZeroTier and ZTNCUI.
https://www.zerotier.com/download
https://key-networks.com/ztncuijohn@Deb10:~# sudo curl -s 'https://raw.githubusercontent.com/zerotier/ZeroTierOne/master/doc/contact%40zerotier.com.gpg' | gpg --import && if z=$(curl -s 'https://install.zerotier.com/' | gpg); then echo "$z" | sudo bash; fi
john@Deb10:~# sudo apt-get install ./ztncui_0.5.8_amd64.deb
john@Deb10:~# sudo sh -c "echo 'HTTPS_PORT=3443' > /opt/key-networks/ztncui/.env"
john@Deb10:~# sudo systemctl restart ztncui
01 F Log onto ZTNCUI and change the default credentials.
In a web browser navigate to https://127.0.0.1:3443 on the ZTNCUI host.
Log on with the following credentials.
UserName: “admin”
PassWord: “password”
Change the password to something more secure.
Log out then log back in.
01 G Create the ZeroTier Network and define the subnet.
In the top panel select “Add network”.
Define the network name. In this example we will use the name “DorkNet” to refer to our example ZeroTier network.
Select “Create Network”
Make note of the Network ID.
Select “easy setup”
Define your subnet and IP assignment pools. In this example we will use the following values. Some further explanation to follow in next steps.
“Network address in CIDR notation” = “10.222.0.0/16”
“Start of IP assignment pool” = “10.222.201.1”
“End of IP assignment pool” = “10.222.255.254”
02 Log onto your OPNSense router as an administrative user.
02 A Install and activate the ZeroTier plugin then add the previously created DorkNet network.
Navigate: System > Firmware > Plugins
Scroll to the bottom and click the “+” to install the ZeroTier Plugin.
Navigate: VPN > ZeroTier > Settings
Select the “Enabled” check box.
Navigate: Networks
Select “+”
Fill out the “Network ID” and “Local Description”.
Select “Save”
Select “i” button for the newly added network.
Make note of the first space delimited string. This is the Leaf ID of the bridge router.
Select the check box under the column “Enabled” to enable the network.
02 B Approve the OPNSense VPN client and assign it a static IP.
Open the ZTNCUI WebUI.
Navigate: Networks > DorkNet / members
Identify the Leaf ID of the newly enabled OPNSense ZeroTier client / bridge router.
Select “Authorized” and “Active Bridge”.
Wait some moments and select the “Refresh” button. An IP address should be assigned to this leaf.
Select on the IP address which has been assigned to this leaf.
Identify a new static IP address for this bridge. Since I am using a large(ish) /16 subnet for the DorkNet I chose to break the bridge IPs into /24 subnets so that each bridge could then have roughly a /24 subnet of natively routable internal clients(excluding NAT clients) and be very easy to remember where the IP allocation bounds lay. Below are some example IP allocations for this scheme.
1st Bridge Router IP: 10.222.1.1
1st Bridge Client IPs: 10.222.1.{2-254}
2nd Bridge Router IP: 10.222.2.1
2nd Bridge Client IPs: 10.222.2.{2-254}
17th Bridge Router IP: 10.222.17.1
17th Bridge Client IPs: 10.222.17.{2-254}
Define the IP Address for this Bridge Router then click the “+” to assign it.
Select the trash can icon next to the old IP allocation to delete it.
Select “Back”.
Open the OPNSense WebUI.
Navigate: VPN > ZeroTier > Overview > Networks
Select the down arrow next to the DorkNet Network ID to expand it.
Validate the IP allocation is accurate. If not then you may want to try cycling the ZeroTier plugin.
02 C Set up the ZeroTier interfaces and bridge interface
Navigate: Interfaces > Assignments
At the bottom of the screen next to “New interface:” select the drop down box and select the interface starting with “zt”.
Select the “+” button then click “save”.
Under the column “Interface” click on the name of the newly assigned interface for the “zt” “NIC”.
Select “Enable Interface”.
Select “Prevent interface removal”.
Add “ZT_DN_GW” to the “Description” field.
Select “Save”.
Navigate: Interfaces > Assignments
At the bottom of the screen next to “New interface:” select the drop down box and select the interface you wish to use as the internal LAN interface to route to the clients and servers hosted locally.
Select the “+” button then click “save”.
Under the column “Interface” click on the name of the newly assigned interface for the LAN “NIC”.
Select “Enable Interface”.
Select “Prevent interface removal”.
Add “ZT_DN_LAN” to the “Description” field.
Select “Save”.
Navigate: Interfaces > Other Types > Bridge
Select “+ Add”.
Under the “Member interfaces” drop down box select the “ZT_DN_GW” and “ZT_DN_LAN” interfaces.
Add “ZT_DN_RB” to the “Description” field.
Select “Save”.
Navigate Interfaces > Assignments
At the bottom of the screen next to “New interface:” select the drop down box and select the bridge interface just created. It should be something like “bridge1”.
Select the “+” button then click “save”.
Under the column “Interface” click on the name of the newly assigned interface for the Bridge “NIC”.
Select “Enable Interface”.
Select “Prevent interface removal”.
Add “ZT_DN_BR” to the “Description” field.
Select “Static IPv4” as the “IPv4 Configuration Type”.
Add the Bridge Router IP previously assigned in the ZTNCUI under “IPv4 address”.
Select “Save”.
02 D Modify relevant tunables
Navigate: Settings > Tunables
Modify “net.link.bridge.pfil_member” to be “0”
Modify “net.link.bridge.pfil_bridge” to be “1”
02 E Define relevant firewall pass rule
Navigate: Firewall > Rules > ZT_DN_BR
Click “+ Add”
“Action” = “Pass”
“Quick” = “Unselected”
“Interface” = “ZT_DN_BR”
“Direction” = “in”
“TCP/IP Version” = “IPv4”
“Protocol” = “Any”
“Source” = “ZT_DN_BR net”
“Destination” = “ZT_DN_BR net”
Select “Save”.
02 F Optional set up NAT for other local subnets into the DorkNet.
Navigate: Firewall > NAT > Outbound
Select “Hybrid outbound NAT rule generation”
Select “Save”
Select “+ Add”
“Interface” = “ZT_DN_BR”
“TCP/IP Version” = “IPv4”
“Protocol” = “any”
“Source address” = Whatever LAN subnet you may have for your other internal traffic
“Source port” = “any”
“Destination address” = “ZT_DN_BR net”
“Destination port” = “any”
“Translation / target” = “Interface address”
Select “Save”
Thats it, now go and help some of your friends get in on the fun.
