Mom & Pop Network

Category: Homelab

  • Running WireGuard over HTTPS via Chisel, or how to bypass your hotel’s firewall

    Right now, I am visiting Amsterdam. While Amsterdam is a nice city and I like my hotel, one of the worst parts is the hotel’s Wi-Fi.

    The Wi-Fi is heavily firewalled. Meaning only mainstream TCP services are allowed. While SSH is allowed, Wireguard isn’t.

    But I discovered Chisel, which lets you proxy TCP or UDP over HTTP and WebSocket.

    So how do you run Chisel?

    Installation

    On a web server, download the latest version of Chisel.

    While most of you are going to run Debian or Ubuntu, I run Rocky Linux as a server so Chisel can be installed with a single dnf command:

    dnf install https://github.com/jpillora/chisel/releases/download/v1.11.8/chisel_1.11.8_linux_amd64.rpm

    You can also install it as a Docker container.

    Running Chisel as a Server

    Assuming you aren’t running Docker or Windows:

    chisel server --auth user:pass --port port --backend http://example.com

    Replace the following:

    • user with your desired username
    • pass with your desired password
    • port with the TCP port you want Chisel to listen on

    You can also set up a systemd service. I am lazy so I used tmux.

    I also recommend a reverse proxy. I use Caddy, so to do it, assuming you have the DNS name domain pointed to your server, add the following server block:

    domain {
  encode gzip
  reverse_proxy 127.0.0.1:port
}

    Make sure port is the Chisel port, and then restart Caddy.

    If you use nginx or another reverse proxy, check the respective instructions.

    Running Chisel as a Client

    I am accessing Chisel on Fedora, so the dnf command also worked there.

    To run Chisel as a client, run the following in your terminal:

    chisel client --auth user:pass https://url local_port:remote_ip:remote_port/udp
    • user with your chosen username
    • pass with your chosen password
    • local_port with the UDP port you want to listen on locally
    • remote_ip with the remote IP you want to connect to
    • remote_port with the UDP port you want to connect to

    You can also remove /udp to connect via TCP.

    Conclusion

    It always sucks when the expensive hotel your family booked blocks most TCP/IP services. But the good thing about open source is that you can tunnel raw UDP and TCP inside HTTPS.

    Yes, China spends unlimited amounts of money on internet censorship. But for-profit hotels, workplaces, public libraries and ISPs who purposefully violate Net Neutrality cannot. And in the latter case, firewalling shouldn’t mean you shouldn’t be able to access your homelab.

  • Running WPA3-Enterprise in my home network

    It’s no secret that by now, most home networks use WPA-Personal. Why? It’s a simple password. This included mine. Well, until today.

    I always wanted to try WPA3-Enterprise in my home network, at least on the SSID/VLAN for my devices (versus the rest of my family). The roadblock was FreeRADIUS not working with Samba and MS-CHAPv2, it was a permissions issue (I don’t want to pirate Windows Server).

    So here’s me connected to my WPA-Enterprise on my phone:

    Why?

    While I know the reasons for WPA3-Enterprise in an office network, it seems overkill for a home network. So why do it?

    Because I can.

    It’s easy enough to do on a Incus UniFi controller. After all, I have a cluster of MikroTik 10G equipment when I could just rent an ISP-provided router and extender like a Joe and focus on “AI” instead.

    Well, I’m done.

  • Cloning a Calix GPON ONT or Quantum Fiber SmartNID

    WARNING: This guide is for instructional purposes only. I am not responsible for any damages caused by this guide. This guide may violate your ISP’s terms of service and is possibly illegal. Proceed at your own risk. I also no longer live in CenturyLink/Quantum territory.

    While I now live in Verizon territory (and lack Verizon FiOS at the moment), when I lived in Seattle and got CenturyLink Fiber (now Quantum Fiber), I initially had an issue which looked like “latency spikes and packet loss”.

    Little did I know that the ONT (Optical Netowrk Terminal) given, a Calix 716GE-I R2 has a hard cap of 16384 TCP sessions, not unlike AT&T Fiber/U-verse routers. Even lesser known is that it’s very easy to clone Calix ONTs with just a JTAG cable and the Calix 803G doesn’t have the issue.

    CenturyLink Fiber then got rebranded as Quantum Fiber which uses SmartNIDs and DHCP, and then sold to AT&T. However, I had a CenturyLink-branded service with a Calix ONT at the time and moved eastwards before I could be switched over.

    Note: This is not for AT&T Fiber itself using a Nokia ONT or BGW320/BGW620. I never had AT&T Fiber, outside of Frontier FiberOptic GPON in Connecticut for a few months which inherited 802.1X from AT&T. Check the 8311 Discord server for AT&T Fiber proper.

    So you want to clone your ONT?

    Pre-requisites

    You need the following devices information:

    • Original, ISP-provided Calix ONT or SmartNID with the ONU MAC and FSAN Serial ready
    • If required, your GPON Password. CenturyLink/Quantum doesn’t use this
    • Calix ONT being cloned to
    • JTAG serial adapter

    About the JTAG adapter, the CP2102 models are recommended. Other models have had issues with Calix’s JTAG console.

    Cloning the ONT

    Now for the fun part.

    First off, you need to open your ONT.

    For the 803G (the destination ONT), these images should be self-explanatory.

    When you are inside, connect the JTAG.

    Note: The TX should go to your JTAG RX and vice-versa.

    Image source: fire-foo.com.

    When you connected your JTAG header pins, open a serial console (e.g. cu or PuTTY), plug in the power while hitting ESC on your keyboard repeatedly.

    When you are on the CFE> prompt, type b and press enter.

    Press enter for most of the values, except the Base MAC Address and GPON Serial Number which should be the ONU MAC and FSAN Serial respectively from your source ONT.

    Some ISPs may need a GPON password, but CenturyLink/Quantum does not. I do now know how to get the GPON password as I was never in that scenario.

    Once you are done, wait for the ONT to boot successfully, then turn off the power, close the ONT, and plug the fiber and power in place of your stock Calix ONT.

    An example of a JTAG session can be seen here.

  • Rocky Linux/RHEL 10: Fixing “Invalid UID in persistent keyring name” with AD and SSSD

    I run a Samba Active Directory in my homelab, with a Wireguard VPN to my dad’s house between my and his MikroTik routers.

    I recently reinstated the HPE ProLiant ML30 Gen9 running Rocky Linux 10 colocated at his house. With that, I rejoined the server to a new AD domain I made. I wasn’t able to log in, since the SSSD cache doesn’t get flushed.

    While I used this guide on Rocky Linux, it should be the same on AlmaLinux, CentOS or RHEL.

    Going back, the error I got was:

    Feb 13 15:11:01 oldsai.sc.lan krb5_child[2258]: Invalid UID in persistent keyring name
    Feb 13 15:11:01 oldsai.sc.lan sshd-session[2254]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=::1 user=blackbird
    Feb 13 15:11:01 oldsai.sc.lan sshd-session[2254]: pam_sss(sshd:auth): received for user blackbird: 4 (System error)

    To fix this, first stop sssd:

    systemctl stop sssd

    Clear the cache with sss_cache:

    sss_cache -E

    Now remove the stray cache files:

    /var/lib/sss/db/*

    Note: this command is important, as SSSD doesn’t flush caches upon unjoining and rejoining, even with different user IDs.

    Now start sssd:

    systemctl start sssd

    The error should go away. Keep in mind that if UIDs changed for a particular user, you will need to delete or chown their home directory.

    Source. Thanks, Jarrod Farncomb.

  • Creating a Samba Active Directory Domain Controller on FreeBSD

    While I now use Fedora as my main desktop and Rocky Linux as my server OS, there are some things which aren’t in the EPEL. That combined with me not having really used Debian since high school means I set up homelab Samba domain controllers on FreeBSD.

    To set one up, you need a static IPv4 address, and a static IPv6 address if your network is dual-stack. You’ll also need to forward your domain’s DNS zone or set the DC as the DNS server, which is out of the scope for this article.

    When you’re ready, if your DC uses UFS (versus ZFS), you’ll first need to edit /etc/fstab:

    /dev/vtbd0s1a / ufs rw,acls 1 1

    Note, you need the ,acls in order to run a Samba DC, as Samba requires this.

    If you haven’t rebooted, run this:

    mount -a

    Now, install Samba:

    pkg install samba422

    Note: newer versions of Samba may have come out. At the time of posting it’s samba422.

    Next, create the domain:

    samba-tool domain provision --use-rfc2307 --realm=SC.LAN --domain=SC --server-role=dc --dns-backend=SAMBA_INTERNAL --adminpass=PASSWORD

    Replace SC.LAN with the DC’s realm, and SC with the domain’s NetBIOS name.

    Then, enable samba_server and winbindd:

    sysrc samba_server_enable=YES
sysrc winbindd_enable=YES

    Now, enable Samba:

    service samba_server start

    Keep in mind you’ll need your DNS server set to the AD DC’s static IP, or forward DNS zones. I use a MikroTIk core router, and forward DNS there.

    If your DNS server or forwarding is set, check if you can resolve it:

    # host sc.lan
sc.lan has address 172.20.0.6
sc.lan has IPv6 address 2602:XXX:2::6
#

    Now, you can add users and groups, and join client machines.

  • Implementing Carrier Grade NAT and Port Block Allocation on FreeBSD and PF

    While I don’t run an ISP (unless you consider my hosting company Fourplex an “ISP”), one project I’ve wanted to try in my homelab is implementing Carrier Grade NAT with Port Block Allocation.

    Yes, we all know Carrier Grade NAT sucks. It makes it hard to host services and use console gaming and such. And yes, I haven’t daily driven FreeBSD in years. But FreeBSD is still a good network appliance even if it’s no longer my desktop.

    Note: I won’t describe other parts of the network, like rc.conf IP assignments, IPv6 and routing protocols.

    Ranting aside, first off you need this in /etc/rc.conf:

    gateway_enable="YES"
pf_enable="YES"

    Subsequently, you need something like this in /etc/pf.conf:

    nat on $ext_if from 100.64.0.0 to any -> 1.2.3.4 port 1000:2999
nat on $ext_if from 100.64.0.1 to any -> 1.2.3.4 port 3000:5999

    What do these lines mean? I’ll explain:

    • $ext_if is the external interface, which has the public IPs CGNAT will use.
    • 100.64.0.X is the CGNAT IPv4 which will be allocated to a client.
    • 1.2.3.4 is the public IPv4 used by CGNAT.
    • After the port is the start and end IPv4 port ranges respectively, separated by :. For instance, 1000:2999 will assign a start port of 1000 and end port of 2999 to a client.

    You will need a nat line for each CGNAT customer, and can use multiple public IPv4s for different clients.

    What about automating generation?

    I should do it eventually. But since I’m not running a broadband ISP (only a VPS/VPN host) it’s not a priority to script it.

    Yes, I’d love to be an ISP. But only if the FCC would actually mandate “line sharing” rules neither party wants. Or NYC builds an “open access” fiber network which they won’t. I won’t do a WISP for various reasons.

    You could also use VyOS for CGNAT instead of FreeBSD which has CGNAT syntax.

  • MikroTik: Add DNS Forwarding Entry for Active Directory DNS

    Also, why didn’t anyone tell me about match-subdomain=yes?

    For almost two years, my homelab’s core router has been a MikroTik, namely a CCR2004-16G-2S+ and then a CCR2004-16G-2S+PC. I also run a FreeBSD Samba Active Directory domain controller.

    The common ‘solution’ is to use domain wildcard regex and that’s usually not pretty.

    Then I learned about the match-subdomain=yes and it worked wonders, that combined with ttl=0s.

    Keep in mind that by using this guide, I am assuming you’re using the DNS server on a MikroTik router, versus an external resolver.

    If you’re using MikroTik’s resolver, add the following configuration:

    /ip dns static
    add forward-to=IP match-subdomain=yes name=DOMAIN ttl=0s type=FWD

    Replace IP with your DC’s IP address, and DOMAIN with the domain name used by AD.

    Note: you need ttl=0s, otherwise it gets out of sync. match-subdomain=yes is a shortcut Google never told us about (even without “AI” responses), but is ultra useful here.

    If you wish to forward reverse DNS entries, you can do:

    /ip dns static
    add forward-to=IP match-subdomain=yes name=xx.in-addr.arpa ttl=0s type=FWD
    add forward-to=IP match-subdomain=yes name=yy.ip6.arpa ttl=0s type=FWD

    Replace IP with the IP address of the DNS server, and xx (IPv4) and yy (IPv6) with the Reverse DNS zones used by your network.

    This should give you AD DNS using MikroTik’s DNS server.

  • My Homelab and Home Network, January 2026 Edition

    I’ve been running a homelab since October 2013 (I’m good at dates). Back then, I used a special VPN and a homebuilt Pentium 4 desktop.

    Fast forward twelve and a half years, my homelab has significantly changed.

    So what’s in it? I’ll tell you.

    Network

    Right now, I use MikroTik for routing and switching, and UniFi for access points.

    On the top is the core router: a CCR2004-16G-2S+PC which handles L2TP termination, Wireguard, NAT and firewalling.

    The middle is the core switch: a CRS309-1G-8S+IN. This connects to my SFP+ servers via DAC, and 10GbE copper devices via SFP+ modules.

    The bottom is the PoE switch: a CSS610-8P-2S+IN. This connects to four UniFi U6 Pros (indoors) and one U6 Mesh (outdoors)

    The internet connection is not really a traditional ISP like Spectrum or Verizon, but instead a Calyx Sprout SIM. Sprout is an unlocked and unlimited T-Mobile SIM card. I combine that with a L2TP VPN to a BGP VPS to give myself a public IPv4 and IPv6 block.

    The 5G modem is a Hitron D60, of which I disabled Wi-Fi.

    Since I live in NYC and not a RV, I can get Spectrum. But not FiOS or any other fiber ISP, and Spectrum is dragging their feet on high-split. I use Calyx for 75 Mbps upload speeds versus 35.

    Server

    I use two Minisforum MS-01 Mini PCs. Both have i9-13900H CPUs and 96GB RAM, and run Rocky Linux 10.

    The bottom one, “twin” has 2x4TB Crucial SSDs, and the top one, “triplet” has 2x2TB FanXiang SSDs. Both are in RAID 1.

    Twin runs:

    • MariaDB master-master node (Rocky Linux 10, Incus)
    • Nextcloud (Rocky Linux 10, Incus)
    • UniFi Controller (Debian 12, Incus)

    Triplet runs:

    • MariaDB master-master node (Rocky Linux 10, Incus)
    • FreeBSD Ports development (FreeBSD 15, KVM)
    • EVE-NG (Ubuntu 22.04, KVM)

    Sometimes, I also run a Windows 2000 or 2003 VM for the kicks. They usually get deleted in a few days.

    Future Plans

    My last project was to remove noisy components and replace them with “passive” equivalents where possible. However, future plans depend on:

    • Spectrum’s DOCSIS 4.0 upgrades (which keep getting delayed)
    • Whether or not fiber comes (rights of way is hard in NYC)
    • The RAM shortage

    The former two points might mean an upgrade to Wi-Fi 7 (or 8) APs and a UniFi PoE switch instead of the CSS610-8P-2S+IN. MikroTik doesn’t make a multi-Gig PoE switch, so if I did it today, I’d need UniFi.

    The RAM shortage is plain stupid: the RAM industry is putting all their eggs in AI, even when AI is just froth. I just wish someone could fund a DRAM antitrust lawsuit. Heck, I’d love for my startup Fourplex to participate if someone’s willing to fund me (I won’t sue myself).