From Chaos to Control: Rebuilding My Infrastructure

2026-03-13T00:00:00+00:00

[0x00] Introduction

I’ve always believed that everyone deserves a second chance, especially when things were poorly designed from the start and inevitably went sideways later on.

So I’m finally back. It has been a few months since I took down my original blog site, blog.rayzhang.top^[1], along with several other services. I made the decision to rebuild everything around Nix/NixOS. It was a rough ride, but meaningful and necessary in the end.

TL;DR

This site is built with Zola, with the source available here.

All server configurations are defined using NixOS, with the repository available here.

Secrets and tokens are managed through sops-nix, which builds on sops. The related configuration can be found here.

[0x01] The Early Chaos

When I first started deploying my services, I had just enough Linux experience to be dangerous. Things worked—but mostly by accident.

Over the past five years, my setup went through several phases:

Stage I. Everything on One Box

A single CentOS VPS, binaries pulled from GitHub, and handwritten systemd units. Certificates were renewed manually. The blog was built locally and uploaded over SSH.

It worked—but only because I didn’t know better.
Stage II. Arch BTW

After a few painful VPS migrations, I switched to Arch for its package ecosystem. It was flexible, powerful, and surprisingly usable in production—at least for a “toy project.”

This was also when I moved to ACME for automated certificates.
Stage III. Functionality First

With more resources, I started treating my setup like a real system.

I spread services across multiple VPS providers and built a private network using Tailscale (with Headscale). Services became modular—gateways, backends, databases—all running on different nodes.

It looked more “professional,” but complexity started to grow faster than I could manage.
Stage IV. Hitting the Wall

Eventually, the cracks became obvious.

Even with better tooling, I still couldn’t reliably migrate or rebuild my infrastructure. Too many moving parts—configs, data, and implicit state scattered everywhere.

Dotfiles weren’t enough. Backups weren’t enough.

That’s when I started seriously looking at Nix.

[0x02] Why Nix Now?

I didn’t go looking for Nix. I got cornered into it.

By the time I reached that point, the problem was no longer about package managers or distributions. It was about control. I couldn’t reliably rebuild my own infrastructure.

Every machine had its own state. Services depended on configurations that lived half in Git, half on disk, and half in my memory. Migrations were slow, fragile, and stressful. Even when I thought I had everything backed up, there was always something missing.

What I wanted was simple:

A way to define the entire system in one place
A way to reproduce it on any machine
A way to make changes without breaking everything

Everything I had tried before only solved part of the problem.

Nix was the first thing that addressed it as a whole.

Instead of treating packages, services, and system configuration as separate layers, Nix describes them together—using the same language, in the same place. The system becomes something you declare, rather than something you gradually mutate over time. That shift changes how you think about infrastructure.

Rebuilding a machine is no longer a special operation—it’s just applying the same configuration somewhere else. Switching between stable and bleeding-edge components is no longer a risky upgrade—it’s just choosing a different input.

Of course, this comes with a cost. Nix introduces its own complexity, and understanding it takes time. At a glance, it can feel like just another abstraction layer—something we’ve all learned to be cautious about. But this time, the trade-off made sense.

For the first time, I felt like I could describe my system completely, version it, and rebuild it when needed. Not partially. Not approximately. But consistently. That was enough reason to start over.

[0x03] Design Philosophy

I had accumulated many ideas long before I decided to nuke everything and start from scratch. Without clear design goals, the system would inevitably drift into something messy and difficult to maintain.

So before rebuilding anything, it was necessary to step back and rethink what I actually wanted this infrastructure to achieve.

Network Architecture

In China, ISPs are usually very strict about inbound traffic. In many cases, residential networks do not allow users to open ports for their own services. Even if you manage to obtain a dedicated public IP and port, the availability of that service is not always guaranteed.

This creates a practical challenge for self-hosting. Many of my services require significant CPU, memory, and storage, which makes them better suited to run on dedicated hardware at home. However, exposing those services directly to the public internet is often difficult under these network restrictions.

As a result, many people rely on VPS servers as the public entry point for their infrastructure. But VPS instances are typically limited in resources unless you are willing to pay a significant amount of money.

To address this problem, I decided to build my infrastructure around a VPN-based network and expose services through a layered gateway design.

The basic rules of this architecture are:

As many devices as possible are included in my Tailscale VPN network.
VPS servers act as L1 gateways using Nginx. They receive traffic from the public internet and forward it to internal nodes through reverse proxy.
L2 gateways, also powered by Nginx, run on the actual service hosts (usually dedicated hardware at home) and route traffic to the corresponding services.
Inside the Tailscale network, any authenticated client can access services through the L2 gateways. This allows both public and private services to be reachable within the VPN.
From the public internet, access is strictly controlled at the L1 gateway. Only explicitly configured domains for public services are allowed to pass through.


flowchart LR
%% Internet
USER[Internet Users]

%% ================= VPN =================
subgraph TS["Tailscale VPN Network"]
    direction LR

    %% Clients
    subgraph CLIENTS["Clients"]
        LAPTOP[Laptop]
        PHONE[Phone]
    end

    %% L1 Gateways
    subgraph VPS["VPS Edge Nodes"]
        L1A["L1 Gateway
Nginx Reverse Proxy"]
        L1B["L1 Gateway
Nginx Reverse Proxy"]
    end

    %% ================= Service Nodes =================
    subgraph SERVICES["Home Service Nodes"]
        subgraph RPI["Raspberry Pi"]
            L2R["L2 Gateway
Nginx"]
            subgraph PUBLIC_RPI["🌐 Public Internet-Exposed"]
                BLOG["Static Blog"]
            end
        end

        subgraph ROCK["ROCK5B+"]
            L2K["L2 Gateway
Nginx"]
            subgraph PUBLIC_ROCK["🌐 Public Internet-Exposed"]
                FORGEJO["Forgejo"]
                VAULT["Vaultwarden"]
                S3["Garage S3"]
            end
        end

        subgraph MINI["Mini PC"]
            L2M["L2 Gateway
Nginx"]
            subgraph PRIVATE_MINI["🔒 Private VPN Only"]
                WIKI["Internal Wiki"]
                HOME["Smart Home"]
            end
        end

        %% -------- Data Layer inside Home --------
        subgraph DATA["🗄️ Data Layer Internal"]
            PG[("PostgreSQL")]
        end
    end
end

%% ================= FLOWS =================
USER-->|HTTPS|L1A
USER-->|HTTPS|L1B

L1A-->|VPN|L2R
L1A-->|VPN|L2K
L1B-->|VPN|L2K

LAPTOP-->|VPN|L2M
PHONE-->|VPN|L2M
LAPTOP-->|VPN|L2K

L2R-->BLOG
L2K-->FORGEJO
L2K-->VAULT
L2K-->S3
L2M-->WIKI
L2M-->HOME

FORGEJO-->|VPN|PG
VAULT-->|VPN|PG
WIKI-->|VPN|PG

%% ================= STYLES =================
classDef l1 fill:#ff9f43,stroke:#ffffff,stroke-width:2px,color:#000
classDef l2 fill:#feca57,stroke:#ffffff,stroke-width:2px,color:#000
classDef public fill:#2ecc71,stroke:#ffffff,stroke-width:2px,color:#000
classDef private fill:#3498db,stroke:#ffffff,stroke-width:2px,color:#fff
classDef data fill:#9b59b6,stroke:#ffffff,stroke-width:2px,color:#fff
classDef internet fill:#e74c3c,stroke:#ffffff,stroke-width:2px,color:#fff

class L1A,L1B l1
class L2R,L2K,L2M l2
class BLOG,FORGEJO,VAULT,S3 public
class WIKI,HOME private
class PG data
class USER internet

style PUBLIC_RPI fill:transparent,stroke:#2ecc71,stroke-width:2px,stroke-dasharray: 5 5
style PUBLIC_ROCK fill:transparent,stroke:#2ecc71,stroke-width:2px,stroke-dasharray: 5 5
style PRIVATE_MINI fill:transparent,stroke:#3498db,stroke-width:2px,stroke-dasharray: 5 5

Domain Naming Strategy

I bought my first domain, rayzhang.top, back in 2021. It was enough when everything lived in one place. That changed once I started splitting services across different environments.

Some services were meant to be public. Others were strictly internal, accessible only through my Tailscale network. Mixing them under the same domain quickly became confusing—not just in naming, but also in access control and routing.

There was also a more practical concern. Since most of my infrastructure is deployed outside China, the legal status of certain deployments isn’t always clear. If I ever want to host services within mainland China legally, I would need a separate domain with ICP registration.

At that point, I stopped treating domain names as simple labels and started treating them as boundaries—between public and private, internal and external, global and mainland.

To make this explicit, I defined a set of naming conventions:

Domain Rules	DNS	Purpose	Notes	Examples
`node-${location}.rayzhang.top`	Public	Resolve a server by its location.	Not intended for direct service access. Typically implemented as CNAME records pointing to actual hosts.	`node-lax.rayzhang.top` `node-hel.rayzhang.top` `node-lhr.rayzhang.top` `node-ngb.rayzhang.top` `node-intl.rayzhang.top`
`${deviceName}.netscale.cc`	Tailscale	Resolve a machine by its device name.	Part of Tailscale MagicDNS.	`mjolnir.netscale.cc` `brokk.netscale.cc` `heimdall.netscale.cc` `odin.netscale.cc` `forseti.netscale.cc`
`${serviceName}.netscale.cc`	Tailscale	Resolve private services by name.	Accessible only within the VPN (L2 gateway).	`nexus.netscale.cc`
`${serviceName}.rayzhang.top`	Public	Resolve public services by name.	Exposed through L1 gateway.	`flux.rayzhang.top` `keystone.rayzhang.top` `journal.rayzhang.top`
`${serviceName}.netscale.work`	Public	Public services hosted within mainland China.	Requires ICP registration. Used globally for consistency when a service is defined under this domain.	`derper-bifrost.netscale.work` `derper-heimdall.netscale.work` `derper-forseti.netscale.work`

[0x04] Implementation: Making It Real

At this point, the architecture and naming conventions were clear. The remaining question was how to actually implement them in a way that wouldn’t fall apart again in six months.

This is where Nix started to matter—not as a tool I wanted to use, but as a tool that could enforce the structure I had already designed.

System as a Single Source of Truth

In the past, my infrastructure existed in fragments. Some configurations were in Git. Others lived directly on servers. A few only existed in my memory. Rebuilding a machine meant retracing steps, hoping nothing was forgotten.

With Nix, I started treating the entire system as a single, versioned definition.

Every machine is declared.
Every service is defined.
Every dependency is pinned.

Adding a new node is no longer a manual process. It’s just another entry in the configuration. Rebuilding a server doesn’t feel like recovery anymore—it feels like routine.

At the same time, I started organizing my machines more intentionally. Each device has its own configuration file, and I name them using Norse mythology—partly for fun but also to make them easier to recognize at a glance. For example:

Raspberry Pi 3B+ BROKK
Rock 5B+ MJOLNIR
DMIT LAX VPS HEIMDALL

Each of these configurations is not written from scratch. Instead, they are composed of reusable modules. Services are defined once, as independent building blocks, and then imported where needed—almost like assembling LEGO pieces.

A typical configuration looks like this:

{ self, lib, system, sops, prism, deploy-rs, journal, ... }:
let
  deviceName = "brokk";
  modules = lib.scanPath
    {
      path = ./.;
      recursive = true;
    } ++ map lib.relativeToRoot [
    "modules/nixos/base"
    "modules/nixos/server/shairport-sync.nix"
    "modules/nixos/network/wifi.nix"
    "modules/nixos/server/acme-pusher.nix"
    "modules/nixos/server/zola-blog.nix"
  ];
in
{
  nixosConfigurations.${deviceName} = lib.nixosSystem {
    inherit system modules;
    specialArgs = { inherit lib sops prism journal deviceName; };
  };
  # .....
}

This approach keeps the system consistent while still allowing each machine to have its own role.

Instead of configuring servers individually, I’m assembling them from shared components. That makes the overall structure easier to understand—and much easier to change.

Even more, each node in my infrastructure is just a Nix configuration. Adding a new server is no longer “setting up a machine”—it’s adding a file.

Secrets Management

One thing that always broke the illusion of a “fully declarative system” was secrets.

You can define packages. You can define services. You can even define entire machines. But you can’t just commit passwords, API tokens, or private keys into your repository.

In my earlier setups, secrets were always handled separately—and usually manually. I would copy them over, store them in random files, or rely on memory. It worked, but it also meant the system was never truly reproducible. There was always a missing piece.

With Nix, I wanted to avoid that gap. The system should be fully defined but still safe to store in a public or shared repository. To achieve this, I integrated encrypted secrets using sops-nix. Secrets are stored alongside the configuration but encrypted in a separate repository Prism. Only the target machines can decrypt them at deployment time using their own keys.

This setup also allows me to handle more than just simple credentials. For example, I use it to distribute my acme.sh certificates.

Certificates are encrypted and pushed to the repository via an automatic cert sync bot service. The encrypted files (ending in .enc) can be found here.

With sops-nix, decrypting and using secrets becomes part of the system configuration itself. For example, my Vaultwarden config looks like this:

{ prism, config, deviceName, ... }:

let
  domain = "keystone.rayzhang.top";
in
{
  # .....
  sops.secrets."certs/${domain}/fullchain.pem" = {
    sopsFile = "${prism}/certs/rayzhang.top/fullchain.pem.enc";
    owner = "nginx";
    group = "nginx";
    mode = "0640";
    format = "binary";
    restartUnits = [ "nginx.service" ];
  };

  sops.secrets."certs/${domain}/key.pem" = {
    sopsFile = "${prism}/certs/rayzhang.top/key.pem.enc";
    owner = "nginx";
    group = "nginx";
    mode = "0640";
    format = "binary";
    restartUnits = [ "nginx.service" ];
  };
  
  #.....
  services.nginx.virtualHosts.${domain} = {
    forceSSL = true;
    sslCertificate = config.sops.secrets."certs/${domain}/fullchain.pem".path;
    sslCertificateKey = config.sops.secrets."certs/${domain}/key.pem".path;
    quic = true;
    http3 = true;
    kTLS = true;
    # .....
  };

}

This changes the workflow in a subtle but important way. Secrets are no longer external artifacts that need to be managed separately. They become part of the system definition—just in a protected form.

From the outside, nothing changes. Services still receive the same environment variables. Applications still read the same configuration files. But internally, everything is now consistent.

[0x05] Trade-offs and Reality

This approach is not free. Nix solves many of the problems I had before—but it also introduces a different kind of complexity.

The learning curve is real. Understanding how everything fits together takes time, and sometimes the error messages don’t make things easier. When something breaks, debugging can feel like peeling layers of abstraction, trying to figure out what actually went wrong underneath.

One area where this becomes especially noticeable is state management. Applications are not purely declarative. They generate state at runtime—databases, cache files, user data, and various artifacts in the home directory. These files are not part of the system definition, yet they shape how the system behaves at runtime.

That makes them a weak point—an upgrade or rollback can easily introduce subtle inconsistencies, especially when the application’s expectations no longer align with the existing state.

This creates a gap. You can fully define how a service is deployed, but not always how its data evolves over time. NixOS tries to address this with mechanisms like stateVersion.

At a high level, stateVersion acts as a reference point for how certain system components should behave, especially when defaults or data formats change over time. It helps maintain compatibility across upgrades.

But it also reveals something important: even in NixOS, the system cannot fully escape its past. Compatibility has to be preserved somewhere. In practice, this means there is always a boundary between: what Nix can define and what must be managed as a state. And that boundary doesn’t disappear just because the system is declarative.

I’ve found that trying to force everything into a stateless model doesn’t always work. Some parts of the system—databases, user data, application state—are inherently persistent. They need to be treated differently, with backups, migrations, and careful handling during upgrades.

Nix doesn’t eliminate that responsibility. It just makes the boundary more visible. So while the system feels declarative, it’s not entirely stateless. And that’s okay.

References

blog.rayzhang.top was my first attempt to build a static blog using Hexo Butterfly. It was written in zh-Hans. ↩

Ray's Journal

From Chaos to Control: Rebuilding My Infrastructure