Syncing Application Data Between Clusters using Cloudflared and VolSync

Introduction

The achilles heel of all homelab setups is the residential internet connections we rely on. These constitute a single point-of-failure that we have no control of, and outages can be extremely disruptive events. Since I host some services that are pretty critical to my life, I set out to try and find a way to have redundancy cross-clusters. Of course, for stateful applications, this requires data to also be synchronized. For this, I settled on VolSync + Cloudflared to send data back and forth between clusters.

Prerequisites

1. More than one cluster - I have a homelab cluster and a cluster deployed on Oracle Cloud using their always-free resources
2. Using Cloudflare for DNS

Cloudflared Setup

Cloudflared isn't actually necessary for VolSync to function. However, it does have some great benefits, including the hiding our of internal IP. Additionally, we don't have to open any new ports in our router. It does add some extra overhead, but I think it's well worth it for the benefits it offers. Note that you only need one receiving tunnel per cluster, whereas as you'll see later in the guide you'll need one access tunnel per application you'd like to expose on each cluster.

Cloudflare has a good guide on getting things set up on Kubernetes here so I'm just going to highlight what I think is most important, the ConfigMap which handles where traffic from Cloudflare is routed to:

# Name of the tunnel you want to run
tunnel: example-tunnel
credentials-file: /etc/cloudflared/creds/credentials.json
# Serves the metrics server under /metrics and the readiness server under /ready
metrics: 0.0.0.0:2000
no-autoupdate: true
# The `ingress` block tells cloudflared which local service to route incoming
# requests to. For more about ingress rules, see
# https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ingress
#
# Remember, these rules route traffic from cloudflared to a local service. To route traffic
# from the internet to cloudflared, run `cloudflared tunnel route dns <tunnel> <hostname>`.
# E.g. `cloudflared tunnel route dns example-tunnel tunnel.example.com`.
ingress:
  # Sends traffic to volsync pod in security namespace
  - hostname: vs-vaultwarden-prod.${SECRET_DOMAIN}
    service: tcp://volsync-sync-vaultwarden-data.security.svc.cluster.local:22000
  - hostname: testcf.${SECRET_DOMAIN}
    service: http://cloudflared-test-nginx:80
  # This rule sends traffic to the built-in hello-world HTTP server. This can help debug connectivity
  # issues. If hello.example.com resolves and tunnel.example.com does not, then the problem is
  # in the connection from cloudflared to your local service, not from the internet to cloudflared.
  - hostname: hellocf.${SECRET_DOMAIN}
    service: hello_world
  # This rule matches any traffic which didn't match a previous rule, and responds with HTTP 404.
  - service: http_status:404

The key to this ConfigMap is the ingress: block, which routes traffic from the tunnel to various services in the cluster. For each application you want to sync, you'll need an entry here as well a corresponding DNS entry in Cloudflare.

VolSync Setup

To deploy the VolSync operator, I used the provided Helm Chart.

With the operator deployed, all we need to do now is create a ReplicationSource pointing at the PVC we'd like to synchronize on each cluster. First, create an empty ReplicationSource as such:

apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: test-volsync-replicationsource
spec:
  sourcePVC: pvc-to-replicate
  syncthing:
    serviceType: ClusterIP

You'll notice that we don't expose it externally via LoadBalancer. This is because we'll use our Cloudflared access tunnel to connect to other ReplicationSources.

You'll notice a new pod is spun up by the operator in the same namespace as the PVC you'd like to replicate. For example, here's the pod created by my replication of my Vaultwarden instance:

Putting it all Together

With VolSync and Cloudflared deployed, we just need to set up a way for data to be sent from one cluster to another. Once in the other cluster, the data will be routed to the VolSync pod for the application that we created earlier via cloudflared.

To do this, we will create cloudflare access tunnels that our VolSync pods can use to communicate with their peers outside of the cluster. Essentially, we need a pod running alongside our app that executes cloudflared access tcp --hostname <other-cluster-url>.site.com --url 0.0.0.0:9210. Then, VolSync can communicate with this internal pod, which sends the data externally. Here's how I've deployed it on my home cluster:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: cloudflared-access-tunnel
spec:
  values:
    image:
      repository: cloudflare/cloudflared
      tag: "2023.3.1"
    args:
      - access
      - tcp
      - --hostname
      - ${PHX_APPLICATION_VS_URL}.${SECRET_DOMAIN}
      - --url
      - 0.0.0.0:9210
    service:
      main:
        ports:
          http:
            enabled: false
            primary: false
          tcp:
            port: 9210
            protocol: TCP
            primary: true
    probes:
      liveness:
        enabled: false
      readiness:
        enabled: false
      startup:
        enabled: false
    persistence:
      tunnel-credentials:
        enabled: true
        type: secret
        name: cloudflared-tunnel-credentials
        mountPath: /etc/cloudflared/creds
        readOnly: true

Once we do this for both clusters, we can point the ReplicationSource on each cluster at each other by editting its peers as such:

apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: test-volsync-replicationsource
spec:
  sourcePVC: pvc-to-replicate
  syncthing:
    serviceType: ClusterIP
    peers:
      - ID: ...
        address: tcp://cloudflared-access-tunnel:9210
        introducer: false

And with that, we're pretty much done! Syncthing has a handy UI you can expose if you'd like to see synchronization status visually. You can do this by exposing the volsync-sync-*-api Service and using the credentials in the Secret volsync-sync-*.

For more in depth code, you can check out my public homelab repository. In it, I use other tools like ExternalDNS to handle the creation of Cloudflare DNS records that are out of scope for this article.

Thanks for reading!