Elixir Clusters on Railway

09 Feb, 2025

Not interested in the story? Jump to the code/config.

For a while now, I've been wanting to setup an elixir cluster, fantasizing about calling a function on a remote node as if it were local, starting a background task that distributes work across the cluster. I want to see the nodes connect. I want a glimpse of this power.

Recently, I found out about Railway and started messing around in their environment. I was shocked at how easy it was to set things up. I ended up deploying an elixir cluster and had a blast experimenting with it.

This wasn't completely pain-free of course. So join me while I walk through setting up the cluster and discuss some learning points along the way. 🛤️

Elixir Shells

At this time, railway doesn't support generic shells into the stuff you deploy (unless it's database specific). Since I wanted some way to test that things were working, I cooked up a basic UI that renders an interactive elixir "shell" for each elixir node in my railway environment.

elixir-shells

In case you were wondering, the background color is cornflower-blue.

Each of these "shells", allow you to execute any valid iex command on the node (or even call functions defined in the application's code). The commands entered in shells of remote nodes are sent to the node via http requests, executed on the node, and the output is sent back in the request's response in order to display in the shell rendered on your client. For the primary node, what you might consider the control-plane, commands are executed directly. In all cases, the command you enter eventually makes its way to the node and is executed with Code.eval_string/1. Step aside security.

Cluster Auto-Discovery

It'd be nice if the nodes in our cluster connected automatically. After a bit of research libcluster looks promising in this regard. Since railway generates internal dns names for the services you create, we'll use the DNSPoll strategy.

In my config.exs the libcluster config looks like:

config :libcluster,
  topologies: [
    convoy_topology: [
      strategy: Cluster.Strategy.DNSPoll,
      config: [
        polling_interval: 5_000,
        query: "convoy.railway.internal",
        node_basename: "convoy"
      ]
    ]
  ]

For the control plane, the node's name will be convoy@convoy.railway.internal and for remote nodes it will be some flavor of: convoy@convoy-jfb7.railway.internal. Each node needs some environment variables¹ specifying how it should self-identify.

RELEASE_DISTRIBUTION=name
# convoy@convoy.railway.internal, convoy@convoy-jfb7.railway.internal, etc.
RELEASE_NODE=convoy@<node-name>.railway.internal
RELEASE_COOKIE="super-secret-cookie"
# specifying the port is critical, especially on railway
PORT=4000
# these are standard, not special for this situation
PHX_HOST=...
SECRET_KEY_BASE=".."

In our railway environment, we'll be able to dynamically add elixir nodes. When they spin up, libcluster will automatically attempt to connect the node to the control plane. libcluster, with the DNSPoll strategy, uses dns queries and attempts to connect to whatever those queries resolve to.

Attempt 0

I deployed the app on railway and was able to spin up a few nodes. Hopeful it just might work first try, I checked if the nodes were connected. They were not...

iex> Node.list
[]

Looks like libcluster's not connecting the nodes, in the logs of all of the nodes I found:

3:50:24.158 [warning] [libcluster:convoy_topology] unable to connect to :"convoy@fd12:98a1:4fac:0:2000:15:da65:1988"

That name sure doesn't look right, but let's deal with libcluster later, and just try to connect these nodes manually.

Does the node self-identify correctly?

# on the control plane
iex> Node.self
:"convoy@convoy.railway.internal"

# on a remote node
iex> Node.self
:"convoy@convoy-jfb7.railway.internal"

Looks correct. Let's try to connect.

# on the control plane
iex> Node.connect(:"convoy@convoy-jfb7.railway.internal")
false

Strange. Let's verify the cookie.

# on the control plane
iex> Node.get_cookie
:"super-secret-cookie"

# on a remote node
iex> Node.get_cookie
:"super-secret-cookie"

The cookies match, as expected. Let's try a dns lookup on the control-plane's generated dns name.

# on a remote node
iex> :inet.gethostbyname('convoy.railway.internal', :inet6)

{:ok, {:hostent, ~c"convoy.railway.internal", [], :inet6, 16, [{64786, 39073, 20396, 0, 8192, 9, 16560, 40612}]}}

Nice. The internal dns properly resolves to a host, our control-plane, and provides an ipv6 address. Does a generic tcp connection work?

# on a remote node
iex> :gen_tcp.connect({64786, 39073, 20396, 0, 8192, 11, 41282, 21287}, 80, [:inet6])
{:error, :econnrefused}

Perhaps it's not allowing traffic on port 80, let's try port 4000.

# on a remote node
iex> :gen_tcp.connect({64786, 39073, 20396, 0, 8192, 11, 41282, 21287}, 4000, [:inet6])
{:ok, #Port<0.124>}

Oh yeah. Let's verify that epmd is running.

# on the control plane
iex> :os.cmd(~c"epmd -names")
~c"epmd: up and running on port 4369 with data:\nname convoy at port 42041\n"

# on a remote node
iex> :os.cmd(~c"epmd -names")
~c"epmd: up and running on port 4369 with data:\nname convoy at port 43515\n"

And can it receive traffic on that port?

# on remote node
iex> :gen_tcp.connect({64786, 39073, 20396, 0, 8192, 11, 41282, 21287}, 4369, [:inet6])
{:ok, #Port<0.178>}

What about the port for inter-node comms?

# on remote node
iex> :gen_tcp.connect({64786, 39073, 20396, 0, 8192, 4, 6082, 5148}, 42041, [:inet6])
{:error, :econnrefused}

Bingo.

The command :os.cmd(~c"epmd -names") gives us two bits of information, the port where the Erlang Port Mapper Daemon (EPMD) is running 4369 and the node's distribution port 42041.

EPMD acts as a name server for erlang nodes, allowing nodes to find each other on a network. The distribution port handles messages between nodes. If a node wants to connect to another node it uses this port. Also if one node wants to run a function on another node (using :rpc for example) that would also happen on this port.

Under the hood, the erlang distribution protocol is using a tcp connection². So if we can figure out how to make a generic tcp connection happen on the distribution port, we'll really be cookin.

Perhaps there's an issue with the high number random port (42041) or maybe railway doesn't allow traffic on those ranges or something. I dug around and found that we can specify the distribution port instead of having it randomly assigned -- by providing some extra params to erl the program that starts an Erlang runtime system³.

Appending to my current environment variables, I now have:

ERL_AFLAGS="-proto_dist inet6_tcp -kernel inet_dist_listen_min 4001 inet_dist_listen_max 4001"

-proto_dist inet6_tcp - forces erlang's distribution protocol to use ipv6, railway's internal private network only supports ipv6.

-kernel inet_dist_listen_min 4001 inet_dist_listen_max 4001 - ensures that the distribution port is set to 4001 which prevents it from being assigned to some random high-numbered port.

Attempt 1

I deployed my environment variable changes, thus restarting the nodes. Checking the logs reveals that libcluster is still having trouble. Let's continue to ignore that for now.

3:50:24.158 [warning] [libcluster:convoy_topology] unable to connect to :"convoy@fd12:98a1:4fac:0:2000:15:da65:1988"

Let's try to establish a tcp connection directly to the inter-node comms port:

:gen_tcp.connect({64786, 39073, 20396, 0, 8192, 43, 43332, 50874}, 4001, [:inet6])
{:error, :econnrefused}

Shucks.

After taking a step back and poking around railway's network settings for the service attached to my control-plane, I can see that 4001 may be used for something else. I'm not sure if this is a conflicting process or not -- perhaps it's reserved. So let's try a port that is likely not in use already.

# .. same env variables above
ERL_AFLAGS="-proto_dist inet6_tcp -kernel inet_dist_listen_min 4444 inet_dist_listen_max 4444"

Attempt 2

I deployed the changes, waited for the nodes to restart, verified the distribution port was configured and tested another tcp connection.

# on a remote node
iex> :os.cmd(~c"epmd -names")
~c"epmd: up and running on port 4369 with data:\nname convoy at port 4444\n"


iex> :gen_tcp.connect({64786, 39073, 20396, 0, 8192, 34, 26795, 13620}, 4444, [:inet6])
{:ok, #Port<0.205>}

Oh YEAHHHHHH.

So if a tcp connection on the port used for inter-node comms works, let's see if we can connect

# on the control plane
iex> Node.connect(:"convoy@convoy-jfb7.railway.internal")
true

OHHHHHHHH!

# on the control plane
iex>Node.list
[:"convoy@convoy-jfb7.railway.internal", :"convoy@convoy.railway.internal"]

YEAHHHHH!!

Looks like a little port magic was required. This is pretty cool. But it'd be even cooler if the nodes connected automatically. Looking at you libcluster.

Before we dig into that, let's summarize what we know so far:

Internal dns lookups are properly resolving to hosts, so internal dns is working as expected.
EPMD is running and on the port we expect; 4369.
Inter-node comms for the erlang distribution protocol is working on port 4444.
We can successfully connect via tcp on 4369 and 4444.
Nodes are properly self identifying as <app-name>@<dns name> like convoy@convoy.railway.internal for the control-plane and convoy@convoy-jfb7.railway.internal for other nodes.
And our nodes can connect to each other with Node.connect/1.

Node Auto-Discovery

Let's figure out what's going on with libcluster. Checking out the logs again I notice that the node name contains an ipv6 address instead of a dns name. It makes sense that trying to connect with this node name wouldn't work because our nodes self identify as app-name@dns-name.

3:50:24.158 [warning] [libcluster:convoy_topology] unable to connect to :"convoy@fd12:98a1:4fac:0:2000:15:da65:1988"

My question is, how and why is the node's name being built this way? Perhaps if the node name followed the format that we expect they would connect.

After taking a look at libcluster's code a little, specifically the Cluster.Strategy.DNSPoll strategy module and the Cluster.Supervisor module, two important bits stick out to me.

libcluster's default connect function is connect_node/1 defined in erlang's networking kernel⁴. It expects a node name as an atom in order to connect. So node@ip and node@dns-name are both valid as far as this function is concerned and should work similarly to the function we used when manually connecting nodes Node.connect/1.

# Cluster.Supervisor
connect_mfa = Keyword.get(spec, :connect, {:net_kernel, :connect_node, []})

The main culprit looks to be this lookup.

# Cluster.Strategy.DNSPoll
def lookup_all_ips(q) do
  Enum.flat_map([:a, :aaaa], fn t -> :inet_res.lookup(q, :in, t) end)
end

This is potentially returning a list of ip's. I'm thinking we should tweak this and see what happens when we return dns hostnames instead of ip's.

So I duped the DNSPoll module, created DnsPollRailway and made some changes.

When getbyname properly resolves and gives us a dns hostname we return that hostname. We're also only concerned with ipv6 addresses so we specify :aaaa for the type of dns record we are interested in.

# DnsPollRailway
defp lookup_all_names(q) do
  case :inet_res.getbyname(q, :aaaa) do
    {:ok, {:hostent, name, _, _, _, _}} ->
      [name]

    {:error, reason} ->
      IO.warn("failed to get by name: #{inspect(reason)}")
      [nil]
  end
end

Since this potentially returns nil, we make sure that we reject any nils in the resolve function.

# DnsPollRailway
query
|> resolver.()
|> Enum.reject(fn n -> is_nil(n) end)
|> Enum.map(&format_node(&1, node_basename))
|> Enum.reject(fn n -> n == me end)

Then, update the libcluster config to use the custom DnsPollRailway strategy.

config :libcluster,
  topologies: [
    convoy_topology: [
      strategy: Convoy.DnsPollRailway,
      config: [
        polling_interval: 5_000,
        query: "convoy.railway.internal",
        node_basename: "convoy"
      ]
    ]
  ]

Let's see if it works.

Attempt 3

We remove the previous child nodes as they'll have an outdated version of the strategy running, then deploy our new strategy to the control-plane. Since our control-plane's service is configured to listen to changes on the main branch, once we push to main it starts the deployment process automatically.

Slight tangent, but I must admit, railway makes it ridiculously easy to iterate in a deployed environment.

Anyway, once our deployment is finished, we spin up 2 new nodes, and wait for them to become available.

Moment of truth.

# on our control plane (convoy@convoy.railway.internal)
iex(0)>Node.list
[:"convoy@convoy-8ryx.railway.internal", :"convoy@convoy-xfbx.railway.internal"]

THERE IT IS!!

Let's check the other nodes as well.

# on convoy@convoy-8ryx.railway.internal
iex(1)> Node.self
:"convoy@convoy-8ryx.railway.internal"

iex(2)> Node.list
[:"convoy@convoy.railway.internal", :"convoy@convoy-xfbx.railway.internal"]

# on convoy@convoy-xfbx.railway.internal
iex(1)> Node.self
:"convoy@convoy-xfbx.railway.internal"

iex(2)> Node.list
[:"convoy@convoy.railway.internal", :"convoy@convoy-8ryx.railway.internal"]

Awesome. And for a little icing on the cake, let's check what the libcluster logs say for our services.

# convoy-xfbx deployment logs
Starting Container
14:44:26.584 [info] [libcluster:convoy_topology] connected to :"convoy@convoy.railway.internal"

# convoy-8ryx deployment logs
Starting Container
14:44:20.385 [info] [libcluster:convoy_topology] connected to :"convoy@convoy.railway.internal"

Nice. No problemo. What happens when we remove the nodes?

# on convoy@convoy.railway.internal
iex(1)> Node.list
[]

Just like that we have an elixir cluster that auto-discovers nodes utilizing internal dns lookups on railway. We can spin up and spin down nodes in our deployed environment and the cluster forms automatically.

Final thoughts

Elixir/Erlang clusters are pretty cool. You can call a function on another node as if it were local. You can have multiple nodes subscribed to pub-sub topics. You can distribute or offload work from one node to another. All with a few lines of code.

In some cases, it may be desirable to have node scheduling, scaling, and load-balancing logic closer to your application code.

The possibilities of elixir clusters are exciting to think about.

At the end of this post you'll find a code and config summary; might be helpful for reference purposes.

Thanks for reading!

Code and Config Summary

These variables must be set on the elixir nodes in your railway environment. On railway, you can set these in the variables tab of your service. Or if you're launching nodes using railway's api, you can specify these in a serviceCreate mutation, here's an example.

RELEASE_DISTRIBUTION=name
# convoy@convoy.railway.internal, convoy@convoy-jfb7.railway.internal, etc.
RELEASE_NODE=convoy@<node-name>.railway.internal
RELEASE_COOKIE="super-secret-cookie"
# specifying the port is critical, especially on railway
PORT=4000
# these are standard, not special for this situation
PHX_HOST=...
SECRET_KEY_BASE=".."
# ensure that the distribution protocol is communicating over ipv6
# and that inter-node communications are explicitly set to 4444
ERL_AFLAGS="-proto_dist inet6_tcp -kernel inet_dist_listen_min 4444 inet_dist_listen_max 4444"

libcluster's DNSPoll strategy won't work out of the box on railway so we use our own custom strategy. You can see the full module here, in our app it is placed in lib/dns_poll_railway.ex and named Convoy.DnsPollRailway and used in libcluster's configuration.

The DnsPollRailway strategy closely resembles DNSPoll but the main difference is
how dns lookups are performed.

# DNSPoll - resolves to an ipv6 address
:inet_res.lookup(query)

# DNSPollRailway - resolves to a hostname and ipv6 address
:inet_res.getbyname(query)

In config.exs, this is what the libcluster config looks like.

config :libcluster,
  topologies: [
    convoy_topology: [
      strategy: Convoy.DnsPollRailway,
      config: [
        polling_interval: 5_000,
        query: "convoy.railway.internal", # our control plane's dns name
        node_basename: "convoy" # base name for all nodes
      ]
    ]
  ]

And of course, you should update your supervision tree in application.ex.

  @impl true
  def start(_type, _args) do
    children = [
      # ... other children
      {Cluster.Supervisor,
       [Application.get_env(:libcluster, :topologies), [name: Convoy.ClusterSupervisor]]},
      # ... other children
    ]

    #..
    opts = [strategy: :one_for_one, name: Convoy.Supervisor]
    Supervisor.start_link(children, opts)
  end

You can find all of this in my repo

If you're having trouble connecting your nodes, here's some helpful debugging commands:

# show the node's name
iex> Node.self
# list connected nodes
iex> Node.list
# connect to a node
iex> Node.connect(:"convoy@convoy-jfb7.railway.internal")
# get the cookie for a node
iex> Node.get_cookie
# dns resolution related
iex> :inet.gethostbyname('convoy.railway.internal', :inet6)
iex> :inet_res.getbyname(~c"convoy.railway.inernal")
iex> :inet_res.lookup(~c"convoy.railway.inernal")
# see where epmd is running and what distribution port is set to
iex> :os.cmd(~c"epmd -names")
# tcp connections
iex> :gen_tcp.connect({64786, 39073, 20396, 0, 8192, 11, 41282, 21287}, 4000, [:inet6])
iex> :gen_tcp.connect({64786, 39073, 20396, 0, 8192, 11, 41282, 21287}, 4369, [:inet6])
iex> :gen_tcp.connect({64786, 39073, 20396, 0, 8192, 4, 6082, 5148}, <DISTRIBUTION-PORT>, [:inet6])

Additional reading

Erlang Distribution over TLS

Erlang Remote Procedure Call

Foot notes

This RELEASE environment variables reference goes into more detail. There's quite a few that could be helpful in other situations.↩
Manual page for the Erlang Distribution Protocol.↩
The erl program.↩
Spec for :net_kernel's connect_node/1 function. Looks pretty similar to Node.connect/1.↩