Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added Thumb-2 support to armv6m JIT backend, optimizing code for ARMv7-M and later cores
- Added support for `binary:split/2,3` list patterns and `trim` / `trim_all` options
- Added `timer:send_after/2`, `timer:send_after/3` and `timer:apply_after/4`
- Added Erlang distribution over serial (uart)

### Changed
- ~10% binary size reduction by rewriting module loading logic
Expand Down
148 changes: 147 additions & 1 deletion doc/src/distributed-erlang.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,15 @@ Distribution is currently available on all platforms with TCP/IP communication,
- ESP32
- RP2 (Pico)

Two examples are provided:
Distribution over serial (UART) is also available for point-to-point
connections between any two nodes, including microcontrollers without
networking (e.g. STM32). See [Serial distribution](#serial-distribution).

Three examples are provided:

- disterl in `examples/erlang/disterl.erl`: distribution on Unix systems
- epmd\_disterl in `examples/erlang/esp32/epmd_disterl.erl`: distribution on ESP32 devices
- serial\_disterl in `examples/erlang/serial_disterl.erl`: distribution over serial (ESP32 and Unix)

## Starting and stopping distribution

Expand Down Expand Up @@ -94,6 +99,147 @@ fun (DistCtrlr, Length :: pos_integer(), Timeout :: timeout()) -> {ok, Packet} |

AtomVM's distribution is based on `socket_dist` and `socket_dist_controller` modules which can also be used with BEAM by definining `BEAM_INTERFACE` to adjust for the difference.

## Serial distribution

AtomVM supports distribution over serial (UART) connections using the
`serial_dist` module. This is useful for microcontrollers that lack
WiFi/TCP (e.g. STM32) but have UART, and for testing distribution
locally using virtual serial ports.

### Quick start

```erlang
{ok, _} = net_kernel:start('mynode@serial.local', #{
name_domain => longnames,
proto_dist => serial_dist,
avm_dist_opts => #{
uart_opts => [{peripheral, "UART1"}, {speed, 115200},
{tx, 17}, {rx, 16}],
uart_module => uart
}
}).
```

On Unix, the `peripheral` is a device path such as `"/dev/ttyUSB0"` and
the `uart_module` is `uart` from the `avm_unix` library.

### serial\_dist options

- `uart_opts` — proplist passed to `UartModule:open/1` for a single port
(see `uart_hal` for common parameters: `peripheral`, `speed`,
`data_bits`, `stop_bits`, `parity`, `flow_control`)
- `uart_ports` — list of proplists, one per UART port. Use instead of
`uart_opts` when connecting to multiple peers.
- `uart_module` — module implementing the `uart_hal` behaviour. Defaults
to `uart`.

### Wire protocol

All packets on the wire use the same frame format:

```
<<16#AA, 16#55, Length:LenBits/big, Payload:Length/binary, CRC32:32/big>>
```

where `LenBits` is 16 during the handshake phase and 32 during the data
phase. The CRC32 covers the `Length` and `Payload` bytes (everything
between the sync marker and the CRC itself).

The receiver scans for the `<<16#AA, 16#55>>` sync marker, reads the
length field, validates it against a maximum frame size (to reject false
sync matches where the marker appears in stale data), then verifies the
CRC32. On CRC failure the connection is torn down.

**Sync markers**

Both sides periodically send bare 2-byte sync markers
(`<<16#AA, 16#55>>`) on the UART outside of any frame. These serve two
purposes:

- **Liveness detection**: a node knows its peer is alive when it
receives sync markers.
- **Stale data recovery**: after a failed handshake attempt, leftover
bytes remain in the UART buffer. The frame scanner skips over any
data (including stale sync markers) that does not form a valid frame
with a correct length and CRC.

**Handshake phase (16-bit length)**

During the Erlang distribution handshake, the `Length` field is 16 bits.
The handshake follows the standard Erlang distribution protocol
(send\_name, send\_status, send\_challenge, send\_challenge\_reply,
send\_challenge\_ack).

**Data phase (32-bit length)**

After the handshake completes, the `Length` field switches to 32 bits.
Tick (keepalive) messages are sent as a frame with a zero-length payload
(i.e. `Length = 0`).

### Peer-to-peer connection model

Unlike TCP distribution which uses a client/server model (one side
listens, the other connects), serial is point-to-point: both nodes
share a single UART link.

A **link manager** process on each node is the sole owner of UART
reads. On each iteration it:

1. Checks its mailbox for a `setup` request from `net_kernel`
(non-blocking). If found, enters the **setup** path (initiator)
immediately without proceeding to subsequent steps.
2. Sends a sync marker.
3. Reads from the UART with a short timeout.
4. Passes the buffer to `scan_frame` which searches for a valid framed
handshake packet.
5. If a complete or partial frame is detected, enters the **accept**
path (responder).
6. Otherwise, loops.

This design ensures only one process reads from the UART at any time,
avoiding the race condition that would occur if separate accept and
setup processes competed for the same byte stream.

If a handshake fails (the distribution controller process exits), the
link manager flushes stale `setup` messages from its mailbox and
restarts the loop, allowing retries.

### Testing with socat

On Unix, `socat` can create virtual serial port pairs for testing:

```bash
socat -d -d pty,raw,echo=0 pty,raw,echo=0
```

This creates two pseudo-terminal devices (e.g. `/dev/ttys003` and
`/dev/ttys004`) connected back-to-back. Each AtomVM node uses one side:

```erlang
%% Node A
{ok, _} = net_kernel:start('a@serial.local', #{
name_domain => longnames,
proto_dist => serial_dist,
avm_dist_opts => #{
uart_opts => [{peripheral, "/dev/ttys003"}, {speed, 115200}],
uart_module => uart
}
}).

%% Node B (separate AtomVM process)
{ok, _} = net_kernel:start('b@serial.local', #{
name_domain => longnames,
proto_dist => serial_dist,
avm_dist_opts => #{
uart_opts => [{peripheral, "/dev/ttys004"}, {speed, 115200}],
uart_module => uart
}
}).

%% From Node B, trigger autoconnect:
{some_registered_name, 'a@serial.local'} ! {self(), hello}.
```

## Distribution features

Distribution implementation is (very) partial. The most basic features are available:
Expand Down
1 change: 1 addition & 0 deletions examples/erlang/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ pack_runnable(network_console network_console estdlib eavmlib alisp)
pack_runnable(logging_example logging_example estdlib eavmlib)
pack_runnable(http_client http_client estdlib eavmlib avm_network)
pack_runnable(disterl disterl estdlib)
pack_runnable(serial_disterl serial_disterl eavmlib estdlib DIALYZE_AGAINST avm_esp32 avm_unix)
pack_runnable(i2c_scanner i2c_scanner eavmlib estdlib DIALYZE_AGAINST avm_esp32 avm_rp2 avm_stm32)
pack_runnable(i2c_lis3dh i2c_lis3dh eavmlib estdlib DIALYZE_AGAINST avm_esp32 avm_rp2 avm_stm32)
pack_runnable(spi_flash spi_flash eavmlib estdlib DIALYZE_AGAINST avm_esp32 avm_rp2)
Expand Down
151 changes: 151 additions & 0 deletions examples/erlang/serial_disterl.erl
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
%
% This file is part of AtomVM.
%
% Copyright 2026 Paul Guyot <pguyot@kallisys.net>
%
% Licensed under the Apache License, Version 2.0 (the "License");
% you may not use this file except in compliance with the License.
% You may obtain a copy of the License at
%
% http://www.apache.org/licenses/LICENSE-2.0
%
% Unless required by applicable law or agreed to in writing, software
% distributed under the License is distributed on an "AS IS" BASIS,
% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
% See the License for the specific language governing permissions and
% limitations under the License.
%
% SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later
%

%% @doc Example: distributed Erlang over serial (UART).
%%
%% This example starts distribution using one or more UART connections
%% instead of TCP/IP. It works on ESP32 and Unix (using real serial
%% devices or virtual serial ports created with socat).
%%
%% <h3>ESP32 wiring (single peer)</h3>
%%
%% ```
%% ESP32 TX (GPIO 17) -> Peer RX
%% ESP32 RX (GPIO 16) -> Peer TX
%% ESP32 GND -> Peer GND
%% '''
%%
%% <h3>ESP32 with two peers</h3>
%%
%% Many ESP32 boards have UART1 and UART2. Set `SERIAL_MULTI=true'
%% to connect to two peers simultaneously:
%%
%% ```
%% UART1: TX=17, RX=16 -> Peer A
%% UART2: TX=4, RX=5 -> Peer B
%% '''
%%
%% <h3>Unix with socat</h3>
%%
%% Create a virtual serial port pair:
%% ```
%% socat -d -d pty,raw,echo=0 pty,raw,echo=0
%% '''
%% Then set the SERIAL_DEVICE environment variable to one of the pty
%% paths before running this example. The peer node uses the other pty.
%%
%% <h3>Connecting</h3>
%%
%% The node name is derived from the serial device, e.g.
%% `ttys003@serial.local' or `uart1@serial.local'. Once both nodes are
%% running, trigger autoconnect from either side:
%% ```
%% {serial_disterl, 'ttys003@serial.local'} ! {hello, node()}.
%% '''
-module(serial_disterl).

-export([start/0]).

start() ->
UartConfigs = uart_configs(),
NodeName = make_node_name(hd(UartConfigs)),
DistOpts =
case UartConfigs of
[Single] -> #{uart_opts => Single};
Multiple -> #{uart_ports => Multiple}
end,
{ok, _NetKernelPid} = net_kernel:start(NodeName, #{
name_domain => longnames,
proto_dist => serial_dist,
avm_dist_opts => DistOpts
}),
io:format("Distribution started over serial (~p port(s))~n", [length(UartConfigs)]),
io:format("Node: ~p~n", [node()]),
net_kernel:set_cookie(<<"AtomVM">>),
io:format("Cookie: ~s~n", [net_kernel:get_cookie()]),
register(serial_disterl, self()),
io:format("Registered as 'serial_disterl'. Waiting for messages.~n"),
io:format("From the peer:~n"),
io:format(" {serial_disterl, '~s'} ! {hello, node()}.~n", [node()]),
loop().

%% Build a node name from the serial device.
%% e.g. "UART1" -> 'uart1@serial.local'
%% "/dev/ttys003" -> 'ttys003@serial.local'
make_node_name(UartOpts) ->
Peripheral = proplists:get_value(peripheral, UartOpts, "serial"),
BaseName = basename(Peripheral),
list_to_atom(string:to_lower(BaseName) ++ "@serial.local").

basename(Path) ->
case lists:last(string:split(Path, "/", all)) of
[] -> Path;
Name -> Name
end.

%% Platform-specific UART configuration.
%% Returns a list of UART option proplists (one per port).
uart_configs() ->
case erlang:system_info(machine) of
"ATOM" ->
case atomvm:platform() of
esp32 ->
case os:getenv("SERIAL_MULTI") of
"true" ->
%% Two UARTs: connect to two peers
[
[{peripheral, "UART1"}, {speed, 115200}, {tx, 17}, {rx, 16}],
[{peripheral, "UART2"}, {speed, 115200}, {tx, 4}, {rx, 5}]
];
_ ->
[[{peripheral, "UART1"}, {speed, 115200}, {tx, 17}, {rx, 16}]]
end;
generic_unix ->
Device = os:getenv("SERIAL_DEVICE"),
case Device of
false ->
io:format("Error: set SERIAL_DEVICE env var to a serial port path~n"),
io:format(" e.g. /dev/ttyUSB0 or a socat pty~n"),
exit(no_serial_device);
_ ->
[[{peripheral, Device}, {speed, 115200}]]
end;
Other ->
io:format("Error: unsupported platform ~p~n", [Other]),
exit({unsupported_platform, Other})
end;
"BEAM" ->
io:format("Error: this example requires AtomVM~n"),
io:format(" See serial_dist module doc for BEAM usage~n"),
exit(beam_not_supported)
end.

loop() ->
receive
quit ->
io:format("Received quit, stopping.~n"),
ok;
{hello, From} ->
io:format("Hello from ~p!~n", [From]),
loop();
Other ->
io:format("Received: ~p~n", [Other]),
loop()
end.
3 changes: 3 additions & 0 deletions libs/estdlib/src/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ set(ERLANG_MODULES
maps
math
net
os
proc_lib
sys
logger
Expand All @@ -68,6 +69,8 @@ set(ERLANG_MODULES
queue
sets
socket
serial_dist
serial_dist_controller
socket_dist
socket_dist_controller
ssl
Expand Down
5 changes: 4 additions & 1 deletion libs/estdlib/src/net_kernel.erl
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,10 @@ handle_cast(_Message, State) ->
{noreply, State}.

%% @hidden
handle_info({accept, AcceptPid, SocketPid, inet, tcp}, #state{proto_dist = ProtoDist} = State) ->
handle_info(
{accept, AcceptPid, SocketPid, _Family, _Protocol},
#state{proto_dist = ProtoDist} = State
) ->
Pid = ProtoDist:accept_connection(AcceptPid, SocketPid, State#state.node, [], ?SETUPTIME),
AcceptPid ! {self(), controller, Pid},
{noreply, State};
Expand Down
Loading
Loading