fix: embedded etcd should only connect to self

[jason-lynch]

Summary

Fixes a bug where the etcd clients in hosts with embedded etcd were configured to connect to all cluster members that existed when the client was initialized. The correct behavior is that hosts with embedded etcd should only connect to themselves. This was the original intent and functionality, but I changed it while implementing support for remote Etcd. I did a few different implementations of the remote Etcd feature before settling on the current one, and I think this was just an accidental inclusion from a different implementation of that feature.

PLAT-581

Testing

This PR includes a cluster test for the issue:

# Using TEST_RERUN_FAILS in case the OS steals our ephemeral ports
make test-cluster TEST_RERUN_FAILS=2 CLUSTER_TEST_RUN=TestRollingAddRemove

To run the scenario by hand:

# start the servers in an uninitialized state
make dev-reset
make dev-watch

# join 4 hosts to the cluster
cp1-req init-cluster
cp2-req join-cluster $(cp1-req get-join-token)
cp3-req join-cluster $(cp1-req get-join-token)

# stop host-1
cp-docker-compose stop host-1

# remove host-1 - remember that this is asynchronous, so watch the logs for it to finish
cp2-req remove-host host-1

# add another host to the cluster
cp4-req join-cluster $(cp2-req get-join-token)

Note that our dev setup is slightly different than the scenario in the ticket and our cluster test because hosts 4-6 are in client mode. It still reproduces and validates the issue because the failure occurs in host-2.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR adds a new rolling add/remove host test for cluster management and refactors internal state update ordering. Changes include reordering the Cluster.Remove method's state updates and updating EmbeddedEtcd's client configuration to use ClientEndpoints instead of cluster URLs.

Changes

Cohort / File(s)	Summary
Cluster Test Suite clustertest/add_remove_host_test.go, clustertest/cluster_test.go	New TestRollingAddRemove test exercises iterative host addition and removal with cluster initialization. Cluster.Remove method refactored to update host state and recreate client before waiting on removal task completion rather than after.
Embedded Etcd Configuration server/internal/etcd/embedded.go	GetClient method updated to build clientConfig using ClientEndpoints instead of cluster ClientURLs, with added clarification comment about connecting only to the embedded node's endpoint.

Poem

🐰 A cluster grows, then shrinks with care,
Adding hosts and removing there,
The etcd endpoint takes new ground,
State updates dance without a sound,
Rolling changes, smooth and fair! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: fixing embedded etcd to connect only to itself rather than all cluster members.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description is comprehensive and follows the template structure with Summary, Testing, and clear documentation of the bug fix with testing instructions.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/PLAT-581/embedded-should-only-connect-to-self

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 0 duplication

Metric	Results
Duplication	0

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}