mirror of
https://github.com/EasyTier/EasyTier.git
synced 2026-09-20 03:22:05 +00:00
fix(web): harden managed config sync between console and clients (#2567)
* fix(web): fence managed config runtime reconciliation Keep runtime reconciliation tied to the currently authorized session so stale connections cannot mutate a replacement session runtime. Accumulate only contiguous dirty IDs and load their latest SQLite state. Require the applied revision to match the earliest Patch base and the persisted revision to match the latest target. Otherwise, reconcile the full desired state. Use separate runtime-state and config-cache epochs. Managed updates can reuse observed configs; direct mutations invalidate them. Update sync documentation to match. * fix(web): interrupt validation retry on state changes Track meaningful validation state changes separately from periodic dirty signals. Applied revision changes wake a failed validation immediately, while heartbeat-driven revalidation retains the retry backoff. Treat Notify as a wake-up hint and recheck the state-change epoch after every wake so stored permits and periodic heartbeats cannot cause retry storms. * fix(web): retry unconfirmed connected webhooks Retry node-connected webhook delivery on retryable errors with a short 100ms/500ms backoff and give up immediately on non-retryable errors. Re-check that the session still owns the connection before every attempt and before recording the delivery, so a replaced session can no longer record a stale connected binding. * fix(web): fence disconnects by session ownership Return whether session removal actually removed the current route owner, and emit disconnected only for that owner. Replaced sessions can no longer invalidate a newer connected route. * fix(web): hot-patch managed hostnames Include hostname changes in the hot-patch path instead of falling back to a full restart. When a full overwrite run is required and the desired config has no hostname, inherit the current runtime hostname so an unmanaged value survives until it is explicitly cleared. Read back the runtime config after an overwrite run and verify it converged instead of assuming the desired state was applied. * fix(web): retry transient runtime reconciliation failures Keep the per-session managed runtime reconciliation worker alive when a single database round fails. Retry from the next heartbeat so persisted managed revisions can still converge after restart-time contention. Reserve terminal worker shutdown for destroyed session or storage state, and cover recovery after a transient revision read failure. * fix(web): accept omitted hostname after runtime apply Release 2.6.4 omits hostname from config readback when it matches the device hostname. Trust a successful hostname mutation only when the returned field is absent, while continuing to verify every other field and rejecting explicit mismatches. * fix(web): ignore unmanaged runtime device names Windows release 2.6.4 generates a random interface name when the managed config leaves dev_name empty. Exclude that runtime-owned value from reconciliation unless the desired config explicitly sets a non-empty device name, preventing endless overwrite restarts. * feat(web): report failed network instances to console Expose stopped Core instances with startup errors in heartbeats. Merge Core failures with direct managed-run RPC failures in easytier-web. Send failed instance IDs during token validation without error text. Prune local run failures when managed configs are deleted. * fix(web): distinguish unknown runtime application state Track whether the current session has observed its applied revision separately from the optional revision value. Report this fact through validate-token so Console can preserve application state across receiver restarts while recognizing deliberate pending mutations. * feat(web): configure heartbeat timing from server Heartbeat responses now provide the interval and RPC timeout. Legacy servers use local defaults and remote values are clamped. Web configuration and session receive timeout follow the policy. * fix(web): reject inactive control sessions Route control RPCs by machine id only to sessions whose RPC manager is still running, so a session that has been stopped or replaced can no longer receive control traffic addressed to the device. * fix(core): filter network info before collection When a collect-network-info request names specific instances, collect those instances only instead of collecting every instance and filtering the result afterwards, so unrequested instances no longer run per-collection work on every request. * feat(web): enable focused runtime diagnostics Enable easytier-web info logs by default while preserving explicit log configuration. Record startup settings, session lifecycle, failed instance changes, webhook queue and request latency, and managed runtime operation timings for production diagnosis. * fix(web): preserve managed revision across reconnects Keep one runtime identifier for each Core WebClient lifetime. Reuse its managed runtime state after transport reconnects. Retain applied revisions and reconcile hints while disconnected. Preserve runtime epochs so stale work cannot mark a revision applied. Reject stale sessions from reclaiming routes after reconnect. Core or Web restarts and legacy clients still use unknown state. Immediately revalidate a restored revision after authentication. Document local management RPC drift as an accepted trade-off. This lets Console converge without waiting for periodic validation. * fix(web): satisfy clippy across managed config sync tests Scope managed runtime guards to blocks in runtime revision tests so no std MutexGuard is held across await points, return the applied revision directly instead of through a let binding, and pass WebhookValidationInput to request_heartbeat_validation instead of expanding it into eight separate arguments. * fix(core): stop reporting failed instances as running in heartbeats A stopped instance with a startup error appeared in both running_network_instances and failed_network_instances, so the server treated it as running and never re-ran its managed config. Exclude failed instance ids when building the running list so the reconciler restarts them. * fix(core): close missed-wakeup race in instance state changes wait_for_change created the Notified future before reading the generation but only registered it when awaited. A change landing in between fired notify_waiters with no registered waiter and delayed the heartbeat by a full interval. Enable the future before reading the generation so every change wakes a waiting heartbeat. * fix(web): address review findings Fence webhook validation and connection transitions against stale state, redact credentials from default-level logs, and stabilize runtime reconciliation: - Record connected bindings only while the session still owns the machine route, and skip disconnect compensation once a replacement owns the route so a stale disconnect cannot revoke it. - Discard webhook validation results when the change epoch moved during the HTTP round, so a stale rejection cannot invalidate the current session. - Drop user_token fields from info and warn logs that became visible with info-level defaults. - Restore a hostname omitted by the 2.6.4 readback into the cached runtime config after a successful mutation, so later rounds stop re-sending the same hostname patch. - Reconcile running web configs when no revision is tracked so legacy unrevisioned updates converge, and wake sessions for unrevisioned full updates instead of waiting for the next heartbeat. * chore(go): regenerate web proto bindings for heartbeat fields Add failed_network_instances, support_heartbeat_policy, and the heartbeat policy response fields to the checked-in Go bindings. Other proto packages are left as-is because their drift predates this change. * fix(web): redact user tokens from positional log arguments Three runtime reconciliation info logs and the user lookup error contexts printed user_token through format arguments, which the earlier field-syntax redaction missed. The reconcile log now fires every round for unrevisioned machines, so remove the token from these messages as well. * fix(web): fence stale validation and runtime reconcile rounds Check webhook validation epochs while holding the session write lock, so stale success and rejection responses cannot change session state. Advance the runtime epoch for unrevisioned full config updates, and exclude failed instances from heartbeat and RPC reconciliation lists so stopped instances are restarted instead of repeatedly hot-patched. Release test read guards before awaiting validation apply calls. Set up the no-pending condition before asserting that an applied revision is a no-op, and verify that its runtime epoch remains unchanged. Validation: all 137 client_manager tests passed. * test(credentials): cover P2P with active VPN portal Model an admin and temporary credential peer connected as a foreign network through a public server with data relay disabled. Verify their direct connection can be replaced after a WireGuard portal client comes online. * test(credentials): stabilize two-admins failover assertions The two-admins non-reusable credential test could fail on slow convergence: after dropping the winning peer it relied on a single route sample passing a bare AND condition, then re-asserted the same expectations through one-shot checks seconds later. A transient route flap in that window (for example a briefly resurrected winner route from stale conn info) turned a passing convergence into a hard assert failure. This matches the 48.9s CI flake of credential_non_reusable_across_two_admins_allows_only_one_peer observed on 2026-08-12. Changes: - wait for bidirectional admin connectivity (AND) with a 20s budget before issuing the credential, instead of a one-directional OR - replace the failover wait_for_condition with wait_stable_failover_visibility_on_admins, which requires three consecutive samples of loser-present and winner-absent on both admins within the same 60s budget and logs every sample - enrich the stable-single-winner timeout message with per-admin visibility flags and elapsed time for triage All existing contracts are preserved; only observation windows and diagnostics change. Validated in the rust container: three passes at normal speed (54.1s / 53.8s / 53.1s) plus one slow-convergence round (172.7s) that would have raced the old one-shot sampling; it now passes with failover samples logged. cargo fmt and clippy -D warnings clean.
This commit is contained in:
KKRainbow
@@ -7,9 +7,9 @@
|
||||
- 上游依赖：后续由 Console 计算并发送 Patch
|
||||
- 兼容要求：保留现有 Full PUT
|
||||
|
||||
本文记录当前接收端方案。Session 在能够证明 Patch base 与已应用 revision 连续时
|
||||
只收敛 touched instances；重启、通知丢失、revision 断链或并发积压时沿用 Full
|
||||
reconcile。
|
||||
本文记录当前接收端方案。Session 合并已持久化 Patch 的 touched instance IDs，
|
||||
并在运行态收敛时读取这些实例的最新持久化状态。重启、通知丢失或无法安全判断
|
||||
实例 ownership 时沿用 Full reconcile。
|
||||
|
||||
## 1. 背景与结论
|
||||
|
||||
@@ -30,8 +30,10 @@ Console 每次发布都会向该路径发送完整 Exact Set。实例很多时
|
||||
3. PATCH 使用 `expected_config_revision` 做 compare-and-swap（CAS）。
|
||||
4. Full/Patch 的配置变更与 revision 更新在一个 SQLite transaction 中提交。
|
||||
5. Patch 只查询和写入 touched instances，不扫描完整 Target。
|
||||
6. 写入成功后通知 Session 本次 base、target 和 touched instance IDs。
|
||||
7. Session 仅在 applied revision 精确匹配 base 时增量收敛，否则安全回退 Full。
|
||||
6. 写入成功后通知 Session 本次 expected、target 和 transaction 实际 touched
|
||||
instance IDs。
|
||||
7. Session 只合并 revision 连续的 touched IDs，并以 SQLite 当前状态为准增量
|
||||
收敛；可信 runtime base、通知链或 persisted target 无法证明连续时回退 Full。
|
||||
|
||||
普通变更的接收端成本由：
|
||||
|
||||
@@ -371,20 +373,33 @@ revision。只影响 user-owned rows 的操作不清除 managed revision。
|
||||
- 只有带 target revision 的 `Applied` 才通知匹配的 live Session；
|
||||
`AlreadyApplied`、legacy unrevisioned Full、conflict 和失败不重复通知。
|
||||
- Notification 必须发生在 commit 之后。
|
||||
- Full notification 清除任何 pending delta，触发完整收敛。
|
||||
- Full notification 将 pending reconcile hint 提升为 Full，触发完整收敛。
|
||||
- Patch notification 携带 expected revision、target revision、upsert IDs 和本次
|
||||
transaction 实际接受删除的 web-owned IDs。请求删除但数据库原本不存在的 ID
|
||||
仍是 no-op，不能借机删除 Core 中同 ID 的 user-owned 实例。只有 Session applied
|
||||
revision 精确等于 expected revision，且没有更早的 Patch 等待处理时，才保留该
|
||||
delta。
|
||||
- 两次 Patch 在前一次完成前积压时不合并 delta；Session 清除 pending delta，并在
|
||||
最新 heartbeat/revision 上执行一次 Full。这避免引入 Patch queue 或 delivery FSM。
|
||||
- 增量 round 只读取 upsert rows，只删除本次 delete IDs，只对 touched running
|
||||
instances 执行 runtime Patch/Run。完成前再次校验 persisted target revision；只有
|
||||
全部 touched instances 成功且 target 仍相同，才推进 applied revision。
|
||||
仍是 no-op，不能借机删除 Core 中同 ID 的 user-owned 实例。
|
||||
- Session 将尚未应用的 Patch touched IDs 合并为一个 Dirty set，并始终以 SQLite
|
||||
最新 revision 下的 rows 为准。它不重放历史 Patch，也不维护 Patch queue 或
|
||||
delivery FSM。只有 incoming expected 等于 pending target 的通知才能合并；Dirty
|
||||
hint 保留最早 expected 和最新 target。乱序、不连续或无法证明顺序的通知将 hint
|
||||
提升为 Full。多个连续 Patch 积压时，旧 round 由 runtime epoch 拦截，下一 round
|
||||
直接收敛到最新 target。
|
||||
- Session 分开记录对外报告的 applied revision 和内部可信的 runtime base。开始任何
|
||||
runtime side effect 前清除 applied；Patch round 的 side effects 完全包含在 Dirty
|
||||
set 中，因此失败或被新通知拦截时仍保留最早 runtime base，以便按最新持久化状态
|
||||
重试 Dirty set。Full round、direct mutation、授权失败或 Session ownership 中断会
|
||||
清除 runtime base。
|
||||
- 只有可信 runtime base 等于 Dirty 最早 expected，并且 SQLite persisted revision
|
||||
等于 Dirty 最新 target 时，才允许增量 round。重连后 runtime base 未知、通知
|
||||
丢失，或 SQLite 已经提交了更靠后的 revision 而通知尚未送达时都回退 Full，避免
|
||||
不完整的 Dirty set 把完整 target revision 误标为已应用。
|
||||
- 增量 round 逐个读取 Dirty set 中的最新 row。仍然存在且启用的 web-owned row
|
||||
使用其最新 config；已经删除的 row 进入 delete set；遇到 disabled 或非 web-owned
|
||||
row 时回退 Full，以保留 ownership 规则。完成前再次校验 persisted target
|
||||
revision；只有全部 touched instances 成功且 target 仍相同，才推进 applied
|
||||
revision。
|
||||
- 任何通过 EasyTier Web mutation route 直接 Run、Save、Delete 或切换实例状态的
|
||||
操作在执行前和结束后（包括部分 side effect 后返回错误）都清除 Session applied
|
||||
revision 与 pending delta、增加运行配置 cache epoch，并唤醒一次 Full
|
||||
revision、可信 runtime base 与 pending hint，增加运行配置 cache epoch，并唤醒一次 Full
|
||||
reconcile。旧 round 只有 epoch 仍匹配时才能推进 applied revision；新一轮不得
|
||||
信任 mutation 前缓存的 runtime config。否则 runtime-only mutation 或 Core 成功、
|
||||
SQLite 失败的复合 mutation 可能在 persisted revision 不变时破坏 Patch base 的
|
||||
@@ -491,11 +506,13 @@ response 当作旧 receiver 并静默换一种 mutation contract；出现 404
|
||||
- 超限 Full 稳定返回 413/422，而不是耗尽进程内存；
|
||||
- 并发请求无 deadlock，且 CAS 结果确定。
|
||||
|
||||
Session 测试还必须验证：精确 base/target 使用 touched-instance reconcile；base
|
||||
不匹配、目标 revision 已变化、Full notification 和 Patch backlog 都使用 Full；
|
||||
touched runtime apply 失败不推进 applied revision；删除只作用于本次 delete IDs。
|
||||
运行态 Config Get/Patch/Run/Delete 数量应随 touched instances 增长。为确认运行实例
|
||||
身份而进行的一次 list/meta RPC 可以保留，它不发送或重写所有实例配置。
|
||||
Session 测试还必须验证：连续 Patch 的 Dirty IDs 会合并且保留最早 expected；未知
|
||||
或不匹配的 runtime base、乱序/不连续通知使用 Full；增量 round 读取最新 row；已经
|
||||
删除的 web-owned row 只删除对应 Dirty ID；Full notification 覆盖 Dirty hint；目标
|
||||
revision 已变化或 touched runtime apply 失败时不推进 applied revision；直接 runtime
|
||||
mutation 使 revision 与运行配置 cache 同时失效。运行态 Config Get/Patch/Run/Delete
|
||||
数量应随 touched instances 增长。为确认运行实例身份而进行的一次 list/meta RPC
|
||||
可以保留，它不发送或重写所有实例配置。
|
||||
|
||||
## 11. Observability
|
||||
|
||||
@@ -523,9 +540,13 @@ Rollout acceptance：
|
||||
|
||||
### 12.1 Session runtime delta apply（已实现）
|
||||
|
||||
Patch commit outcome 已携带 touched IDs。Session 只在 applied revision 正好等于
|
||||
Patch base 时执行 touched-instance reconcile；重启、revision 断链、通知丢失或
|
||||
并发 Patch backlog 都退回 Full。接收端不保存 Patch queue，也不合并 delta。
|
||||
Patch commit outcome 已携带 expected、target 和 transaction 实际 touched IDs。
|
||||
Session 只合并 expected/target 连续的 Dirty IDs，并在每一轮从 SQLite 读取最新
|
||||
target revision 对应的当前 rows；因此正常积压只增加 Dirty set，不需要保留中间
|
||||
revision 的 Patch queue。可信 runtime base 必须等于 Dirty 最早 expected；Patch
|
||||
side effect 失败可保留该 base 重试，通知丢失、乱序、进程重启或新 Session 尚无
|
||||
runtime base 时回退 Full。Full
|
||||
notification、disabled row 或 ownership 无法证明时也回退 Full。
|
||||
|
||||
### 12.2 Chunked Full
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ use dashmap::DashMap;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::config::toml::TomlConfig;
|
||||
use crate::instance::{CoreInstance, CoreInstanceHost};
|
||||
use crate::instance::{CoreInstance, CoreInstanceHost, CoreInstanceState};
|
||||
use crate::process_runtime::CoreProcessRuntime;
|
||||
#[cfg(feature = "web-client")]
|
||||
use crate::{
|
||||
@@ -205,6 +205,39 @@ struct ActiveStopGuard {
|
||||
notifier: Arc<tokio::sync::Notify>,
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct InstanceStateChanges {
|
||||
generation: AtomicUsize,
|
||||
notify: tokio::sync::Notify,
|
||||
}
|
||||
|
||||
impl InstanceStateChanges {
|
||||
fn generation(&self) -> usize {
|
||||
self.generation.load(Ordering::Acquire)
|
||||
}
|
||||
|
||||
fn mark_changed(&self) {
|
||||
self.generation.fetch_add(1, Ordering::AcqRel);
|
||||
self.notify.notify_waiters();
|
||||
}
|
||||
|
||||
async fn wait_for_change(&self, generation: usize) -> usize {
|
||||
loop {
|
||||
let notified = self.notify.notified();
|
||||
tokio::pin!(notified);
|
||||
// Register the waiter before reading the generation: notify_waiters
|
||||
// does not retain permits, so a change landing between the read
|
||||
// and the await would otherwise be missed until the next change.
|
||||
notified.as_mut().enable();
|
||||
let current = self.generation();
|
||||
if current != generation {
|
||||
return current;
|
||||
}
|
||||
notified.await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for ActiveStopGuard {
|
||||
fn drop(&mut self) {
|
||||
let previous = self.active_stops.fetch_sub(1, Ordering::AcqRel);
|
||||
@@ -224,6 +257,7 @@ pub struct InstanceManager<F: InstanceFactory> {
|
||||
mutation_lock: Arc<tokio::sync::Mutex<()>>,
|
||||
runtime_handle: Option<tokio::runtime::Handle>,
|
||||
active_stops: Arc<AtomicUsize>,
|
||||
instance_state_changes: Arc<InstanceStateChanges>,
|
||||
}
|
||||
|
||||
impl<F: InstanceFactory> InstanceManager<F> {
|
||||
@@ -238,6 +272,7 @@ impl<F: InstanceFactory> InstanceManager<F> {
|
||||
mutation_lock: Arc::new(tokio::sync::Mutex::new(())),
|
||||
runtime_handle,
|
||||
active_stops: Arc::new(AtomicUsize::new(0)),
|
||||
instance_state_changes: Arc::new(InstanceStateChanges::default()),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -311,6 +346,16 @@ impl<F: InstanceFactory> InstanceManager<F> {
|
||||
notifier: self.notifier.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn instance_state_generation(&self) -> usize {
|
||||
self.instance_state_changes.generation()
|
||||
}
|
||||
|
||||
pub(crate) async fn wait_for_instance_state_change(&self, generation: usize) -> usize {
|
||||
self.instance_state_changes
|
||||
.wait_for_change(generation)
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
||||
impl<F: ProcessRuntimeProvider> InstanceManager<F> {
|
||||
@@ -339,10 +384,12 @@ where
|
||||
let instance_id = instance.instance_id();
|
||||
self.config_controls.insert(instance_id, control);
|
||||
let notifier = self.notifier.clone();
|
||||
let instance_state_changes = self.instance_state_changes.clone();
|
||||
runtime.spawn(async move {
|
||||
if let Err(error) = instance.start().await {
|
||||
tracing::error!(%error, %instance_id, "instance failed to start");
|
||||
}
|
||||
instance_state_changes.mark_changed();
|
||||
notifier.notify_one();
|
||||
});
|
||||
Ok(instance_id)
|
||||
@@ -373,6 +420,7 @@ where
|
||||
drop(active_stop);
|
||||
return Ok(self.instance_ids());
|
||||
}
|
||||
self.instance_state_changes.mark_changed();
|
||||
|
||||
runtime
|
||||
.spawn(async move {
|
||||
@@ -403,6 +451,19 @@ where
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub fn failed_instance_ids(&self) -> Vec<Uuid> {
|
||||
self.list()
|
||||
.into_iter()
|
||||
.filter(|instance| {
|
||||
instance.state() == CoreInstanceState::Stopped
|
||||
&& instance
|
||||
.latest_error()
|
||||
.is_some_and(|error| !error.trim().is_empty())
|
||||
})
|
||||
.map(|instance| instance.instance_id())
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub fn instance(&self, instance_id: Uuid) -> Option<Arc<CoreInstance<H>>> {
|
||||
self.get(instance_id)
|
||||
}
|
||||
@@ -533,6 +594,39 @@ where
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod instance_state_change_tests {
|
||||
use super::InstanceStateChanges;
|
||||
use std::{sync::Arc, time::Duration};
|
||||
|
||||
#[tokio::test]
|
||||
async fn state_change_wait_observes_existing_and_future_changes() {
|
||||
let changes = Arc::new(InstanceStateChanges::default());
|
||||
|
||||
let observed = changes.generation();
|
||||
changes.mark_changed();
|
||||
assert_eq!(
|
||||
tokio::time::timeout(Duration::from_secs(1), changes.wait_for_change(observed))
|
||||
.await
|
||||
.expect("an existing change must not be lost"),
|
||||
1
|
||||
);
|
||||
|
||||
let observed = changes.generation();
|
||||
let waiter_changes = changes.clone();
|
||||
let waiter = tokio::spawn(async move { waiter_changes.wait_for_change(observed).await });
|
||||
tokio::task::yield_now().await;
|
||||
changes.mark_changed();
|
||||
assert_eq!(
|
||||
tokio::time::timeout(Duration::from_secs(1), waiter)
|
||||
.await
|
||||
.expect("a future change must wake the waiter")
|
||||
.unwrap(),
|
||||
2
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use std::{
|
||||
|
||||
@@ -1395,6 +1395,74 @@ virtual_ip = "10.82.0.2/24"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn manager_reports_only_stopped_instances_with_errors() {
|
||||
use crate::instance::manager::{InstanceFactory, InstanceManager};
|
||||
|
||||
struct StateTestFactory;
|
||||
|
||||
impl InstanceFactory for StateTestFactory {
|
||||
type Instance = CoreInstance<TestHost>;
|
||||
type CreateContext = ();
|
||||
type Error = anyhow::Error;
|
||||
|
||||
fn create(
|
||||
&self,
|
||||
config: TomlConfig,
|
||||
(): Self::CreateContext,
|
||||
) -> Result<Arc<Self::Instance>, Self::Error> {
|
||||
let (packet_sink, _packet_receiver) = tokio::sync::mpsc::channel(16);
|
||||
CoreInstance::from_toml(config, adapters(None, Arc::new(packet_sink)))
|
||||
}
|
||||
}
|
||||
|
||||
fn create_instance(
|
||||
manager: &InstanceManager<StateTestFactory>,
|
||||
name: &str,
|
||||
) -> Arc<CoreInstance<TestHost>> {
|
||||
let config = TomlConfig::new_from_str(&format!("instance_name = \"{name}\"")).unwrap();
|
||||
manager.create(config, ()).unwrap()
|
||||
}
|
||||
|
||||
let manager = InstanceManager::new(StateTestFactory, None);
|
||||
let running = create_instance(&manager, "running");
|
||||
running
|
||||
.latest_error
|
||||
.write()
|
||||
.replace("old startup error".to_owned());
|
||||
running.set_state(CoreInstanceState::Running);
|
||||
|
||||
let starting = create_instance(&manager, "starting");
|
||||
starting
|
||||
.latest_error
|
||||
.write()
|
||||
.replace("old startup error".to_owned());
|
||||
starting.set_state(CoreInstanceState::Starting);
|
||||
|
||||
let stopped_without_error = create_instance(&manager, "stopped-without-error");
|
||||
stopped_without_error.set_state(CoreInstanceState::Stopped);
|
||||
|
||||
let stopped_with_blank_error = create_instance(&manager, "stopped-with-blank-error");
|
||||
stopped_with_blank_error
|
||||
.latest_error
|
||||
.write()
|
||||
.replace(" \n".to_owned());
|
||||
stopped_with_blank_error.set_state(CoreInstanceState::Stopped);
|
||||
|
||||
let failed = create_instance(&manager, "failed");
|
||||
failed
|
||||
.latest_error
|
||||
.write()
|
||||
.replace("startup failed".to_owned());
|
||||
failed.set_state(CoreInstanceState::Stopped);
|
||||
let failed_id = failed.instance_id();
|
||||
|
||||
assert_eq!(manager.failed_instance_ids(), vec![failed_id]);
|
||||
|
||||
manager.delete_network_instances([failed_id]).await.unwrap();
|
||||
assert!(manager.failed_instance_ids().is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn aborting_host_prepare_runs_unified_cleanup() {
|
||||
#[derive(Default)]
|
||||
@@ -1739,6 +1807,142 @@ virtual_ip = "10.82.0.2/24"
|
||||
assert!(instances.instances().is_empty());
|
||||
}
|
||||
|
||||
#[cfg(feature = "web-client")]
|
||||
#[tokio::test]
|
||||
async fn process_management_rpc_collects_only_requested_instances() {
|
||||
use std::{collections::VecDeque, sync::Mutex as StdMutex};
|
||||
|
||||
use crate::{
|
||||
config::toml::TomlConfig,
|
||||
instance::manager::InstanceFactory,
|
||||
management::{InstanceManager, ProcessManagementRpc, UnsupportedConfigFileStorage},
|
||||
};
|
||||
use easytier_proto::{
|
||||
api::manage::{CollectNetworkInfoRequest, WebClientService},
|
||||
rpc_types::controller::BaseController,
|
||||
};
|
||||
|
||||
#[derive(Default)]
|
||||
struct RecordingRuntimeHost {
|
||||
collection_count: AtomicUsize,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl InstanceRuntimeHost for RecordingRuntimeHost {
|
||||
async fn prepare(
|
||||
&self,
|
||||
_packet_plane: Arc<CorePacketPlane>,
|
||||
) -> anyhow::Result<Option<Arc<dyn DhcpIpv4Host>>> {
|
||||
Ok(None)
|
||||
}
|
||||
|
||||
async fn shutdown(&self) {}
|
||||
|
||||
fn management_events(&self) -> Vec<String> {
|
||||
self.collection_count.fetch_add(1, Ordering::Relaxed);
|
||||
Vec::new()
|
||||
}
|
||||
}
|
||||
|
||||
struct RecordingFactory {
|
||||
process_runtime: Arc<CoreProcessRuntime>,
|
||||
runtime_hosts: StdMutex<VecDeque<Arc<RecordingRuntimeHost>>>,
|
||||
}
|
||||
|
||||
impl InstanceFactory for RecordingFactory {
|
||||
type Instance = CoreInstance<TestHost>;
|
||||
type CreateContext = ();
|
||||
type Error = anyhow::Error;
|
||||
|
||||
fn create(
|
||||
&self,
|
||||
config: TomlConfig,
|
||||
(): Self::CreateContext,
|
||||
) -> Result<Arc<Self::Instance>, Self::Error> {
|
||||
let runtime_host = self.runtime_hosts.lock().unwrap().pop_front().unwrap();
|
||||
let (packet_sink, _packet_receiver) = tokio::sync::mpsc::channel(16);
|
||||
let mut adapters = adapters_with_process_runtime(
|
||||
None,
|
||||
Arc::new(packet_sink),
|
||||
self.process_runtime.clone(),
|
||||
);
|
||||
adapters.instance_runtime = runtime_host;
|
||||
CoreInstance::from_toml(config, adapters)
|
||||
}
|
||||
}
|
||||
|
||||
let requested_runtime = Arc::new(RecordingRuntimeHost::default());
|
||||
let unrequested_runtime = Arc::new(RecordingRuntimeHost::default());
|
||||
let instances = Arc::new(InstanceManager::new(
|
||||
RecordingFactory {
|
||||
process_runtime: CoreProcessRuntime::new(),
|
||||
runtime_hosts: StdMutex::new(VecDeque::from([
|
||||
requested_runtime.clone(),
|
||||
unrequested_runtime.clone(),
|
||||
])),
|
||||
},
|
||||
Some(tokio::runtime::Handle::current()),
|
||||
));
|
||||
let requested_id = uuid::Uuid::new_v4();
|
||||
let unrequested_id = uuid::Uuid::new_v4();
|
||||
for instance_id in [requested_id, unrequested_id] {
|
||||
let config = TomlConfig::default();
|
||||
config.set_id(instance_id);
|
||||
config.set_listeners(Vec::new());
|
||||
instances
|
||||
.create(config, ())
|
||||
.unwrap()
|
||||
.set_state(CoreInstanceState::Running);
|
||||
}
|
||||
let rpc = ProcessManagementRpc::<RecordingFactory>::new(
|
||||
instances,
|
||||
Arc::new(()),
|
||||
Arc::new(UnsupportedConfigFileStorage),
|
||||
);
|
||||
|
||||
let response = rpc
|
||||
.collect_network_info(
|
||||
BaseController::default(),
|
||||
CollectNetworkInfoRequest {
|
||||
inst_ids: vec![
|
||||
requested_id.into(),
|
||||
requested_id.into(),
|
||||
uuid::Uuid::new_v4().into(),
|
||||
],
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let info = response.info.unwrap().map;
|
||||
assert_eq!(info.len(), 1);
|
||||
assert!(info.contains_key(&requested_id.to_string()));
|
||||
assert_eq!(
|
||||
requested_runtime.collection_count.load(Ordering::Relaxed),
|
||||
1
|
||||
);
|
||||
assert_eq!(
|
||||
unrequested_runtime.collection_count.load(Ordering::Relaxed),
|
||||
0
|
||||
);
|
||||
|
||||
let response = rpc
|
||||
.collect_network_info(
|
||||
BaseController::default(),
|
||||
CollectNetworkInfoRequest::default(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(response.info.unwrap().map.len(), 2);
|
||||
assert_eq!(
|
||||
requested_runtime.collection_count.load(Ordering::Relaxed),
|
||||
2
|
||||
);
|
||||
assert_eq!(
|
||||
unrequested_runtime.collection_count.load(Ordering::Relaxed),
|
||||
1
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(feature = "management")]
|
||||
#[tokio::test]
|
||||
async fn owned_selection_and_cleanup_share_the_canonical_transaction() {
|
||||
|
||||
@@ -28,7 +28,7 @@ use crate::{
|
||||
|
||||
use super::{
|
||||
ConfigFileControl, ConfigFilePermission, InstanceManager, config_source_from_rpc,
|
||||
config_source_to_rpc,
|
||||
config_source_to_rpc, network_instance_running_info,
|
||||
};
|
||||
|
||||
#[async_trait::async_trait]
|
||||
@@ -633,17 +633,29 @@ where
|
||||
let included = request
|
||||
.inst_ids
|
||||
.into_iter()
|
||||
.map(|id| uuid::Uuid::from(id).to_string())
|
||||
.map(uuid::Uuid::from)
|
||||
.collect::<HashSet<_>>();
|
||||
let map = self
|
||||
.management
|
||||
.instances
|
||||
.collect_network_infos()
|
||||
.await?
|
||||
.into_iter()
|
||||
.map(|(id, info)| (id.to_string(), info))
|
||||
.filter(|(id, _)| included.is_empty() || included.contains(id))
|
||||
.collect();
|
||||
let map = if included.is_empty() {
|
||||
self.management
|
||||
.instances
|
||||
.collect_network_infos()
|
||||
.await?
|
||||
.into_iter()
|
||||
.map(|(id, info)| (id.to_string(), info))
|
||||
.collect()
|
||||
} else {
|
||||
let mut map = std::collections::BTreeMap::new();
|
||||
for instance_id in included {
|
||||
let Some(instance) = self.management.instances.instance(instance_id) else {
|
||||
continue;
|
||||
};
|
||||
map.insert(
|
||||
instance_id.to_string(),
|
||||
network_instance_running_info(instance.as_ref()).await?,
|
||||
);
|
||||
}
|
||||
map
|
||||
};
|
||||
Ok(CollectNetworkInfoResponse {
|
||||
info: Some(NetworkInstanceRunningInfoMap { map }),
|
||||
})
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
use std::collections::HashSet;
|
||||
use std::sync::{
|
||||
Arc, Weak,
|
||||
atomic::{AtomicBool, Ordering},
|
||||
@@ -7,7 +8,7 @@ use async_trait::async_trait;
|
||||
use easytier_proto::{
|
||||
rpc_types::controller::BaseController,
|
||||
web::{
|
||||
DeviceOsInfo, GetFeatureRequest, GetFeatureResponse, HeartbeatRequest,
|
||||
DeviceOsInfo, GetFeatureRequest, GetFeatureResponse, HeartbeatRequest, HeartbeatResponse,
|
||||
WebServerServiceClientFactory,
|
||||
},
|
||||
};
|
||||
@@ -33,6 +34,64 @@ const RETRY_INTERVAL: std::time::Duration = std::time::Duration::from_secs(1);
|
||||
// Keep retry ownership in this loop when transport or protocol handshakes stall.
|
||||
const CONNECT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(20);
|
||||
const FEATURE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(3);
|
||||
const DEFAULT_HEARTBEAT_INTERVAL_MS: u32 = 3_500;
|
||||
const DEFAULT_HEARTBEAT_TIMEOUT_MS: u32 = 15_000;
|
||||
const MIN_HEARTBEAT_INTERVAL_MS: u32 = 1_000;
|
||||
const MAX_HEARTBEAT_INTERVAL_MS: u32 = 60_000;
|
||||
const MIN_HEARTBEAT_TIMEOUT_MS: u32 = 5_000;
|
||||
const MAX_HEARTBEAT_TIMEOUT_MS: u32 = 120_000;
|
||||
const MIN_HEARTBEAT_TIMEOUT_MARGIN_MS: u32 = 5_000;
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
struct HeartbeatPolicy {
|
||||
interval: std::time::Duration,
|
||||
timeout_ms: i32,
|
||||
}
|
||||
|
||||
impl Default for HeartbeatPolicy {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
interval: std::time::Duration::from_millis(DEFAULT_HEARTBEAT_INTERVAL_MS.into()),
|
||||
timeout_ms: DEFAULT_HEARTBEAT_TIMEOUT_MS as i32,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl HeartbeatPolicy {
|
||||
fn from_response(response: &HeartbeatResponse) -> (Self, bool) {
|
||||
let requested_interval = response
|
||||
.heartbeat_interval_ms
|
||||
.unwrap_or(DEFAULT_HEARTBEAT_INTERVAL_MS);
|
||||
let requested_timeout = response
|
||||
.heartbeat_timeout_ms
|
||||
.unwrap_or(DEFAULT_HEARTBEAT_TIMEOUT_MS);
|
||||
let interval_ms =
|
||||
requested_interval.clamp(MIN_HEARTBEAT_INTERVAL_MS, MAX_HEARTBEAT_INTERVAL_MS);
|
||||
let timeout_ms = requested_timeout
|
||||
.clamp(MIN_HEARTBEAT_TIMEOUT_MS, MAX_HEARTBEAT_TIMEOUT_MS)
|
||||
.max(interval_ms.saturating_add(MIN_HEARTBEAT_TIMEOUT_MARGIN_MS));
|
||||
(
|
||||
Self {
|
||||
interval: std::time::Duration::from_millis(interval_ms.into()),
|
||||
timeout_ms: timeout_ms as i32,
|
||||
},
|
||||
interval_ms != requested_interval || timeout_ms != requested_timeout,
|
||||
)
|
||||
}
|
||||
|
||||
fn controller(self) -> BaseController {
|
||||
BaseController {
|
||||
timeout_ms: self.timeout_ms,
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
fn remaining_interval(self, elapsed: std::time::Duration) -> Option<std::time::Duration> {
|
||||
self.interval
|
||||
.checked_sub(elapsed)
|
||||
.filter(|delay| !delay.is_zero())
|
||||
}
|
||||
}
|
||||
|
||||
async fn connect_config_server(
|
||||
connector: &dyn TunnelDialer,
|
||||
@@ -100,6 +159,16 @@ pub(crate) trait WebClientBackend: Send + Sync + 'static {
|
||||
fn register(&self, registry: &ServiceRegistry);
|
||||
|
||||
async fn instance_ids(&self) -> anyhow::Result<Vec<uuid::Uuid>>;
|
||||
|
||||
fn failed_instance_ids(&self) -> Vec<uuid::Uuid>;
|
||||
|
||||
fn instance_state_generation(&self) -> usize {
|
||||
0
|
||||
}
|
||||
|
||||
async fn wait_for_instance_state_change(&self, _generation: usize) -> usize {
|
||||
std::future::pending().await
|
||||
}
|
||||
}
|
||||
|
||||
struct NativeWebClientBackend<F>
|
||||
@@ -141,11 +210,26 @@ where
|
||||
async fn instance_ids(&self) -> anyhow::Result<Vec<uuid::Uuid>> {
|
||||
Ok(self.instances.instance_ids())
|
||||
}
|
||||
|
||||
fn failed_instance_ids(&self) -> Vec<uuid::Uuid> {
|
||||
self.instances.failed_instance_ids()
|
||||
}
|
||||
|
||||
fn instance_state_generation(&self) -> usize {
|
||||
self.instances.instance_state_generation()
|
||||
}
|
||||
|
||||
async fn wait_for_instance_state_change(&self, generation: usize) -> usize {
|
||||
self.instances
|
||||
.wait_for_instance_state_change(generation)
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
||||
struct WebClientController {
|
||||
config: WebClientConfig,
|
||||
backend: Arc<dyn WebClientBackend>,
|
||||
runtime_id: uuid::Uuid,
|
||||
}
|
||||
|
||||
/// Portable config-server client. Hosts only supply identity and adapters.
|
||||
@@ -201,7 +285,11 @@ impl<F> WebClient<F> {
|
||||
backend: Arc<dyn WebClientBackend>,
|
||||
manager_guard: Option<DaemonGuard>,
|
||||
) -> Self {
|
||||
let controller = Arc::new(WebClientController { config, backend });
|
||||
let controller = Arc::new(WebClientController {
|
||||
config,
|
||||
backend,
|
||||
runtime_id: uuid::Uuid::new_v4(),
|
||||
});
|
||||
let connected = Arc::new(AtomicBool::new(false));
|
||||
let tasks = AbortOnDropHandle::new(tokio::spawn(web_client_routine(
|
||||
controller.clone(),
|
||||
@@ -312,6 +400,59 @@ struct WebClientSession {
|
||||
tasks: Mutex<JoinSet<()>>,
|
||||
}
|
||||
|
||||
fn running_instances_for_heartbeat(
|
||||
instance_ids: Vec<uuid::Uuid>,
|
||||
failed_instance_ids: &[uuid::Uuid],
|
||||
) -> Vec<uuid::Uuid> {
|
||||
let failed_instance_ids: HashSet<_> = failed_instance_ids.iter().copied().collect();
|
||||
instance_ids
|
||||
.into_iter()
|
||||
.filter(|instance_id| !failed_instance_ids.contains(instance_id))
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn build_heartbeat_request(
|
||||
config: &WebClientConfig,
|
||||
runtime_id: uuid::Uuid,
|
||||
running_network_instances: Vec<uuid::Uuid>,
|
||||
failed_network_instances: Vec<uuid::Uuid>,
|
||||
) -> HeartbeatRequest {
|
||||
HeartbeatRequest {
|
||||
machine_id: Some(config.machine_id.into()),
|
||||
inst_id: Some(runtime_id.into()),
|
||||
user_token: config.token.clone(),
|
||||
easytier_version: config.easytier_version.clone(),
|
||||
hostname: config.hostname.clone(),
|
||||
report_time: chrono::Local::now().to_rfc3339(),
|
||||
device_os: Some(config.device_os.clone()),
|
||||
support_config_source: true,
|
||||
running_network_instances: running_network_instances
|
||||
.into_iter()
|
||||
.map(Into::into)
|
||||
.collect(),
|
||||
failed_network_instances: failed_network_instances
|
||||
.into_iter()
|
||||
.map(Into::into)
|
||||
.collect(),
|
||||
support_heartbeat_policy: true,
|
||||
}
|
||||
}
|
||||
|
||||
async fn wait_for_next_heartbeat(
|
||||
backend: &dyn WebClientBackend,
|
||||
observed_generation: usize,
|
||||
policy: HeartbeatPolicy,
|
||||
elapsed: std::time::Duration,
|
||||
) {
|
||||
let Some(delay) = policy.remaining_interval(elapsed) else {
|
||||
return;
|
||||
};
|
||||
tokio::select! {
|
||||
_ = time::sleep(delay) => {}
|
||||
_ = backend.wait_for_instance_state_change(observed_generation) => {}
|
||||
}
|
||||
}
|
||||
|
||||
impl WebClientSession {
|
||||
fn new(tunnel: Box<dyn Tunnel>, controller: Arc<WebClientController>) -> Self {
|
||||
let rpc = BidirectRpcManager::new();
|
||||
@@ -339,46 +480,60 @@ impl WebClientSession {
|
||||
tasks: &mut JoinSet<()>,
|
||||
) {
|
||||
let controller = controller.upgrade().expect("web client controller");
|
||||
let machine_id = controller.config.machine_id;
|
||||
let session_id = uuid::Uuid::new_v4();
|
||||
let token = controller.config.token.clone();
|
||||
let hostname = controller.config.hostname.clone();
|
||||
let device_os = controller.config.device_os.clone();
|
||||
let easytier_version = controller.config.easytier_version.clone();
|
||||
let controller = Arc::downgrade(&controller);
|
||||
let client = rpc
|
||||
.rpc_client()
|
||||
.scoped_client::<WebServerServiceClientFactory<BaseController>>(1, 1, String::new());
|
||||
let mut tick = time::interval(std::time::Duration::from_secs(1));
|
||||
|
||||
tasks.spawn(async move {
|
||||
let mut heartbeat_policy = HeartbeatPolicy::default();
|
||||
loop {
|
||||
tick.tick().await;
|
||||
let heartbeat_started_at = std::time::Instant::now();
|
||||
let Some(controller) = controller.upgrade() else {
|
||||
break;
|
||||
};
|
||||
let observed_generation = controller.backend.instance_state_generation();
|
||||
let failed_network_instances = controller.backend.failed_instance_ids();
|
||||
let running_network_instances = match controller.backend.instance_ids().await {
|
||||
Ok(instance_ids) => instance_ids.into_iter().map(Into::into).collect(),
|
||||
Ok(instance_ids) => {
|
||||
running_instances_for_heartbeat(instance_ids, &failed_network_instances)
|
||||
}
|
||||
Err(error) => {
|
||||
tracing::error!(%error, "failed to list config-server instances");
|
||||
break;
|
||||
}
|
||||
};
|
||||
let request = HeartbeatRequest {
|
||||
machine_id: Some(machine_id.into()),
|
||||
inst_id: Some(session_id.into()),
|
||||
user_token: token.clone(),
|
||||
easytier_version: easytier_version.clone(),
|
||||
hostname: hostname.clone(),
|
||||
report_time: chrono::Local::now().to_rfc3339(),
|
||||
device_os: Some(device_os.clone()),
|
||||
support_config_source: true,
|
||||
let request = build_heartbeat_request(
|
||||
&controller.config,
|
||||
controller.runtime_id,
|
||||
running_network_instances,
|
||||
};
|
||||
failed_network_instances,
|
||||
);
|
||||
|
||||
match client.heartbeat(BaseController::default(), request).await {
|
||||
match client
|
||||
.heartbeat(heartbeat_policy.controller(), request)
|
||||
.await
|
||||
{
|
||||
Ok(response) => {
|
||||
tracing::debug!(?response, "config-server heartbeat response");
|
||||
let (next_policy, adjusted) = HeartbeatPolicy::from_response(&response);
|
||||
if adjusted {
|
||||
tracing::warn!(
|
||||
requested_interval_ms = ?response.heartbeat_interval_ms,
|
||||
requested_timeout_ms = ?response.heartbeat_timeout_ms,
|
||||
applied_interval_ms = next_policy.interval.as_millis(),
|
||||
applied_timeout_ms = next_policy.timeout_ms,
|
||||
"config-server heartbeat policy was outside safe bounds"
|
||||
);
|
||||
}
|
||||
heartbeat_policy = next_policy;
|
||||
wait_for_next_heartbeat(
|
||||
controller.backend.as_ref(),
|
||||
observed_generation,
|
||||
heartbeat_policy,
|
||||
heartbeat_started_at.elapsed(),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
Err(error) => {
|
||||
tracing::error!(?error, "config-server heartbeat failed");
|
||||
@@ -430,6 +585,25 @@ mod tests {
|
||||
attempts: AtomicUsize,
|
||||
}
|
||||
|
||||
struct ImmediateStateChangeBackend;
|
||||
|
||||
#[async_trait]
|
||||
impl WebClientBackend for ImmediateStateChangeBackend {
|
||||
fn register(&self, _registry: &ServiceRegistry) {}
|
||||
|
||||
async fn instance_ids(&self) -> anyhow::Result<Vec<uuid::Uuid>> {
|
||||
Ok(Vec::new())
|
||||
}
|
||||
|
||||
fn failed_instance_ids(&self) -> Vec<uuid::Uuid> {
|
||||
Vec::new()
|
||||
}
|
||||
|
||||
async fn wait_for_instance_state_change(&self, generation: usize) -> usize {
|
||||
generation.wrapping_add(1)
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl TunnelDialer for StalledThenReadyDialer {
|
||||
async fn connect(&self) -> anyhow::Result<Box<dyn Tunnel>> {
|
||||
@@ -446,6 +620,20 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn heartbeat_hides_failed_instances_from_the_running_list() {
|
||||
let running = uuid::Uuid::new_v4();
|
||||
let failed = uuid::Uuid::new_v4();
|
||||
let stopped_clean = uuid::Uuid::new_v4();
|
||||
let instance_ids = vec![running, failed, stopped_clean];
|
||||
let failed_instance_ids = vec![failed];
|
||||
|
||||
let reported = running_instances_for_heartbeat(instance_ids.clone(), &failed_instance_ids);
|
||||
|
||||
assert_eq!(reported, vec![running, stopped_clean]);
|
||||
assert!(running_instances_for_heartbeat(instance_ids, &[]).len() == 3);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stalled_connection_attempt_times_out_and_allows_redial() {
|
||||
let connector = StalledThenReadyDialer {
|
||||
@@ -463,6 +651,26 @@ mod tests {
|
||||
assert_eq!(connector.attempts.load(Ordering::Relaxed), 2);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn instance_state_change_interrupts_a_long_heartbeat_interval() {
|
||||
let policy = HeartbeatPolicy {
|
||||
interval: std::time::Duration::from_secs(60),
|
||||
timeout_ms: 65_000,
|
||||
};
|
||||
|
||||
tokio::time::timeout(
|
||||
std::time::Duration::from_secs(1),
|
||||
wait_for_next_heartbeat(
|
||||
&ImmediateStateChangeBackend,
|
||||
0,
|
||||
policy,
|
||||
std::time::Duration::ZERO,
|
||||
),
|
||||
)
|
||||
.await
|
||||
.expect("instance state change must wake heartbeat before its interval");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn endpoint_normalizes_non_websocket_paths() {
|
||||
let endpoint =
|
||||
@@ -502,4 +710,87 @@ mod tests {
|
||||
fn endpoint_rejects_an_empty_token() {
|
||||
assert!(ConfigServerEndpoint::parse("udp://example.com", |_| true).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn heartbeat_request_carries_registered_and_failed_instance_ids() {
|
||||
let runtime_id = uuid::Uuid::new_v4();
|
||||
let registered = uuid::Uuid::new_v4();
|
||||
let failed = uuid::Uuid::new_v4();
|
||||
let request = build_heartbeat_request(
|
||||
&WebClientConfig {
|
||||
token: "token".to_owned(),
|
||||
machine_id: uuid::Uuid::new_v4(),
|
||||
hostname: "host".to_owned(),
|
||||
device_os: DeviceOsInfo::default(),
|
||||
easytier_version: "test-version".to_owned(),
|
||||
secure_mode: false,
|
||||
},
|
||||
runtime_id,
|
||||
vec![registered],
|
||||
vec![failed],
|
||||
);
|
||||
|
||||
assert_eq!(request.inst_id.map(uuid::Uuid::from), Some(runtime_id));
|
||||
assert_eq!(
|
||||
request
|
||||
.running_network_instances
|
||||
.into_iter()
|
||||
.map(uuid::Uuid::from)
|
||||
.collect::<Vec<_>>(),
|
||||
vec![registered]
|
||||
);
|
||||
assert_eq!(
|
||||
request
|
||||
.failed_network_instances
|
||||
.into_iter()
|
||||
.map(uuid::Uuid::from)
|
||||
.collect::<Vec<_>>(),
|
||||
vec![failed]
|
||||
);
|
||||
assert!(request.support_heartbeat_policy);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn heartbeat_policy_uses_safe_defaults_for_legacy_servers() {
|
||||
let (policy, adjusted) = HeartbeatPolicy::from_response(&HeartbeatResponse::default());
|
||||
|
||||
assert!(!adjusted);
|
||||
assert_eq!(
|
||||
policy.interval,
|
||||
std::time::Duration::from_millis(DEFAULT_HEARTBEAT_INTERVAL_MS.into())
|
||||
);
|
||||
assert_eq!(policy.timeout_ms, DEFAULT_HEARTBEAT_TIMEOUT_MS as i32);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn heartbeat_policy_clamps_server_values_and_preserves_timeout_margin() {
|
||||
let (minimum, adjusted) = HeartbeatPolicy::from_response(&HeartbeatResponse {
|
||||
heartbeat_interval_ms: Some(1),
|
||||
heartbeat_timeout_ms: Some(1),
|
||||
});
|
||||
assert!(adjusted);
|
||||
assert_eq!(
|
||||
minimum.interval,
|
||||
std::time::Duration::from_millis(MIN_HEARTBEAT_INTERVAL_MS.into())
|
||||
);
|
||||
assert_eq!(minimum.timeout_ms, 6_000);
|
||||
|
||||
let (maximum, adjusted) = HeartbeatPolicy::from_response(&HeartbeatResponse {
|
||||
heartbeat_interval_ms: Some(u32::MAX),
|
||||
heartbeat_timeout_ms: Some(u32::MAX),
|
||||
});
|
||||
assert!(adjusted);
|
||||
assert_eq!(
|
||||
maximum.interval,
|
||||
std::time::Duration::from_millis(MAX_HEARTBEAT_INTERVAL_MS.into())
|
||||
);
|
||||
assert_eq!(maximum.timeout_ms, MAX_HEARTBEAT_TIMEOUT_MS as i32);
|
||||
|
||||
let (margin, adjusted) = HeartbeatPolicy::from_response(&HeartbeatResponse {
|
||||
heartbeat_interval_ms: Some(60_000),
|
||||
heartbeat_timeout_ms: Some(5_000),
|
||||
});
|
||||
assert!(adjusted);
|
||||
assert_eq!(margin.timeout_ms, 65_000);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -265,6 +265,10 @@ impl WebClientBackend for WasiWebClientBackend {
|
||||
.await?;
|
||||
Ok(response.inst_ids.into_iter().map(Into::into).collect())
|
||||
}
|
||||
|
||||
fn failed_instance_ids(&self) -> Vec<uuid::Uuid> {
|
||||
Vec::new()
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) struct WasiWebClientRuntime {
|
||||
|
||||
@@ -93,6 +93,8 @@ type HeartbeatRequest struct {
|
||||
RunningNetworkInstances []*common.UUID `protobuf:"bytes,7,rep,name=running_network_instances,json=runningNetworkInstances,proto3" json:"running_network_instances,omitempty"`
|
||||
DeviceOs *DeviceOsInfo `protobuf:"bytes,8,opt,name=device_os,json=deviceOs,proto3" json:"device_os,omitempty"`
|
||||
SupportConfigSource bool `protobuf:"varint,9,opt,name=support_config_source,json=supportConfigSource,proto3" json:"support_config_source,omitempty"`
|
||||
FailedNetworkInstances []*common.UUID `protobuf:"bytes,10,rep,name=failed_network_instances,json=failedNetworkInstances,proto3" json:"failed_network_instances,omitempty"`
|
||||
SupportHeartbeatPolicy bool `protobuf:"varint,11,opt,name=support_heartbeat_policy,json=supportHeartbeatPolicy,proto3" json:"support_heartbeat_policy,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -190,10 +192,26 @@ func (x *HeartbeatRequest) GetSupportConfigSource() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *HeartbeatRequest) GetFailedNetworkInstances() []*common.UUID {
|
||||
if x != nil {
|
||||
return x.FailedNetworkInstances
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *HeartbeatRequest) GetSupportHeartbeatPolicy() bool {
|
||||
if x != nil {
|
||||
return x.SupportHeartbeatPolicy
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
type HeartbeatResponse struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
HeartbeatIntervalMs *uint32 `protobuf:"varint,1,opt,name=heartbeat_interval_ms,json=heartbeatIntervalMs,proto3,oneof" json:"heartbeat_interval_ms,omitempty"`
|
||||
HeartbeatTimeoutMs *uint32 `protobuf:"varint,2,opt,name=heartbeat_timeout_ms,json=heartbeatTimeoutMs,proto3,oneof" json:"heartbeat_timeout_ms,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *HeartbeatResponse) Reset() {
|
||||
@@ -226,6 +244,20 @@ func (*HeartbeatResponse) Descriptor() ([]byte, []int) {
|
||||
return file_web_proto_rawDescGZIP(), []int{2}
|
||||
}
|
||||
|
||||
func (x *HeartbeatResponse) GetHeartbeatIntervalMs() uint32 {
|
||||
if x != nil && x.HeartbeatIntervalMs != nil {
|
||||
return *x.HeartbeatIntervalMs
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *HeartbeatResponse) GetHeartbeatTimeoutMs() uint32 {
|
||||
if x != nil && x.HeartbeatTimeoutMs != nil {
|
||||
return *x.HeartbeatTimeoutMs
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
type GetFeatureRequest struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
@@ -314,7 +346,7 @@ const file_web_proto_rawDesc = "" +
|
||||
"\fDeviceOsInfo\x12\x17\n" +
|
||||
"\aos_type\x18\x01 \x01(\tR\x06osType\x12\x18\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\x12\"\n" +
|
||||
"\fdistribution\x18\x03 \x01(\tR\fdistribution\"\x9b\x03\n" +
|
||||
"\fdistribution\x18\x03 \x01(\tR\fdistribution\"\x9d\x04\n" +
|
||||
"\x10HeartbeatRequest\x12+\n" +
|
||||
"\n" +
|
||||
"machine_id\x18\x01 \x01(\v2\f.common.UUIDR\tmachineId\x12%\n" +
|
||||
@@ -327,8 +359,15 @@ const file_web_proto_rawDesc = "" +
|
||||
"\bhostname\x18\x06 \x01(\tR\bhostname\x12H\n" +
|
||||
"\x19running_network_instances\x18\a \x03(\v2\f.common.UUIDR\x17runningNetworkInstances\x12.\n" +
|
||||
"\tdevice_os\x18\b \x01(\v2\x11.web.DeviceOsInfoR\bdeviceOs\x122\n" +
|
||||
"\x15support_config_source\x18\t \x01(\bR\x13supportConfigSource\"\x13\n" +
|
||||
"\x11HeartbeatResponse\"\x13\n" +
|
||||
"\x15support_config_source\x18\t \x01(\bR\x13supportConfigSource\x12F\n" +
|
||||
"\x18failed_network_instances\x18\n" +
|
||||
" \x03(\v2\f.common.UUIDR\x16failedNetworkInstances\x128\n" +
|
||||
"\x18support_heartbeat_policy\x18\v \x01(\bR\x16supportHeartbeatPolicy\"\xb6\x01\n" +
|
||||
"\x11HeartbeatResponse\x127\n" +
|
||||
"\x15heartbeat_interval_ms\x18\x01 \x01(\rH\x00R\x13heartbeatIntervalMs\x88\x01\x01\x125\n" +
|
||||
"\x14heartbeat_timeout_ms\x18\x02 \x01(\rH\x01R\x12heartbeatTimeoutMs\x88\x01\x01B\x18\n" +
|
||||
"\x16_heartbeat_interval_msB\x17\n" +
|
||||
"\x15_heartbeat_timeout_ms\"\x13\n" +
|
||||
"\x11GetFeatureRequest\"C\n" +
|
||||
"\x12GetFeatureResponse\x12-\n" +
|
||||
"\x12support_encryption\x18\x01 \x01(\bR\x11supportEncryption2\x8d\x01\n" +
|
||||
@@ -363,15 +402,16 @@ var file_web_proto_depIdxs = []int32{
|
||||
5, // 1: web.HeartbeatRequest.inst_id:type_name -> common.UUID
|
||||
5, // 2: web.HeartbeatRequest.running_network_instances:type_name -> common.UUID
|
||||
0, // 3: web.HeartbeatRequest.device_os:type_name -> web.DeviceOsInfo
|
||||
1, // 4: web.WebServerService.Heartbeat:input_type -> web.HeartbeatRequest
|
||||
3, // 5: web.WebServerService.GetFeature:input_type -> web.GetFeatureRequest
|
||||
2, // 6: web.WebServerService.Heartbeat:output_type -> web.HeartbeatResponse
|
||||
4, // 7: web.WebServerService.GetFeature:output_type -> web.GetFeatureResponse
|
||||
6, // [6:8] is the sub-list for method output_type
|
||||
4, // [4:6] is the sub-list for method input_type
|
||||
4, // [4:4] is the sub-list for extension type_name
|
||||
4, // [4:4] is the sub-list for extension extendee
|
||||
0, // [0:4] is the sub-list for field type_name
|
||||
5, // 4: web.HeartbeatRequest.failed_network_instances:type_name -> common.UUID
|
||||
1, // 5: web.WebServerService.Heartbeat:input_type -> web.HeartbeatRequest
|
||||
3, // 6: web.WebServerService.GetFeature:input_type -> web.GetFeatureRequest
|
||||
2, // 7: web.WebServerService.Heartbeat:output_type -> web.HeartbeatResponse
|
||||
4, // 8: web.WebServerService.GetFeature:output_type -> web.GetFeatureResponse
|
||||
7, // [7:9] is the sub-list for method output_type
|
||||
5, // [5:7] is the sub-list for method input_type
|
||||
5, // [5:5] is the sub-list for extension type_name
|
||||
5, // [5:5] is the sub-list for extension extendee
|
||||
0, // [0:5] is the sub-list for field type_name
|
||||
}
|
||||
|
||||
func init() { file_web_proto_init() }
|
||||
@@ -379,6 +419,7 @@ func file_web_proto_init() {
|
||||
if File_web_proto != nil {
|
||||
return
|
||||
}
|
||||
file_web_proto_msgTypes[2].OneofWrappers = []any{}
|
||||
type x struct{}
|
||||
out := protoimpl.TypeBuilder{
|
||||
File: protoimpl.DescBuilder{
|
||||
|
||||
@@ -22,9 +22,14 @@ message HeartbeatRequest {
|
||||
repeated common.UUID running_network_instances = 7;
|
||||
DeviceOsInfo device_os = 8;
|
||||
bool support_config_source = 9;
|
||||
repeated common.UUID failed_network_instances = 10;
|
||||
bool support_heartbeat_policy = 11;
|
||||
}
|
||||
|
||||
message HeartbeatResponse {}
|
||||
message HeartbeatResponse {
|
||||
optional uint32 heartbeat_interval_ms = 1;
|
||||
optional uint32 heartbeat_timeout_ms = 2;
|
||||
}
|
||||
|
||||
message GetFeatureRequest {}
|
||||
|
||||
|
||||
@@ -41,8 +41,11 @@ cli:
|
||||
en: "The path to the GeoIP2 database file, used to lookup the location of the client, default is the embedded file (only country information) , recommend https://github.com/P3TERX/GeoLite.mmdb"
|
||||
zh-CN: "GeoIP2 数据库文件路径，用于查找客户端的位置，默认为嵌入文件（仅国家信息），推荐 https://github.com/P3TERX/GeoLite.mmdb"
|
||||
heartbeat_min_response_ms:
|
||||
en: "Minimum response time for config-server heartbeat RPCs in milliseconds, default is 0"
|
||||
zh-CN: "配置服务心跳 RPC 的最短响应时间，单位毫秒，默认为 0"
|
||||
en: "Config-server heartbeat interval in milliseconds, default is 3500"
|
||||
zh-CN: "配置服务心跳周期，单位毫秒，默认为 3500"
|
||||
heartbeat_timeout_ms:
|
||||
en: "Config-server heartbeat RPC timeout in milliseconds, default is 15000"
|
||||
zh-CN: "配置服务心跳 RPC 超时时间，单位毫秒，默认为 15000"
|
||||
disable_registration:
|
||||
en: "Disable user registration"
|
||||
zh-CN: "禁用用户注册"
|
||||
|
||||
@@ -3,15 +3,20 @@ mod runtime_reconcile;
|
||||
pub mod session;
|
||||
pub mod storage;
|
||||
|
||||
use std::sync::{
|
||||
Arc,
|
||||
atomic::{AtomicU32, AtomicU64, Ordering},
|
||||
};
|
||||
use std::time::Duration;
|
||||
use std::{
|
||||
collections::HashSet,
|
||||
sync::{
|
||||
Arc,
|
||||
atomic::{AtomicU32, AtomicU64, Ordering},
|
||||
},
|
||||
};
|
||||
|
||||
use dashmap::DashMap;
|
||||
use easytier::proto::{
|
||||
api::manage::WebClientService, rpc_types::controller::BaseController, web::HeartbeatRequest,
|
||||
api::manage::WebClientService,
|
||||
rpc_types::controller::BaseController,
|
||||
web::{HeartbeatRequest, HeartbeatResponse},
|
||||
};
|
||||
use easytier_core::{
|
||||
management::remote_client::{self, RemoteClientManager},
|
||||
@@ -19,7 +24,7 @@ use easytier_core::{
|
||||
tunnel::{Tunnel, web_security},
|
||||
};
|
||||
use maxminddb::geoip2;
|
||||
use session::{Location, ManagedConfigRevisionDelta, Session};
|
||||
use session::{Location, ManagedConfigPersistedChange, Session};
|
||||
use storage::{Storage, StorageToken};
|
||||
|
||||
use crate::FeatureFlags;
|
||||
@@ -30,6 +35,67 @@ use crate::db::{Db, UserIdInDb, entity::user_running_network_configs};
|
||||
|
||||
pub(crate) use managed_config::ManagedConfigError;
|
||||
|
||||
const DEFAULT_HEARTBEAT_INTERVAL: Duration = Duration::from_millis(3_500);
|
||||
const DEFAULT_HEARTBEAT_TIMEOUT: Duration = Duration::from_secs(15);
|
||||
const MIN_HEARTBEAT_INTERVAL: Duration = Duration::from_secs(1);
|
||||
const MAX_HEARTBEAT_INTERVAL: Duration = Duration::from_secs(60);
|
||||
const MIN_HEARTBEAT_TIMEOUT: Duration = Duration::from_secs(5);
|
||||
const MAX_HEARTBEAT_TIMEOUT: Duration = Duration::from_secs(120);
|
||||
const HEARTBEAT_TIMEOUT_MARGIN: Duration = Duration::from_secs(5);
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub(crate) struct HeartbeatPolicy {
|
||||
interval: Duration,
|
||||
timeout: Duration,
|
||||
}
|
||||
|
||||
impl Default for HeartbeatPolicy {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
interval: DEFAULT_HEARTBEAT_INTERVAL,
|
||||
timeout: DEFAULT_HEARTBEAT_TIMEOUT,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl HeartbeatPolicy {
|
||||
pub(crate) fn from_millis(interval_ms: u64, timeout_ms: u64) -> anyhow::Result<Self> {
|
||||
let interval = if interval_ms == 0 {
|
||||
DEFAULT_HEARTBEAT_INTERVAL
|
||||
} else {
|
||||
Duration::from_millis(interval_ms)
|
||||
};
|
||||
let timeout = Duration::from_millis(timeout_ms);
|
||||
if !(MIN_HEARTBEAT_INTERVAL..=MAX_HEARTBEAT_INTERVAL).contains(&interval) {
|
||||
anyhow::bail!("heartbeat interval must be between 1000 and 60000 milliseconds");
|
||||
}
|
||||
if !(MIN_HEARTBEAT_TIMEOUT..=MAX_HEARTBEAT_TIMEOUT).contains(&timeout) {
|
||||
anyhow::bail!("heartbeat timeout must be between 5000 and 120000 milliseconds");
|
||||
}
|
||||
if timeout < interval.saturating_add(HEARTBEAT_TIMEOUT_MARGIN) {
|
||||
anyhow::bail!(
|
||||
"heartbeat timeout must exceed the interval by at least 5000 milliseconds"
|
||||
);
|
||||
}
|
||||
Ok(Self { interval, timeout })
|
||||
}
|
||||
|
||||
fn response(self) -> HeartbeatResponse {
|
||||
HeartbeatResponse {
|
||||
heartbeat_interval_ms: Some(self.interval.as_millis() as u32),
|
||||
heartbeat_timeout_ms: Some(self.timeout.as_millis() as u32),
|
||||
}
|
||||
}
|
||||
|
||||
fn session_rx_timeout(self) -> Duration {
|
||||
Duration::from_secs(30).max(self.timeout.saturating_add(HEARTBEAT_TIMEOUT_MARGIN))
|
||||
}
|
||||
|
||||
fn legacy_response_delay(self) -> Duration {
|
||||
self.interval.min(DEFAULT_HEARTBEAT_INTERVAL)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(rust_embed::Embed)]
|
||||
#[folder = "resources/"]
|
||||
#[include = "geoip2-cn.mmdb"]
|
||||
@@ -69,14 +135,14 @@ pub struct ClientManager {
|
||||
webhook_config: SharedWebhookConfig,
|
||||
|
||||
geoip_db: Arc<Option<maxminddb::Reader<Vec<u8>>>>,
|
||||
heartbeat_min_response_delay: Duration,
|
||||
heartbeat_policy: HeartbeatPolicy,
|
||||
}
|
||||
|
||||
impl ClientManager {
|
||||
pub fn new(
|
||||
db: Db,
|
||||
geoip_db: Option<String>,
|
||||
heartbeat_min_response_delay: Duration,
|
||||
heartbeat_policy: HeartbeatPolicy,
|
||||
feature_flags: Arc<FeatureFlags>,
|
||||
webhook_config: SharedWebhookConfig,
|
||||
) -> Self {
|
||||
@@ -101,7 +167,7 @@ impl ClientManager {
|
||||
webhook_config,
|
||||
|
||||
geoip_db: Arc::new(load_geoip_db(geoip_db)),
|
||||
heartbeat_min_response_delay,
|
||||
heartbeat_policy,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -117,7 +183,7 @@ impl ClientManager {
|
||||
let listeners_cnt = self.listeners_cnt.clone();
|
||||
let next_session_epoch = self.next_session_epoch.clone();
|
||||
let geoip_db = self.geoip_db.clone();
|
||||
let heartbeat_min_response_delay = self.heartbeat_min_response_delay;
|
||||
let heartbeat_policy = self.heartbeat_policy;
|
||||
let feature_flags = self.feature_flags.clone();
|
||||
let webhook_config = self.webhook_config.clone();
|
||||
self.tasks.spawn(async move {
|
||||
@@ -146,7 +212,7 @@ impl ClientManager {
|
||||
storage.clone(),
|
||||
client_url.clone(),
|
||||
location,
|
||||
heartbeat_min_response_delay,
|
||||
heartbeat_policy,
|
||||
feature_flags.clone(),
|
||||
webhook_config.clone(),
|
||||
next_session_epoch.fetch_add(1, Ordering::Relaxed) + 1,
|
||||
@@ -188,7 +254,7 @@ impl ClientManager {
|
||||
.get_client_url_by_machine_id(user_id, machine_id)?;
|
||||
self.client_sessions
|
||||
.get(&c_url)
|
||||
.map(|item| item.value().clone())
|
||||
.and_then(|item| item.is_running().then(|| item.value().clone()))
|
||||
}
|
||||
|
||||
pub async fn disconnect_session_by_machine_id(
|
||||
@@ -238,11 +304,14 @@ impl ClientManager {
|
||||
if matches!(
|
||||
status,
|
||||
managed_config::ManagedConfigApplyStatus::Applied { .. }
|
||||
) && let Some(config_revision) = config_revision
|
||||
&& let Some(session) = self.get_session_by_machine_id(user_id, &machine_id)
|
||||
) && self.storage.record_full_managed_config_change(
|
||||
user_id,
|
||||
machine_id,
|
||||
config_revision.as_deref(),
|
||||
) && let Some(session) = self.get_session_by_machine_id(user_id, &machine_id)
|
||||
{
|
||||
session
|
||||
.notify_full_config_revision_changed(user_id, machine_id, config_revision)
|
||||
.notify_managed_runtime_state_changed(user_id, machine_id)
|
||||
.await;
|
||||
}
|
||||
Ok(())
|
||||
@@ -259,7 +328,7 @@ impl ClientManager {
|
||||
) -> anyhow::Result<()> {
|
||||
let config_revision = config_revision.trim().to_string();
|
||||
let expected_config_revision = expected_config_revision.trim().to_string();
|
||||
let upsert_instance_ids = upserts
|
||||
let mut dirty_instance_ids: HashSet<_> = upserts
|
||||
.iter()
|
||||
.map(|config| config.instance_id.clone())
|
||||
.collect();
|
||||
@@ -276,23 +345,26 @@ impl ClientManager {
|
||||
if let managed_config::ManagedConfigApplyStatus::Applied {
|
||||
deleted_web_instance_ids,
|
||||
} = status
|
||||
&& let Some(session) = self.get_session_by_machine_id(user_id, &machine_id)
|
||||
{
|
||||
session
|
||||
.notify_patch_config_revision_changed(
|
||||
user_id,
|
||||
machine_id,
|
||||
ManagedConfigRevisionDelta {
|
||||
expected_revision: expected_config_revision,
|
||||
target_revision: config_revision,
|
||||
upsert_instance_ids,
|
||||
delete_instance_ids: deleted_web_instance_ids
|
||||
.into_iter()
|
||||
.map(|instance_id| instance_id.to_string())
|
||||
.collect(),
|
||||
},
|
||||
)
|
||||
.await;
|
||||
dirty_instance_ids.extend(
|
||||
deleted_web_instance_ids
|
||||
.into_iter()
|
||||
.map(|instance_id| instance_id.to_string()),
|
||||
);
|
||||
let changed = self.storage.record_patch_managed_config_change(
|
||||
user_id,
|
||||
machine_id,
|
||||
ManagedConfigPersistedChange {
|
||||
expected_revision: expected_config_revision,
|
||||
target_revision: config_revision,
|
||||
dirty_instance_ids,
|
||||
},
|
||||
);
|
||||
if changed && let Some(session) = self.get_session_by_machine_id(user_id, &machine_id) {
|
||||
session
|
||||
.notify_managed_runtime_state_changed(user_id, machine_id)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -302,9 +374,13 @@ impl ClientManager {
|
||||
user_id: UserIdInDb,
|
||||
machine_id: uuid::Uuid,
|
||||
) {
|
||||
if let Some(session) = self.get_session_by_machine_id(user_id, &machine_id) {
|
||||
if self
|
||||
.storage
|
||||
.invalidate_managed_runtime_state(user_id, machine_id)
|
||||
&& let Some(session) = self.get_session_by_machine_id(user_id, &machine_id)
|
||||
{
|
||||
session
|
||||
.invalidate_applied_config_revision(user_id, machine_id)
|
||||
.notify_managed_runtime_state_changed(user_id, machine_id)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
@@ -474,11 +550,36 @@ mod tests {
|
||||
use sqlx::Executor;
|
||||
|
||||
use crate::{
|
||||
FeatureFlags, client_manager::ClientManager, db::Db, webhook::ManagedNetworkConfig,
|
||||
FeatureFlags,
|
||||
client_manager::{ClientManager, HeartbeatPolicy, session::Session, storage::StorageToken},
|
||||
db::Db,
|
||||
webhook::ManagedNetworkConfig,
|
||||
};
|
||||
|
||||
const MANAGED_CONFIG_TOKEN: &str = "managed-config-token";
|
||||
|
||||
#[test]
|
||||
fn heartbeat_policy_validates_server_configuration() {
|
||||
let policy = HeartbeatPolicy::from_millis(3_500, 15_000).unwrap();
|
||||
let response = policy.response();
|
||||
assert_eq!(response.heartbeat_interval_ms, Some(3_500));
|
||||
assert_eq!(response.heartbeat_timeout_ms, Some(15_000));
|
||||
assert_eq!(policy.session_rx_timeout(), Duration::from_secs(30));
|
||||
|
||||
let legacy_default = HeartbeatPolicy::from_millis(0, 15_000).unwrap();
|
||||
assert_eq!(legacy_default.response().heartbeat_interval_ms, Some(3_500));
|
||||
|
||||
let slow = HeartbeatPolicy::from_millis(60_000, 65_000).unwrap();
|
||||
assert_eq!(slow.session_rx_timeout(), Duration::from_secs(70));
|
||||
assert_eq!(slow.legacy_response_delay(), Duration::from_millis(3_500));
|
||||
|
||||
assert!(HeartbeatPolicy::from_millis(999, 15_000).is_err());
|
||||
assert!(HeartbeatPolicy::from_millis(60_001, 120_000).is_err());
|
||||
assert!(HeartbeatPolicy::from_millis(3_500, 4_999).is_err());
|
||||
assert!(HeartbeatPolicy::from_millis(60_000, 64_999).is_err());
|
||||
assert!(HeartbeatPolicy::from_millis(3_500, 120_001).is_err());
|
||||
}
|
||||
|
||||
async fn wait_for_condition<F, Fut>(mut condition: F, timeout: Duration)
|
||||
where
|
||||
F: FnMut() -> Fut,
|
||||
@@ -655,7 +756,7 @@ mod tests {
|
||||
let mut mgr = ClientManager::new(
|
||||
Db::memory_db().await,
|
||||
None,
|
||||
Duration::ZERO,
|
||||
HeartbeatPolicy::from_millis(0, 15_000).unwrap(),
|
||||
Arc::new(FeatureFlags::default()),
|
||||
webhook_config,
|
||||
);
|
||||
@@ -683,6 +784,51 @@ mod tests {
|
||||
webhook_server.abort();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn non_running_session_is_not_routable_by_machine_id() {
|
||||
let db = Db::memory_db().await;
|
||||
let mgr = ClientManager::new(
|
||||
db.clone(),
|
||||
None,
|
||||
HeartbeatPolicy::default(),
|
||||
Arc::new(FeatureFlags::default()),
|
||||
Arc::new(crate::webhook::WebhookConfig::new(
|
||||
None, None, None, None, None,
|
||||
)),
|
||||
);
|
||||
let user_id = db.auto_create_user("token").await.unwrap().id;
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let client_url = url::Url::parse("udp://127.0.0.1:22020").unwrap();
|
||||
mgr.storage.update_client(
|
||||
StorageToken {
|
||||
token: "token".to_string(),
|
||||
client_url: client_url.clone(),
|
||||
machine_id,
|
||||
user_id,
|
||||
},
|
||||
1,
|
||||
true,
|
||||
);
|
||||
let session = Arc::new(Session::new(
|
||||
mgr.storage.weak_ref(),
|
||||
client_url.clone(),
|
||||
None,
|
||||
HeartbeatPolicy::default(),
|
||||
Arc::new(FeatureFlags::default()),
|
||||
Arc::new(crate::webhook::WebhookConfig::new(
|
||||
None, None, None, None, None,
|
||||
)),
|
||||
1,
|
||||
));
|
||||
assert!(!session.is_running());
|
||||
mgr.client_sessions.insert(client_url, session);
|
||||
|
||||
assert!(
|
||||
mgr.get_session_by_machine_id(user_id, &machine_id)
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
async fn wait_for_validated_user(mgr: &ClientManager, machine_id: uuid::Uuid) -> i32 {
|
||||
tokio::time::timeout(Duration::from_secs(12), async {
|
||||
loop {
|
||||
@@ -1008,7 +1154,7 @@ mod tests {
|
||||
let mut mgr = ClientManager::new(
|
||||
Db::memory_db().await,
|
||||
None,
|
||||
Duration::ZERO,
|
||||
HeartbeatPolicy::from_millis(0, 15_000).unwrap(),
|
||||
Arc::new(FeatureFlags::default()),
|
||||
Arc::new(crate::webhook::WebhookConfig::new(
|
||||
None, None, None, None, None,
|
||||
@@ -1075,7 +1221,7 @@ mod tests {
|
||||
let mut mgr = ClientManager::new(
|
||||
Db::memory_db().await,
|
||||
None,
|
||||
Duration::ZERO,
|
||||
HeartbeatPolicy::from_millis(0, 15_000).unwrap(),
|
||||
Arc::new(FeatureFlags::default()),
|
||||
webhook_config,
|
||||
);
|
||||
@@ -1141,7 +1287,7 @@ mod tests {
|
||||
let mut mgr = ClientManager::new(
|
||||
Db::memory_db().await,
|
||||
None,
|
||||
Duration::ZERO,
|
||||
HeartbeatPolicy::from_millis(0, 15_000).unwrap(),
|
||||
Arc::new(FeatureFlags::default()),
|
||||
webhook_config,
|
||||
);
|
||||
@@ -1333,7 +1479,7 @@ mod tests {
|
||||
let mut mgr = ClientManager::new(
|
||||
Db::memory_db().await,
|
||||
None,
|
||||
Duration::ZERO,
|
||||
HeartbeatPolicy::from_millis(0, 15_000).unwrap(),
|
||||
Arc::new(FeatureFlags::default()),
|
||||
webhook_config,
|
||||
);
|
||||
|
||||
@@ -28,7 +28,7 @@ use easytier::{
|
||||
use super::session::{SessionConfigClient, SessionRpcClient};
|
||||
|
||||
pub(super) enum RuntimeReconcileAction {
|
||||
None,
|
||||
Unchanged(Box<NetworkConfig>),
|
||||
Run {
|
||||
config: Box<NetworkConfig>,
|
||||
overwrite: bool,
|
||||
@@ -269,27 +269,47 @@ fn normalized_managed_credentials(
|
||||
Ok(NetworkConfig::new_from_config(config.gen_config()?)?.managed_credentials)
|
||||
}
|
||||
|
||||
fn is_automatic_windows_dev_name(dev_name: &str) -> bool {
|
||||
let Some((interface_count, suffix)) = dev_name
|
||||
.strip_prefix("et_")
|
||||
.and_then(|value| value.split_once('_'))
|
||||
else {
|
||||
return false;
|
||||
};
|
||||
!interface_count.is_empty()
|
||||
&& interface_count.bytes().all(|byte| byte.is_ascii_digit())
|
||||
&& suffix.len() == 4
|
||||
&& suffix
|
||||
.bytes()
|
||||
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit())
|
||||
}
|
||||
|
||||
fn web_source_runtime_patch(
|
||||
current: &NetworkConfig,
|
||||
desired: &NetworkConfig,
|
||||
) -> anyhow::Result<Option<InstanceConfigPatch>> {
|
||||
if let Some(desired_hostname) = desired
|
||||
.hostname
|
||||
.as_deref()
|
||||
.filter(|hostname| !hostname.is_empty())
|
||||
&& current.hostname.as_deref() != Some(desired_hostname)
|
||||
{
|
||||
return Ok(None);
|
||||
}
|
||||
let mut current_base = hot_patch_base(current)?;
|
||||
let mut desired_base = hot_patch_base(desired)?;
|
||||
current_base.hostname = None;
|
||||
desired_base.hostname = None;
|
||||
if desired.dev_name.is_none()
|
||||
|| (desired.dev_name.as_deref() == Some("")
|
||||
&& current
|
||||
.dev_name
|
||||
.as_deref()
|
||||
.is_some_and(is_automatic_windows_dev_name))
|
||||
{
|
||||
current_base.dev_name = None;
|
||||
desired_base.dev_name = None;
|
||||
}
|
||||
let current_hostname = current_base.hostname.take().unwrap_or_default();
|
||||
let desired_hostname = desired_base.hostname.take().unwrap_or_default();
|
||||
if current_base != desired_base {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
let mut patch = InstanceConfigPatch::default();
|
||||
if desired.hostname.is_some() && current_hostname != desired_hostname {
|
||||
patch.hostname = Some(desired_hostname);
|
||||
}
|
||||
let current_acl = normalized_acl(¤t.acl);
|
||||
let desired_acl = normalized_acl(&desired.acl);
|
||||
if current_acl != desired_acl {
|
||||
@@ -358,14 +378,41 @@ fn web_source_runtime_patch(
|
||||
Ok(Some(patch))
|
||||
}
|
||||
|
||||
fn ensure_runtime_config_converged(
|
||||
// Release 2.6.4 omits a configured hostname that matches the device
|
||||
// hostname from config readback. After a successful hostname mutation the
|
||||
// desired value must be restored into the observed config, otherwise every
|
||||
// later plan re-sends the same hostname patch.
|
||||
pub(super) fn restore_omitted_hostname(
|
||||
current: &mut NetworkConfig,
|
||||
desired: &NetworkConfig,
|
||||
hostname_applied: bool,
|
||||
) {
|
||||
if hostname_applied && current.hostname.is_none() && desired.hostname.is_some() {
|
||||
current.hostname = desired.hostname.clone();
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn ensure_runtime_config_converged(
|
||||
current: &NetworkConfig,
|
||||
desired: &NetworkConfig,
|
||||
hostname_applied: bool,
|
||||
) -> anyhow::Result<()> {
|
||||
let patch = web_source_runtime_patch(current, desired)?;
|
||||
match patch {
|
||||
Some(patch) if patch == InstanceConfigPatch::default() => Ok(()),
|
||||
Some(_) => anyhow::bail!("runtime config still needs patch after reconcile"),
|
||||
Some(mut patch) => {
|
||||
// Release 2.6.4 omits a configured hostname when it equals
|
||||
// the device hostname. The successful mutation is therefore
|
||||
// authoritative for hostname, while every other field remains
|
||||
// verified from the runtime readback.
|
||||
if hostname_applied && current.hostname.is_none() {
|
||||
patch.hostname = None;
|
||||
}
|
||||
if patch == InstanceConfigPatch::default() {
|
||||
Ok(())
|
||||
} else {
|
||||
anyhow::bail!("runtime config still needs patch after reconcile")
|
||||
}
|
||||
}
|
||||
None => anyhow::bail!("runtime config still needs full overwrite after reconcile"),
|
||||
}
|
||||
}
|
||||
@@ -429,13 +476,19 @@ pub(super) fn prepare_web_source_runtime_reconcile_from_current(
|
||||
desired_config: NetworkConfig,
|
||||
) -> anyhow::Result<RuntimeReconcileAction> {
|
||||
let Some(patch) = web_source_runtime_patch(current_config, &desired_config)? else {
|
||||
let mut run_config = desired_config;
|
||||
if run_config.hostname.is_none() {
|
||||
run_config.hostname = current_config.hostname.clone();
|
||||
}
|
||||
return Ok(RuntimeReconcileAction::Run {
|
||||
config: Box::new(desired_config),
|
||||
config: Box::new(run_config),
|
||||
overwrite: true,
|
||||
});
|
||||
};
|
||||
if patch == InstanceConfigPatch::default() {
|
||||
return Ok(RuntimeReconcileAction::None);
|
||||
return Ok(RuntimeReconcileAction::Unchanged(Box::new(
|
||||
current_config.clone(),
|
||||
)));
|
||||
}
|
||||
|
||||
Ok(RuntimeReconcileAction::Patch(Box::new(patch)))
|
||||
@@ -449,12 +502,17 @@ pub(super) async fn apply_web_source_runtime_reconcile(
|
||||
action: RuntimeReconcileAction,
|
||||
) -> anyhow::Result<NetworkConfig> {
|
||||
match action {
|
||||
RuntimeReconcileAction::None => Ok(desired_config),
|
||||
RuntimeReconcileAction::Unchanged(current_config) => Ok(*current_config),
|
||||
RuntimeReconcileAction::Run { config, overwrite } => {
|
||||
let hostname_applied = config.hostname.is_some();
|
||||
run_web_source_instance(rpc_client, inst_id, *config, overwrite).await?;
|
||||
Ok(desired_config)
|
||||
let mut current_config = get_runtime_config(rpc_client, inst_id).await?;
|
||||
ensure_runtime_config_converged(¤t_config, &desired_config, hostname_applied)?;
|
||||
restore_omitted_hostname(&mut current_config, &desired_config, hostname_applied);
|
||||
Ok(current_config)
|
||||
}
|
||||
RuntimeReconcileAction::Patch(patch) => {
|
||||
let hostname_applied = patch.hostname.is_some();
|
||||
config_client
|
||||
.patch_config(
|
||||
BaseController::default(),
|
||||
@@ -464,8 +522,9 @@ pub(super) async fn apply_web_source_runtime_reconcile(
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
let current_config = get_runtime_config(rpc_client, inst_id).await?;
|
||||
ensure_runtime_config_converged(¤t_config, &desired_config)?;
|
||||
let mut current_config = get_runtime_config(rpc_client, inst_id).await?;
|
||||
ensure_runtime_config_converged(¤t_config, &desired_config, hostname_applied)?;
|
||||
restore_omitted_hostname(&mut current_config, &desired_config, hostname_applied);
|
||||
Ok(current_config)
|
||||
}
|
||||
}
|
||||
@@ -759,6 +818,63 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_reconcile_ignores_automatic_device_name_when_unmanaged() {
|
||||
let mut current = config_with_port_forwards(Vec::new());
|
||||
current.dev_name = Some("et_3_abcd".to_string());
|
||||
let desired = config_with_port_forwards(Vec::new());
|
||||
|
||||
let action = prepare_web_source_runtime_reconcile_from_current(¤t, desired)
|
||||
.expect("prepare reconcile");
|
||||
|
||||
assert!(matches!(action, RuntimeReconcileAction::Unchanged(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_reconcile_ignores_automatic_device_name_for_empty_desired_name() {
|
||||
let mut current = config_with_port_forwards(Vec::new());
|
||||
current.dev_name = Some("et_3_abcd".to_string());
|
||||
let mut desired = config_with_port_forwards(Vec::new());
|
||||
desired.dev_name = Some(String::new());
|
||||
|
||||
let action = prepare_web_source_runtime_reconcile_from_current(¤t, desired)
|
||||
.expect("prepare reconcile");
|
||||
|
||||
assert!(matches!(action, RuntimeReconcileAction::Unchanged(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_reconcile_clears_explicit_device_name() {
|
||||
let mut current = config_with_port_forwards(Vec::new());
|
||||
current.dev_name = Some("managed-device".to_string());
|
||||
let mut desired = config_with_port_forwards(Vec::new());
|
||||
desired.dev_name = Some(String::new());
|
||||
|
||||
let action = prepare_web_source_runtime_reconcile_from_current(¤t, desired)
|
||||
.expect("prepare reconcile");
|
||||
let RuntimeReconcileAction::Run { overwrite, .. } = action else {
|
||||
panic!("clearing an explicit device name should require a full overwrite");
|
||||
};
|
||||
|
||||
assert!(overwrite);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_reconcile_applies_explicit_device_name() {
|
||||
let mut current = config_with_port_forwards(Vec::new());
|
||||
current.dev_name = Some("et_3_abcd".to_string());
|
||||
let mut desired = config_with_port_forwards(Vec::new());
|
||||
desired.dev_name = Some("managed-device".to_string());
|
||||
|
||||
let action = prepare_web_source_runtime_reconcile_from_current(¤t, desired)
|
||||
.expect("prepare reconcile");
|
||||
let RuntimeReconcileAction::Run { overwrite, .. } = action else {
|
||||
panic!("explicit device name should require a full overwrite");
|
||||
};
|
||||
|
||||
assert!(overwrite);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_convergence_rejects_stale_extra_port_forward() {
|
||||
let current = config_with_port_forwards(vec![
|
||||
@@ -769,7 +885,7 @@ mod tests {
|
||||
let desired =
|
||||
config_with_port_forwards(vec![port_forward(23000, 5174), port_forward(23007, 3389)]);
|
||||
|
||||
let err = ensure_runtime_config_converged(¤t, &desired)
|
||||
let err = ensure_runtime_config_converged(¤t, &desired, false)
|
||||
.expect_err("extra runtime port forward should not converge");
|
||||
|
||||
assert!(
|
||||
@@ -791,13 +907,14 @@ mod tests {
|
||||
.expect("hot patch");
|
||||
|
||||
assert_eq!(patch, InstanceConfigPatch::default());
|
||||
ensure_runtime_config_converged(¤t, &desired).expect("runtime converged");
|
||||
ensure_runtime_config_converged(¤t, &desired, false).expect("runtime converged");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_patch_rejects_non_hot_config_change() {
|
||||
let current = config_with_port_forwards(Vec::new());
|
||||
let mut desired = current.clone();
|
||||
|
||||
desired.network_secret = Some("new-secret".to_string());
|
||||
|
||||
let patch = web_source_runtime_patch(¤t, &desired).expect("build patch");
|
||||
@@ -805,6 +922,36 @@ mod tests {
|
||||
assert!(patch.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn full_overwrite_preserves_unmanaged_hostname_for_later_explicit_clear() {
|
||||
let mut current = config_with_port_forwards(Vec::new());
|
||||
current.hostname = Some("runtime-host".to_string());
|
||||
let mut unmanaged_desired = current.clone();
|
||||
unmanaged_desired.hostname = None;
|
||||
unmanaged_desired.network_secret = Some("new-secret".to_string());
|
||||
|
||||
let action =
|
||||
prepare_web_source_runtime_reconcile_from_current(¤t, unmanaged_desired.clone())
|
||||
.expect("prepare full overwrite");
|
||||
let RuntimeReconcileAction::Run { config, overwrite } = action else {
|
||||
panic!("non-hot change should require a full overwrite");
|
||||
};
|
||||
assert!(overwrite);
|
||||
assert_eq!(config.hostname.as_deref(), Some("runtime-host"));
|
||||
|
||||
let observed_after_run = *config;
|
||||
let mut explicit_clear = unmanaged_desired;
|
||||
explicit_clear.hostname = Some(String::new());
|
||||
let action =
|
||||
prepare_web_source_runtime_reconcile_from_current(&observed_after_run, explicit_clear)
|
||||
.expect("prepare explicit clear");
|
||||
let RuntimeReconcileAction::Patch(patch) = action else {
|
||||
panic!("explicit clear should patch the preserved runtime hostname");
|
||||
};
|
||||
|
||||
assert_eq!(patch.hostname.as_deref(), Some(""));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_patch_replaces_managed_credentials_without_full_run() {
|
||||
let current = config_with_port_forwards(Vec::new());
|
||||
@@ -1086,14 +1233,136 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_patch_rejects_explicit_desired_hostname_change() {
|
||||
let current = config_with_port_forwards(vec![port_forward(23000, 5174)]);
|
||||
let mut desired =
|
||||
config_with_port_forwards(vec![port_forward(23000, 5174), port_forward(23007, 3389)]);
|
||||
fn runtime_reconcile_hot_patches_explicit_desired_hostname_change() {
|
||||
let current = config_with_port_forwards(Vec::new());
|
||||
let mut desired = config_with_port_forwards(Vec::new());
|
||||
desired.hostname = Some("desired-host".to_string());
|
||||
|
||||
let patch = web_source_runtime_patch(¤t, &desired).expect("build patch");
|
||||
let action = prepare_web_source_runtime_reconcile_from_current(¤t, desired)
|
||||
.expect("prepare reconcile");
|
||||
let RuntimeReconcileAction::Patch(patch) = action else {
|
||||
panic!("hostname-only change should use a hot patch");
|
||||
};
|
||||
|
||||
assert!(patch.is_none());
|
||||
assert_eq!(patch.hostname.as_deref(), Some("desired-host"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_convergence_accepts_hostname_only_readback_difference() {
|
||||
let current = config_with_port_forwards(Vec::new());
|
||||
let mut desired = config_with_port_forwards(Vec::new());
|
||||
desired.hostname = Some("device-host".to_string());
|
||||
|
||||
ensure_runtime_config_converged(¤t, &desired, true)
|
||||
.expect("hostname-only readback difference should be converged");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_convergence_rejects_omitted_hostname_before_apply() {
|
||||
let current = config_with_port_forwards(Vec::new());
|
||||
let mut desired = config_with_port_forwards(Vec::new());
|
||||
desired.hostname = Some("device-host".to_string());
|
||||
|
||||
let err = ensure_runtime_config_converged(¤t, &desired, false)
|
||||
.expect_err("omitted hostname before apply should not converge");
|
||||
|
||||
assert!(
|
||||
err.to_string()
|
||||
.contains("runtime config still needs patch after reconcile")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_convergence_rejects_explicit_wrong_hostname_after_apply() {
|
||||
let mut current = config_with_port_forwards(Vec::new());
|
||||
current.hostname = Some("wrong-host".to_string());
|
||||
let mut desired = config_with_port_forwards(Vec::new());
|
||||
desired.hostname = Some("device-host".to_string());
|
||||
|
||||
let err = ensure_runtime_config_converged(¤t, &desired, true)
|
||||
.expect_err("explicit wrong hostname should not converge");
|
||||
|
||||
assert!(
|
||||
err.to_string()
|
||||
.contains("runtime config still needs patch after reconcile")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_patch_clears_explicit_desired_hostname() {
|
||||
let mut current = config_with_port_forwards(Vec::new());
|
||||
current.hostname = Some("runtime-host".to_string());
|
||||
let mut desired = config_with_port_forwards(Vec::new());
|
||||
desired.hostname = Some(String::new());
|
||||
|
||||
let patch = web_source_runtime_patch(¤t, &desired)
|
||||
.expect("build patch")
|
||||
.expect("hot patch");
|
||||
|
||||
assert_eq!(patch.hostname.as_deref(), Some(""));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_patch_normalizes_missing_runtime_hostname_for_explicit_clear() {
|
||||
let current = config_with_port_forwards(Vec::new());
|
||||
let mut desired = config_with_port_forwards(Vec::new());
|
||||
desired.hostname = Some(String::new());
|
||||
|
||||
let action = prepare_web_source_runtime_reconcile_from_current(¤t, desired)
|
||||
.expect("prepare reconcile");
|
||||
|
||||
assert!(matches!(action, RuntimeReconcileAction::Unchanged(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_patch_skips_matching_explicit_hostname() {
|
||||
let mut current = config_with_port_forwards(Vec::new());
|
||||
current.hostname = Some("desired-host".to_string());
|
||||
let desired = current.clone();
|
||||
|
||||
let action = prepare_web_source_runtime_reconcile_from_current(¤t, desired)
|
||||
.expect("prepare reconcile");
|
||||
|
||||
assert!(matches!(action, RuntimeReconcileAction::Unchanged(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_patch_uses_core_normalized_hostname() {
|
||||
let current = config_with_port_forwards(Vec::new());
|
||||
let mut desired = config_with_port_forwards(Vec::new());
|
||||
desired.hostname = Some("a".repeat(33));
|
||||
|
||||
let patch = web_source_runtime_patch(¤t, &desired)
|
||||
.expect("build patch")
|
||||
.expect("hot patch");
|
||||
|
||||
assert_eq!(patch.hostname.as_deref(), Some("a".repeat(32).as_str()));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_patch_removes_hostname_control_characters() {
|
||||
let current = config_with_port_forwards(Vec::new());
|
||||
let mut desired = config_with_port_forwards(Vec::new());
|
||||
desired.hostname = Some("node\u{7}-name".to_string());
|
||||
|
||||
let patch = web_source_runtime_patch(¤t, &desired)
|
||||
.expect("build patch")
|
||||
.expect("hot patch");
|
||||
|
||||
assert_eq!(patch.hostname.as_deref(), Some("node-name"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn runtime_patch_normalizes_control_only_hostname_to_clear() {
|
||||
let mut current = config_with_port_forwards(Vec::new());
|
||||
current.hostname = Some("runtime-host".to_string());
|
||||
let mut desired = config_with_port_forwards(Vec::new());
|
||||
desired.hostname = Some("\u{7}\n".to_string());
|
||||
|
||||
let patch = web_source_runtime_patch(¤t, &desired)
|
||||
.expect("build patch")
|
||||
.expect("hot patch");
|
||||
|
||||
assert_eq!(patch.hostname.as_deref(), Some(""));
|
||||
}
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -25,6 +25,8 @@ pub(super) struct WebhookValidationInput {
|
||||
pub(super) webhook_config: SharedWebhookConfig,
|
||||
pub(super) client_url: url::Url,
|
||||
pub(super) applied_config_revision: Option<String>,
|
||||
pub(super) applied_config_revision_known: bool,
|
||||
pub(super) failed_instance_ids: Vec<String>,
|
||||
pub(super) req: HeartbeatRequest,
|
||||
pub(super) machine_id: uuid::Uuid,
|
||||
}
|
||||
@@ -40,28 +42,39 @@ pub(super) fn retry_delay(machine_id: uuid::Uuid) -> Duration {
|
||||
}
|
||||
|
||||
async fn request_heartbeat_validation(
|
||||
webhook_config: &crate::webhook::WebhookConfig,
|
||||
client_url: &url::Url,
|
||||
input: &WebhookValidationInput,
|
||||
persisted_config_revision: Option<&str>,
|
||||
applied_config_revision: Option<&str>,
|
||||
req: &HeartbeatRequest,
|
||||
machine_id: uuid::Uuid,
|
||||
) -> anyhow::Result<Option<WebhookHeartbeatValidation>> {
|
||||
let webhook_req = crate::webhook::ValidateTokenRequest {
|
||||
token: req.user_token.clone(),
|
||||
machine_id: machine_id.to_string(),
|
||||
public_ip: client_url.host_str().map(str::to_string),
|
||||
hostname: req.hostname.clone(),
|
||||
version: req.easytier_version.clone(),
|
||||
os_type: req.device_os.as_ref().map(|info| info.os_type.clone()),
|
||||
os_version: req.device_os.as_ref().map(|info| info.version.clone()),
|
||||
os_distribution: req.device_os.as_ref().map(|info| info.distribution.clone()),
|
||||
web_instance_id: webhook_config.web_instance_id.clone(),
|
||||
web_instance_api_base_url: webhook_config.web_instance_api_base_url.clone(),
|
||||
token: input.req.user_token.clone(),
|
||||
machine_id: input.machine_id.to_string(),
|
||||
public_ip: input.client_url.host_str().map(str::to_string),
|
||||
hostname: input.req.hostname.clone(),
|
||||
version: input.req.easytier_version.clone(),
|
||||
os_type: input
|
||||
.req
|
||||
.device_os
|
||||
.as_ref()
|
||||
.map(|info| info.os_type.clone()),
|
||||
os_version: input
|
||||
.req
|
||||
.device_os
|
||||
.as_ref()
|
||||
.map(|info| info.version.clone()),
|
||||
os_distribution: input
|
||||
.req
|
||||
.device_os
|
||||
.as_ref()
|
||||
.map(|info| info.distribution.clone()),
|
||||
web_instance_id: input.webhook_config.web_instance_id.clone(),
|
||||
web_instance_api_base_url: input.webhook_config.web_instance_api_base_url.clone(),
|
||||
persisted_config_revision: persisted_config_revision.map(str::to_string),
|
||||
applied_config_revision: applied_config_revision.map(str::to_string),
|
||||
applied_config_revision: input.applied_config_revision.as_deref().map(str::to_string),
|
||||
applied_config_revision_known: input.applied_config_revision_known,
|
||||
failed_instance_ids: input.failed_instance_ids.to_vec(),
|
||||
};
|
||||
let resp = webhook_config
|
||||
let resp = input
|
||||
.webhook_config
|
||||
.validate_token(&webhook_req)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("Webhook token validation failed: {:?}", e))?;
|
||||
@@ -87,7 +100,7 @@ async fn resolve_user_id(storage: &Storage, token: &str) -> anyhow::Result<i32>
|
||||
None => storage
|
||||
.auto_create_user(token)
|
||||
.await
|
||||
.with_context(|| format!("Failed to auto-create webhook user: {:?}", token))?,
|
||||
.with_context(|| "Failed to auto-create webhook user".to_string())?,
|
||||
};
|
||||
|
||||
Ok(user_id)
|
||||
@@ -115,7 +128,7 @@ async fn persisted_config_revision_for_token(
|
||||
|
||||
async fn wait_for_input(
|
||||
session_data: std::sync::Weak<RwLock<SessionData>>,
|
||||
) -> Option<WebhookValidationInput> {
|
||||
) -> Option<(WebhookValidationInput, u64)> {
|
||||
loop {
|
||||
let notify = {
|
||||
let session_data = session_data.upgrade()?;
|
||||
@@ -133,14 +146,28 @@ async fn wait_for_input(
|
||||
let req = data.req.clone()?;
|
||||
let machine_id = req.machine_id.map(Into::into)?;
|
||||
let storage = Storage::try_from(data.storage.clone()).ok()?;
|
||||
return Some(WebhookValidationInput {
|
||||
storage,
|
||||
webhook_config: data.webhook_config.clone(),
|
||||
client_url: data.client_url.clone(),
|
||||
applied_config_revision: data.applied_config_revision.clone(),
|
||||
req,
|
||||
machine_id,
|
||||
});
|
||||
let (applied_config_revision, applied_config_revision_known) = {
|
||||
let runtime = data.managed_runtime();
|
||||
(
|
||||
runtime.applied_config_revision.clone(),
|
||||
runtime.applied_config_revision_known,
|
||||
)
|
||||
};
|
||||
return Some((
|
||||
WebhookValidationInput {
|
||||
storage,
|
||||
webhook_config: data.webhook_config.clone(),
|
||||
client_url: data.client_url.clone(),
|
||||
applied_config_revision,
|
||||
applied_config_revision_known,
|
||||
failed_instance_ids: SessionRpcService::sorted_failed_instance_ids_locked(
|
||||
&data,
|
||||
),
|
||||
req,
|
||||
machine_id,
|
||||
},
|
||||
data.webhook_validation_change_epoch,
|
||||
));
|
||||
}
|
||||
data.webhook_validation_notify.clone()
|
||||
};
|
||||
@@ -148,17 +175,64 @@ async fn wait_for_input(
|
||||
}
|
||||
}
|
||||
|
||||
async fn wait_for_retry_or_state_change(
|
||||
session_data: &std::sync::Weak<RwLock<SessionData>>,
|
||||
machine_id: uuid::Uuid,
|
||||
validation_change_epoch: u64,
|
||||
delay: Duration,
|
||||
) {
|
||||
let retry_deadline = tokio::time::sleep(delay);
|
||||
tokio::pin!(retry_deadline);
|
||||
|
||||
loop {
|
||||
let notify = {
|
||||
let Some(session_data) = session_data.upgrade() else {
|
||||
return;
|
||||
};
|
||||
let data = session_data.read().await;
|
||||
let Some(req) = data.req.as_ref() else {
|
||||
return;
|
||||
};
|
||||
if req.machine_id.map(uuid::Uuid::from) != Some(machine_id)
|
||||
|| matches!(data.auth_state, SessionAuthState::Invalid)
|
||||
{
|
||||
return;
|
||||
}
|
||||
if data.webhook_validation_change_epoch != validation_change_epoch {
|
||||
return;
|
||||
}
|
||||
data.webhook_validation_notify.clone()
|
||||
};
|
||||
|
||||
// Notify is only a wake-up hint. Periodic validation can set dirty,
|
||||
// but only a meaningful validation-state change may bypass backoff.
|
||||
// Recheck the epoch after every wake without resetting the deadline.
|
||||
tokio::select! {
|
||||
_ = &mut retry_deadline => {
|
||||
mark_dirty_if_current(session_data, machine_id).await;
|
||||
return;
|
||||
}
|
||||
_ = notify.notified() => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) async fn run_worker(session_data: std::sync::Weak<RwLock<SessionData>>) {
|
||||
while let Some(input) = wait_for_input(session_data.clone()).await {
|
||||
while let Some((input, validation_change_epoch)) = wait_for_input(session_data.clone()).await {
|
||||
let machine_id = input.machine_id;
|
||||
if let Err(error) = run_round(session_data.clone(), input).await {
|
||||
if let Err(error) = run_round(session_data.clone(), input, validation_change_epoch).await {
|
||||
tracing::warn!(
|
||||
?machine_id,
|
||||
%error,
|
||||
"webhook validation failed, will retry later"
|
||||
);
|
||||
tokio::time::sleep(retry_delay(machine_id)).await;
|
||||
mark_dirty_if_current(&session_data, machine_id).await;
|
||||
wait_for_retry_or_state_change(
|
||||
&session_data,
|
||||
machine_id,
|
||||
validation_change_epoch,
|
||||
retry_delay(machine_id),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -166,6 +240,7 @@ pub(super) async fn run_worker(session_data: std::sync::Weak<RwLock<SessionData>
|
||||
pub(super) async fn run_round(
|
||||
session_data: std::sync::Weak<RwLock<SessionData>>,
|
||||
input: WebhookValidationInput,
|
||||
validation_change_epoch: u64,
|
||||
) -> anyhow::Result<()> {
|
||||
let persisted_config_revision = persisted_config_revision_for_token(
|
||||
&input.storage,
|
||||
@@ -173,26 +248,53 @@ pub(super) async fn run_round(
|
||||
input.machine_id,
|
||||
)
|
||||
.await?;
|
||||
let validation = request_heartbeat_validation(
|
||||
&input.webhook_config,
|
||||
&input.client_url,
|
||||
persisted_config_revision.as_deref(),
|
||||
input.applied_config_revision.as_deref(),
|
||||
&input.req,
|
||||
input.machine_id,
|
||||
)
|
||||
.await?;
|
||||
let validation =
|
||||
request_heartbeat_validation(&input, persisted_config_revision.as_deref()).await?;
|
||||
|
||||
// The HTTP round trip can span heartbeats, revision updates, and
|
||||
// failed-instance changes. Results older than the current epoch are
|
||||
// discarded so a stale rejection cannot invalidate the session and a
|
||||
// stale success cannot emit outdated connection transitions.
|
||||
if !validation_results_are_current(&session_data, &input, validation_change_epoch).await {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let Some(validation) = validation else {
|
||||
apply_rejected(&session_data, &input).await;
|
||||
apply_rejected(&session_data, &input, validation_change_epoch).await;
|
||||
return Ok(());
|
||||
};
|
||||
|
||||
let user_id = resolve_user_id(&input.storage, &input.req.user_token).await?;
|
||||
apply_success(&session_data, input, validation, user_id).await;
|
||||
apply_success(
|
||||
&session_data,
|
||||
input,
|
||||
validation,
|
||||
user_id,
|
||||
validation_change_epoch,
|
||||
)
|
||||
.await;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn validation_results_are_current(
|
||||
session_data: &std::sync::Weak<RwLock<SessionData>>,
|
||||
input: &WebhookValidationInput,
|
||||
validation_change_epoch: u64,
|
||||
) -> bool {
|
||||
let Some(session_data) = session_data.upgrade() else {
|
||||
return false;
|
||||
};
|
||||
let data = session_data.read().await;
|
||||
if data.webhook_validation_change_epoch != validation_change_epoch {
|
||||
tracing::debug!(
|
||||
machine_id = %input.machine_id,
|
||||
"discard stale webhook validation result"
|
||||
);
|
||||
return false;
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
async fn mark_dirty_if_current(
|
||||
session_data: &std::sync::Weak<RwLock<SessionData>>,
|
||||
machine_id: uuid::Uuid,
|
||||
@@ -224,6 +326,7 @@ async fn mark_dirty_if_current(
|
||||
pub(super) async fn apply_rejected(
|
||||
session_data: &std::sync::Weak<RwLock<SessionData>>,
|
||||
input: &WebhookValidationInput,
|
||||
validation_change_epoch: u64,
|
||||
) {
|
||||
let Some(session_data) = session_data.upgrade() else {
|
||||
return;
|
||||
@@ -239,6 +342,13 @@ pub(super) async fn apply_rejected(
|
||||
}) {
|
||||
return;
|
||||
}
|
||||
if data.webhook_validation_change_epoch != validation_change_epoch {
|
||||
tracing::debug!(
|
||||
machine_id = %input.machine_id,
|
||||
"discard stale webhook validation rejection"
|
||||
);
|
||||
return;
|
||||
}
|
||||
tracing::info!(
|
||||
machine_id = %input.machine_id,
|
||||
client_url = %data.client_url,
|
||||
@@ -247,8 +357,6 @@ pub(super) async fn apply_rejected(
|
||||
data.auth_state = SessionAuthState::Invalid;
|
||||
data.webhook_validation_dirty = false;
|
||||
data.binding_version = None;
|
||||
data.applied_config_revision = None;
|
||||
data.pending_managed_config_delta = None;
|
||||
let storage_token = data.storage_token.clone();
|
||||
let disconnect_notification = storage_token.as_ref().and_then(|storage_token| {
|
||||
data.webhook_connected_binding_version
|
||||
@@ -282,6 +390,7 @@ pub(super) async fn apply_success(
|
||||
input: WebhookValidationInput,
|
||||
validation: WebhookHeartbeatValidation,
|
||||
user_id: i32,
|
||||
validation_change_epoch: u64,
|
||||
) {
|
||||
let WebhookHeartbeatValidation {
|
||||
config_revision: _,
|
||||
@@ -296,6 +405,7 @@ pub(super) async fn apply_success(
|
||||
notifier,
|
||||
disconnect_notification,
|
||||
connect_notification,
|
||||
validation_notify,
|
||||
runtime_req,
|
||||
session_epoch,
|
||||
) = {
|
||||
@@ -310,6 +420,13 @@ pub(super) async fn apply_success(
|
||||
) {
|
||||
return;
|
||||
}
|
||||
if data.webhook_validation_change_epoch != validation_change_epoch {
|
||||
tracing::debug!(
|
||||
machine_id = %input.machine_id,
|
||||
"discard stale webhook validation success"
|
||||
);
|
||||
return;
|
||||
}
|
||||
if matches!(data.auth_state, SessionAuthState::Invalid) {
|
||||
tracing::info!(
|
||||
machine_id = %input.machine_id,
|
||||
@@ -321,6 +438,20 @@ pub(super) async fn apply_success(
|
||||
|
||||
let previous_connected_binding_version = data.webhook_connected_binding_version;
|
||||
let client_url = data.client_url.clone();
|
||||
let is_new_storage_token = data.storage_token.is_none();
|
||||
let mut restored_runtime_revision = false;
|
||||
if is_new_storage_token {
|
||||
data.managed_runtime = input.storage.bind_managed_runtime_state(
|
||||
user_id,
|
||||
input.machine_id,
|
||||
SessionRpcService::heartbeat_runtime_id(&runtime_req),
|
||||
data.session_epoch,
|
||||
);
|
||||
let runtime = data.managed_runtime();
|
||||
restored_runtime_revision = runtime.applied_config_revision_known
|
||||
&& (!input.applied_config_revision_known
|
||||
|| input.applied_config_revision != runtime.applied_config_revision);
|
||||
}
|
||||
let storage_token = data.storage_token.get_or_insert_with(|| StorageToken {
|
||||
token: runtime_req.user_token.clone(),
|
||||
client_url,
|
||||
@@ -330,6 +461,16 @@ pub(super) async fn apply_success(
|
||||
let storage_token = storage_token.clone();
|
||||
data.auth_state = SessionAuthState::Authorized;
|
||||
data.binding_version = Some(binding_version);
|
||||
if is_new_storage_token {
|
||||
tracing::info!(
|
||||
machine_id = %input.machine_id,
|
||||
user_id,
|
||||
session_epoch = data.session_epoch,
|
||||
binding_version,
|
||||
client_url = %data.client_url,
|
||||
"session identity established"
|
||||
);
|
||||
}
|
||||
let should_notify_connected = previous_connected_binding_version != Some(binding_version);
|
||||
let disconnect_notification = previous_connected_binding_version
|
||||
.filter(|previous_binding_version| *previous_binding_version != binding_version)
|
||||
@@ -368,12 +509,15 @@ pub(super) async fn apply_success(
|
||||
binding_version: Some(binding_version),
|
||||
},
|
||||
});
|
||||
let validation_notify = restored_runtime_revision
|
||||
.then(|| SessionRpcService::mark_webhook_validation_state_changed_locked(&mut data));
|
||||
|
||||
(
|
||||
storage_token,
|
||||
data.notifier.clone(),
|
||||
disconnect_notification,
|
||||
connect_notification,
|
||||
validation_notify,
|
||||
runtime_req,
|
||||
data.session_epoch,
|
||||
)
|
||||
@@ -384,6 +528,10 @@ pub(super) async fn apply_success(
|
||||
.storage
|
||||
.update_session_client(storage_token, report_time, true, session_epoch);
|
||||
|
||||
if let Some(validation_notify) = validation_notify {
|
||||
validation_notify.notify_one();
|
||||
}
|
||||
|
||||
if disconnect_notification.is_some() || connect_notification.is_some() {
|
||||
wait_webhook_connection_transition(
|
||||
Arc::downgrade(&session_data),
|
||||
@@ -410,3 +558,301 @@ async fn wait_webhook_connection_transition(
|
||||
tracing::warn!(%error, "webhook connection transition task failed");
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
async fn validation_session(machine_id: uuid::Uuid) -> Arc<RwLock<SessionData>> {
|
||||
let storage = Storage::new(crate::db::Db::memory_db().await);
|
||||
let mut data = SessionData::new(
|
||||
storage.weak_ref(),
|
||||
url::Url::parse("http://127.0.0.1").unwrap(),
|
||||
None,
|
||||
Arc::new(crate::FeatureFlags::default()),
|
||||
Arc::new(crate::webhook::WebhookConfig::new(
|
||||
None, None, None, None, None,
|
||||
)),
|
||||
);
|
||||
data.req = Some(HeartbeatRequest {
|
||||
user_token: "token".to_string(),
|
||||
machine_id: Some(machine_id.into()),
|
||||
..Default::default()
|
||||
});
|
||||
data.auth_state = SessionAuthState::Authorized;
|
||||
Arc::new(RwLock::new(data))
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn reconnect_immediately_reports_restored_runtime_revision() {
|
||||
let storage = Storage::new(crate::db::Db::memory_db().await);
|
||||
let user_id = storage.db().auto_create_user("token").await.unwrap().id;
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let runtime_id = uuid::Uuid::new_v4();
|
||||
let shared = storage.bind_managed_runtime_state(user_id, machine_id, Some(runtime_id), 1);
|
||||
{
|
||||
let mut runtime = shared.lock().unwrap();
|
||||
runtime.applied_config_revision = Some("rev-applied".to_string());
|
||||
runtime.applied_config_revision_known = true;
|
||||
}
|
||||
let request = HeartbeatRequest {
|
||||
user_token: "token".to_string(),
|
||||
machine_id: Some(machine_id.into()),
|
||||
inst_id: Some(runtime_id.into()),
|
||||
..Default::default()
|
||||
};
|
||||
let mut data = SessionData::new(
|
||||
storage.weak_ref(),
|
||||
url::Url::parse("http://127.0.0.1").unwrap(),
|
||||
None,
|
||||
Arc::new(crate::FeatureFlags::default()),
|
||||
Arc::new(crate::webhook::WebhookConfig::new(
|
||||
None, None, None, None, None,
|
||||
)),
|
||||
);
|
||||
data.req = Some(request.clone());
|
||||
data.session_identity = Some(SessionRpcService::heartbeat_identity(&request, machine_id));
|
||||
data.session_epoch = 2;
|
||||
let session_data = Arc::new(RwLock::new(data));
|
||||
|
||||
let validation_change_epoch = session_data.read().await.webhook_validation_change_epoch;
|
||||
apply_success(
|
||||
&Arc::downgrade(&session_data),
|
||||
WebhookValidationInput {
|
||||
storage,
|
||||
webhook_config: Arc::new(crate::webhook::WebhookConfig::new(
|
||||
None, None, None, None, None,
|
||||
)),
|
||||
client_url: url::Url::parse("http://127.0.0.1").unwrap(),
|
||||
applied_config_revision: None,
|
||||
applied_config_revision_known: false,
|
||||
failed_instance_ids: Vec::new(),
|
||||
req: request,
|
||||
machine_id,
|
||||
},
|
||||
WebhookHeartbeatValidation {
|
||||
config_revision: "rev-applied".to_string(),
|
||||
binding_version: 1,
|
||||
},
|
||||
user_id,
|
||||
validation_change_epoch,
|
||||
)
|
||||
.await;
|
||||
|
||||
let data = session_data.read().await;
|
||||
assert!(Arc::ptr_eq(&data.managed_runtime, &shared));
|
||||
assert!(data.webhook_validation_dirty);
|
||||
assert_eq!(data.webhook_validation_change_epoch, 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn validation_input_carries_merged_failed_instance_ids() {
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let core_failed = uuid::Uuid::new_v4();
|
||||
let local_failed = uuid::Uuid::new_v4().to_string();
|
||||
let session_data = validation_session(machine_id).await;
|
||||
let storage = Storage::new(crate::db::Db::memory_db().await);
|
||||
{
|
||||
let mut data = session_data.write().await;
|
||||
data.storage = storage.weak_ref();
|
||||
data.req
|
||||
.as_mut()
|
||||
.unwrap()
|
||||
.failed_network_instances
|
||||
.push(core_failed.into());
|
||||
data.direct_run_failed_instance_ids
|
||||
.insert(local_failed.clone());
|
||||
data.webhook_validation_dirty = true;
|
||||
}
|
||||
|
||||
let (input, _) = wait_for_input(Arc::downgrade(&session_data))
|
||||
.await
|
||||
.expect("validation input");
|
||||
let mut expected = vec![core_failed.to_string(), local_failed];
|
||||
expected.sort_unstable();
|
||||
|
||||
assert_eq!(input.failed_instance_ids, expected);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stale_notification_does_not_bypass_validation_retry_delay() {
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let session_data = validation_session(machine_id).await;
|
||||
let notify = session_data.read().await.webhook_validation_notify.clone();
|
||||
notify.notify_one();
|
||||
let weak_session = Arc::downgrade(&session_data);
|
||||
|
||||
let wait =
|
||||
wait_for_retry_or_state_change(&weak_session, machine_id, 0, Duration::from_secs(10));
|
||||
tokio::pin!(wait);
|
||||
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_millis(20), &mut wait)
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
assert!(!session_data.read().await.webhook_validation_dirty);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn validation_retry_deadline_rearms_dirty_state() {
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let session_data = validation_session(machine_id).await;
|
||||
let weak_session = Arc::downgrade(&session_data);
|
||||
|
||||
tokio::time::timeout(
|
||||
Duration::from_secs(1),
|
||||
wait_for_retry_or_state_change(&weak_session, machine_id, 0, Duration::from_millis(20)),
|
||||
)
|
||||
.await
|
||||
.expect("retry deadline should eventually expire");
|
||||
assert!(session_data.read().await.webhook_validation_dirty);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn periodic_dirty_state_does_not_interrupt_validation_retry_delay() {
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let session_data = validation_session(machine_id).await;
|
||||
let weak_session = Arc::downgrade(&session_data);
|
||||
let wait =
|
||||
wait_for_retry_or_state_change(&weak_session, machine_id, 0, Duration::from_secs(10));
|
||||
tokio::pin!(wait);
|
||||
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_millis(20), &mut wait)
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
mark_dirty_if_current(&weak_session, machine_id).await;
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_millis(20), &mut wait)
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
assert!(session_data.read().await.webhook_validation_dirty);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn validation_state_change_interrupts_retry_delay() {
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let session_data = validation_session(machine_id).await;
|
||||
let weak_session = Arc::downgrade(&session_data);
|
||||
let wait =
|
||||
wait_for_retry_or_state_change(&weak_session, machine_id, 0, Duration::from_secs(10));
|
||||
tokio::pin!(wait);
|
||||
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_millis(20), &mut wait)
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
let notify = {
|
||||
let mut data = session_data.write().await;
|
||||
SessionRpcService::mark_webhook_validation_state_changed_locked(&mut data)
|
||||
};
|
||||
notify.notify_one();
|
||||
tokio::time::timeout(Duration::from_millis(500), &mut wait)
|
||||
.await
|
||||
.expect("validation state change should interrupt retry delay");
|
||||
assert!(session_data.read().await.webhook_validation_dirty);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn invalid_session_does_not_rearm_validation_retry() {
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let session_data = validation_session(machine_id).await;
|
||||
session_data.write().await.auth_state = SessionAuthState::Invalid;
|
||||
let weak_session = Arc::downgrade(&session_data);
|
||||
|
||||
tokio::time::timeout(
|
||||
Duration::from_millis(100),
|
||||
wait_for_retry_or_state_change(&weak_session, machine_id, 0, Duration::from_secs(10)),
|
||||
)
|
||||
.await
|
||||
.expect("invalid session should stop waiting");
|
||||
assert!(!session_data.read().await.webhook_validation_dirty);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn validation_results_require_current_epoch_or_live_session() {
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let session_data = validation_session(machine_id).await;
|
||||
let input = WebhookValidationInput {
|
||||
storage: Storage::new(crate::db::Db::memory_db().await),
|
||||
webhook_config: Arc::new(crate::webhook::WebhookConfig::new(
|
||||
None, None, None, None, None,
|
||||
)),
|
||||
client_url: url::Url::parse("http://127.0.0.1").unwrap(),
|
||||
applied_config_revision: None,
|
||||
applied_config_revision_known: false,
|
||||
failed_instance_ids: Vec::new(),
|
||||
req: HeartbeatRequest::default(),
|
||||
machine_id,
|
||||
};
|
||||
let weak_session = Arc::downgrade(&session_data);
|
||||
|
||||
assert!(
|
||||
validation_results_are_current(&weak_session, &input, 0).await,
|
||||
"matching epoch is current"
|
||||
);
|
||||
|
||||
session_data.write().await.webhook_validation_change_epoch = 7;
|
||||
assert!(
|
||||
!validation_results_are_current(&weak_session, &input, 0).await,
|
||||
"epoch bump discards stale results"
|
||||
);
|
||||
|
||||
drop(session_data);
|
||||
assert!(
|
||||
!validation_results_are_current(&weak_session, &input, 7).await,
|
||||
"dropped session discards results"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn apply_paths_discard_results_from_stale_epochs() {
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let session_data = validation_session(machine_id).await;
|
||||
session_data.write().await.webhook_connected_binding_version = Some(3);
|
||||
let input = WebhookValidationInput {
|
||||
storage: Storage::new(crate::db::Db::memory_db().await),
|
||||
webhook_config: Arc::new(crate::webhook::WebhookConfig::new(
|
||||
None, None, None, None, None,
|
||||
)),
|
||||
client_url: url::Url::parse("http://127.0.0.1").unwrap(),
|
||||
applied_config_revision: None,
|
||||
applied_config_revision_known: false,
|
||||
failed_instance_ids: Vec::new(),
|
||||
req: HeartbeatRequest {
|
||||
user_token: "token".to_string(),
|
||||
machine_id: Some(machine_id.into()),
|
||||
..Default::default()
|
||||
},
|
||||
machine_id,
|
||||
};
|
||||
|
||||
let stale_epoch = session_data.read().await.webhook_validation_change_epoch;
|
||||
session_data.write().await.webhook_validation_change_epoch = stale_epoch + 1;
|
||||
|
||||
apply_rejected(&Arc::downgrade(&session_data), &input, stale_epoch).await;
|
||||
let data = session_data.read().await;
|
||||
assert_eq!(data.auth_state, SessionAuthState::Authorized);
|
||||
assert_eq!(data.webhook_connected_binding_version, Some(3));
|
||||
drop(data);
|
||||
|
||||
apply_success(
|
||||
&Arc::downgrade(&session_data),
|
||||
input,
|
||||
WebhookHeartbeatValidation {
|
||||
config_revision: "rev-1".to_string(),
|
||||
binding_version: 9,
|
||||
},
|
||||
1,
|
||||
stale_epoch,
|
||||
)
|
||||
.await;
|
||||
let data = session_data.read().await;
|
||||
assert_eq!(data.binding_version, None);
|
||||
assert_eq!(data.webhook_connected_binding_version, Some(3));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,9 +1,14 @@
|
||||
use std::sync::{Arc, Weak};
|
||||
|
||||
use dashmap::DashMap;
|
||||
use dashmap::{DashMap, mapref::entry::Entry};
|
||||
|
||||
use crate::db::{Db, UserIdInDb};
|
||||
|
||||
use super::session::{
|
||||
ManagedConfigPersistedChange, ManagedConfigReconcileHint, ManagedRuntimeState,
|
||||
SharedManagedRuntimeState, record_managed_config_reconcile_hint,
|
||||
};
|
||||
|
||||
// use this to maintain Storage
|
||||
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
|
||||
pub struct StorageToken {
|
||||
@@ -21,12 +26,46 @@ struct ClientInfo {
|
||||
session_epoch: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
struct ManagedRuntimeContinuity {
|
||||
// Accepted trade-off: continuity assumes managed configuration is only
|
||||
// mutated through easytier-web. A local management RPC can make Core
|
||||
// drift without invalidating this state; detecting that would require a
|
||||
// Core-wide mutation generation outside this compatibility path.
|
||||
runtime_id: Option<uuid::Uuid>,
|
||||
session_epoch: u64,
|
||||
state: SharedManagedRuntimeState,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub struct StorageInner {
|
||||
user_clients_map: DashMap<UserIdInDb, DashMap<uuid::Uuid, ClientInfo>>,
|
||||
managed_runtime_states: DashMap<(UserIdInDb, uuid::Uuid), ManagedRuntimeContinuity>,
|
||||
pub db: Db,
|
||||
}
|
||||
|
||||
impl StorageInner {
|
||||
pub(super) fn owns_authorized_session(
|
||||
&self,
|
||||
stoken: &StorageToken,
|
||||
session_epoch: u64,
|
||||
) -> bool {
|
||||
self.user_clients_map
|
||||
.get(&stoken.user_id)
|
||||
.and_then(|clients| {
|
||||
clients.get(&stoken.machine_id).map(|client| {
|
||||
client.authorized
|
||||
&& client.session_epoch == session_epoch
|
||||
&& client.storage_token.token == stoken.token
|
||||
&& client.storage_token.client_url == stoken.client_url
|
||||
&& client.storage_token.user_id == stoken.user_id
|
||||
&& client.storage_token.machine_id == stoken.machine_id
|
||||
})
|
||||
})
|
||||
.unwrap_or(false)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct Storage(Arc<StorageInner>);
|
||||
pub type WeakRefStorage = Weak<StorageInner>;
|
||||
@@ -43,20 +82,149 @@ impl Storage {
|
||||
pub fn new(db: Db) -> Self {
|
||||
Storage(Arc::new(StorageInner {
|
||||
user_clients_map: DashMap::new(),
|
||||
managed_runtime_states: DashMap::new(),
|
||||
db,
|
||||
}))
|
||||
}
|
||||
|
||||
pub(super) fn bind_managed_runtime_state(
|
||||
&self,
|
||||
user_id: UserIdInDb,
|
||||
machine_id: uuid::Uuid,
|
||||
runtime_id: Option<uuid::Uuid>,
|
||||
session_epoch: u64,
|
||||
) -> SharedManagedRuntimeState {
|
||||
let new_state = || Arc::new(std::sync::Mutex::new(ManagedRuntimeState::default()));
|
||||
match self.0.managed_runtime_states.entry((user_id, machine_id)) {
|
||||
Entry::Occupied(mut entry) => {
|
||||
let current = entry.get();
|
||||
if runtime_id.is_some() && current.runtime_id == runtime_id {
|
||||
let state = current.state.clone();
|
||||
if session_epoch > current.session_epoch {
|
||||
entry.get_mut().session_epoch = session_epoch;
|
||||
}
|
||||
return state;
|
||||
}
|
||||
if session_epoch < current.session_epoch {
|
||||
return new_state();
|
||||
}
|
||||
let state = new_state();
|
||||
entry.insert(ManagedRuntimeContinuity {
|
||||
runtime_id,
|
||||
session_epoch,
|
||||
state: state.clone(),
|
||||
});
|
||||
state
|
||||
}
|
||||
Entry::Vacant(entry) => {
|
||||
let state = new_state();
|
||||
entry.insert(ManagedRuntimeContinuity {
|
||||
runtime_id,
|
||||
session_epoch,
|
||||
state: state.clone(),
|
||||
});
|
||||
state
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn current_managed_runtime_state(
|
||||
&self,
|
||||
user_id: UserIdInDb,
|
||||
machine_id: uuid::Uuid,
|
||||
) -> Option<SharedManagedRuntimeState> {
|
||||
self.0
|
||||
.managed_runtime_states
|
||||
.get(&(user_id, machine_id))
|
||||
.map(|entry| entry.state.clone())
|
||||
}
|
||||
|
||||
pub(super) fn record_full_managed_config_change(
|
||||
&self,
|
||||
user_id: UserIdInDb,
|
||||
machine_id: uuid::Uuid,
|
||||
config_revision: Option<&str>,
|
||||
) -> bool {
|
||||
let Some(state) = self.current_managed_runtime_state(user_id, machine_id) else {
|
||||
return false;
|
||||
};
|
||||
let mut state = state.lock().expect("managed runtime state lock poisoned");
|
||||
// Unrevisioned legacy updates carry no revision to compare against,
|
||||
// so they always invalidate: record the hint and bump the epoch that
|
||||
// fences in-flight reconcile rounds.
|
||||
let target_already_applied = config_revision
|
||||
.is_some_and(|revision| state.applied_config_revision.as_deref() == Some(revision));
|
||||
if target_already_applied && state.pending_managed_config_reconcile.is_none() {
|
||||
return false;
|
||||
}
|
||||
if !target_already_applied {
|
||||
record_managed_config_reconcile_hint(
|
||||
&mut state.pending_managed_config_reconcile,
|
||||
ManagedConfigReconcileHint::Full,
|
||||
);
|
||||
}
|
||||
state.runtime_config_epoch = state.runtime_config_epoch.wrapping_add(1);
|
||||
true
|
||||
}
|
||||
|
||||
pub(super) fn record_patch_managed_config_change(
|
||||
&self,
|
||||
user_id: UserIdInDb,
|
||||
machine_id: uuid::Uuid,
|
||||
change: ManagedConfigPersistedChange,
|
||||
) -> bool {
|
||||
let Some(state) = self.current_managed_runtime_state(user_id, machine_id) else {
|
||||
return false;
|
||||
};
|
||||
let mut state = state.lock().expect("managed runtime state lock poisoned");
|
||||
let target_already_applied =
|
||||
state.applied_config_revision.as_deref() == Some(change.target_revision.as_str());
|
||||
if target_already_applied && state.pending_managed_config_reconcile.is_none() {
|
||||
return false;
|
||||
}
|
||||
if !target_already_applied {
|
||||
record_managed_config_reconcile_hint(
|
||||
&mut state.pending_managed_config_reconcile,
|
||||
ManagedConfigReconcileHint::Dirty {
|
||||
expected_revision: change.expected_revision,
|
||||
target_revision: change.target_revision,
|
||||
instance_ids: change.dirty_instance_ids,
|
||||
},
|
||||
);
|
||||
}
|
||||
state.runtime_config_epoch = state.runtime_config_epoch.wrapping_add(1);
|
||||
true
|
||||
}
|
||||
|
||||
pub(super) fn invalidate_managed_runtime_state(
|
||||
&self,
|
||||
user_id: UserIdInDb,
|
||||
machine_id: uuid::Uuid,
|
||||
) -> bool {
|
||||
let Some(state) = self.current_managed_runtime_state(user_id, machine_id) else {
|
||||
return false;
|
||||
};
|
||||
let mut state = state.lock().expect("managed runtime state lock poisoned");
|
||||
state.applied_config_revision = None;
|
||||
state.applied_config_revision_known = true;
|
||||
state.known_runtime_base_revision = None;
|
||||
state.pending_managed_config_reconcile = Some(ManagedConfigReconcileHint::Full);
|
||||
state.runtime_config_epoch = state.runtime_config_epoch.wrapping_add(1);
|
||||
state.runtime_config_cache_epoch = state.runtime_config_cache_epoch.wrapping_add(1);
|
||||
true
|
||||
}
|
||||
|
||||
fn remove_client_info_map(
|
||||
map: &DashMap<uuid::Uuid, ClientInfo>,
|
||||
stoken: &StorageToken,
|
||||
session_epoch: u64,
|
||||
) {
|
||||
) -> bool {
|
||||
map.remove_if(&stoken.machine_id, |_, v| {
|
||||
v.storage_token.client_url == stoken.client_url
|
||||
&& v.storage_token.user_id == stoken.user_id
|
||||
&& v.session_epoch == session_epoch
|
||||
});
|
||||
})
|
||||
.is_some()
|
||||
}
|
||||
|
||||
fn update_client_info_map(map: &DashMap<uuid::Uuid, ClientInfo>, client_info: &ClientInfo) {
|
||||
@@ -99,6 +267,20 @@ impl Storage {
|
||||
authorized: bool,
|
||||
session_epoch: u64,
|
||||
) {
|
||||
let mut continuity = self
|
||||
.0
|
||||
.managed_runtime_states
|
||||
.entry((stoken.user_id, stoken.machine_id))
|
||||
.or_insert_with(|| ManagedRuntimeContinuity {
|
||||
runtime_id: None,
|
||||
session_epoch,
|
||||
state: Arc::new(std::sync::Mutex::new(ManagedRuntimeState::default())),
|
||||
});
|
||||
if session_epoch < continuity.session_epoch {
|
||||
return;
|
||||
}
|
||||
continuity.session_epoch = session_epoch;
|
||||
|
||||
let inner = self.0.user_clients_map.entry(stoken.user_id).or_default();
|
||||
|
||||
let client_info = ClientInfo {
|
||||
@@ -111,16 +293,30 @@ impl Storage {
|
||||
}
|
||||
|
||||
pub fn remove_client(&self, stoken: &StorageToken) {
|
||||
self.remove_session_client(stoken, 0);
|
||||
let _ = self.remove_session_client(stoken, 0);
|
||||
}
|
||||
|
||||
pub(super) fn remove_session_client(&self, stoken: &StorageToken, session_epoch: u64) {
|
||||
pub(super) fn remove_session_client(&self, stoken: &StorageToken, session_epoch: u64) -> bool {
|
||||
let Some(mut continuity) = self
|
||||
.0
|
||||
.managed_runtime_states
|
||||
.get_mut(&(stoken.user_id, stoken.machine_id))
|
||||
else {
|
||||
return false;
|
||||
};
|
||||
if session_epoch < continuity.session_epoch {
|
||||
return false;
|
||||
}
|
||||
continuity.session_epoch = session_epoch;
|
||||
|
||||
let mut removed = false;
|
||||
self.0
|
||||
.user_clients_map
|
||||
.remove_if(&stoken.user_id, |_, set| {
|
||||
Self::remove_client_info_map(set, stoken, session_epoch);
|
||||
removed = Self::remove_client_info_map(set, stoken, session_epoch);
|
||||
set.is_empty()
|
||||
});
|
||||
removed
|
||||
}
|
||||
|
||||
pub fn weak_ref(&self) -> WeakRefStorage {
|
||||
@@ -262,6 +458,36 @@ mod tests {
|
||||
assert_eq!(storage.get_client_url_by_machine_id(2, &machine_id), None);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unrevisioned_full_change_always_fences_runtime_epochs() {
|
||||
let storage = Storage::new(Db::memory_db().await);
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let state = storage.bind_managed_runtime_state(7, machine_id, None, 1);
|
||||
{
|
||||
let mut runtime = state.lock().unwrap();
|
||||
runtime.applied_config_revision = Some("rev-1".to_string());
|
||||
runtime.applied_config_revision_known = true;
|
||||
}
|
||||
|
||||
assert!(storage.record_full_managed_config_change(7, machine_id, None));
|
||||
{
|
||||
let runtime = state.lock().unwrap();
|
||||
assert_eq!(runtime.runtime_config_epoch, 1);
|
||||
assert!(matches!(
|
||||
runtime.pending_managed_config_reconcile,
|
||||
Some(ManagedConfigReconcileHint::Full)
|
||||
));
|
||||
}
|
||||
|
||||
// Repeated unrevisioned updates keep fencing, while a revision that
|
||||
// is already applied with no pending hint stays a no-op.
|
||||
assert!(storage.record_full_managed_config_change(7, machine_id, None));
|
||||
assert_eq!(state.lock().unwrap().runtime_config_epoch, 2);
|
||||
state.lock().unwrap().pending_managed_config_reconcile = None;
|
||||
assert!(!storage.record_full_managed_config_change(7, machine_id, Some("rev-1")));
|
||||
assert_eq!(state.lock().unwrap().runtime_config_epoch, 2);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn newer_session_epoch_owns_route_until_it_is_removed() {
|
||||
let storage = Storage::new(Db::memory_db().await);
|
||||
@@ -273,18 +499,127 @@ mod tests {
|
||||
storage.update_session_client(current.clone(), 20, true, 2);
|
||||
storage.update_session_client(old.clone(), 30, true, 1);
|
||||
|
||||
assert!(!storage.0.owns_authorized_session(&old, 1));
|
||||
assert!(storage.0.owns_authorized_session(¤t, 2));
|
||||
assert_eq!(
|
||||
storage.get_client_url_by_machine_id(1, &machine_id),
|
||||
Some(current.client_url.clone())
|
||||
);
|
||||
|
||||
storage.remove_session_client(&old, 1);
|
||||
assert!(!storage.remove_session_client(&old, 1));
|
||||
assert_eq!(
|
||||
storage.get_client_url_by_machine_id(1, &machine_id),
|
||||
Some(current.client_url.clone())
|
||||
);
|
||||
storage.remove_session_client(¤t, 2);
|
||||
assert!(storage.remove_session_client(¤t, 2));
|
||||
assert_eq!(storage.get_client_url_by_machine_id(1, &machine_id), None);
|
||||
|
||||
storage.update_session_client(old.clone(), 40, true, 1);
|
||||
assert!(!storage.0.owns_authorized_session(&old, 1));
|
||||
assert_eq!(storage.get_client_url_by_machine_id(1, &machine_id), None);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn same_runtime_reuses_state_across_sessions() {
|
||||
let storage = Storage::new(Db::memory_db().await);
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let runtime_id = uuid::Uuid::new_v4();
|
||||
|
||||
let first = storage.bind_managed_runtime_state(1, machine_id, Some(runtime_id), 1);
|
||||
{
|
||||
let mut state = first.lock().unwrap();
|
||||
state.applied_config_revision = Some("rev-a".to_string());
|
||||
state.applied_config_revision_known = true;
|
||||
}
|
||||
|
||||
let reconnected = storage.bind_managed_runtime_state(1, machine_id, Some(runtime_id), 2);
|
||||
|
||||
assert!(Arc::ptr_eq(&first, &reconnected));
|
||||
let state = reconnected.lock().unwrap();
|
||||
assert_eq!(state.applied_config_revision.as_deref(), Some("rev-a"));
|
||||
assert!(state.applied_config_revision_known);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn changed_or_missing_runtime_id_starts_with_unknown_state() {
|
||||
let storage = Storage::new(Db::memory_db().await);
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let first =
|
||||
storage.bind_managed_runtime_state(1, machine_id, Some(uuid::Uuid::new_v4()), 1);
|
||||
{
|
||||
let mut state = first.lock().unwrap();
|
||||
state.applied_config_revision = Some("rev-a".to_string());
|
||||
state.applied_config_revision_known = true;
|
||||
}
|
||||
|
||||
let restarted =
|
||||
storage.bind_managed_runtime_state(1, machine_id, Some(uuid::Uuid::new_v4()), 2);
|
||||
assert!(!Arc::ptr_eq(&first, &restarted));
|
||||
assert!(!restarted.lock().unwrap().applied_config_revision_known);
|
||||
|
||||
let legacy = storage.bind_managed_runtime_state(1, machine_id, None, 3);
|
||||
assert!(!Arc::ptr_eq(&restarted, &legacy));
|
||||
let legacy_reconnected = storage.bind_managed_runtime_state(1, machine_id, None, 4);
|
||||
assert!(!Arc::ptr_eq(&legacy, &legacy_reconnected));
|
||||
assert!(
|
||||
!legacy_reconnected
|
||||
.lock()
|
||||
.unwrap()
|
||||
.applied_config_revision_known
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn stale_session_cannot_replace_current_runtime_state() {
|
||||
let storage = Storage::new(Db::memory_db().await);
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let current_runtime_id = uuid::Uuid::new_v4();
|
||||
let current =
|
||||
storage.bind_managed_runtime_state(1, machine_id, Some(current_runtime_id), 2);
|
||||
|
||||
let stale =
|
||||
storage.bind_managed_runtime_state(1, machine_id, Some(uuid::Uuid::new_v4()), 1);
|
||||
assert!(!Arc::ptr_eq(¤t, &stale));
|
||||
|
||||
let reconnected =
|
||||
storage.bind_managed_runtime_state(1, machine_id, Some(current_runtime_id), 3);
|
||||
assert!(Arc::ptr_eq(¤t, &reconnected));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn patch_hint_survives_disconnect_until_same_runtime_reconnects() {
|
||||
let storage = Storage::new(Db::memory_db().await);
|
||||
let machine_id = uuid::Uuid::new_v4();
|
||||
let runtime_id = uuid::Uuid::new_v4();
|
||||
let state = storage.bind_managed_runtime_state(1, machine_id, Some(runtime_id), 1);
|
||||
{
|
||||
let mut state = state.lock().unwrap();
|
||||
state.applied_config_revision = Some("rev-a".to_string());
|
||||
state.applied_config_revision_known = true;
|
||||
state.known_runtime_base_revision = Some("rev-a".to_string());
|
||||
}
|
||||
|
||||
assert!(storage.record_patch_managed_config_change(
|
||||
1,
|
||||
machine_id,
|
||||
ManagedConfigPersistedChange {
|
||||
expected_revision: "rev-a".to_string(),
|
||||
target_revision: "rev-b".to_string(),
|
||||
dirty_instance_ids: std::collections::HashSet::from(["instance-a".to_string(),]),
|
||||
},
|
||||
));
|
||||
|
||||
let reconnected = storage.bind_managed_runtime_state(1, machine_id, Some(runtime_id), 2);
|
||||
let state = reconnected.lock().unwrap();
|
||||
assert_eq!(
|
||||
state.pending_managed_config_reconcile,
|
||||
Some(ManagedConfigReconcileHint::Dirty {
|
||||
expected_revision: "rev-a".to_string(),
|
||||
target_revision: "rev-b".to_string(),
|
||||
instance_ids: std::collections::HashSet::from(["instance-a".to_string(),]),
|
||||
})
|
||||
);
|
||||
assert_eq!(state.runtime_config_epoch, 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
#[macro_use]
|
||||
extern crate rust_i18n;
|
||||
|
||||
use std::net::IpAddr;
|
||||
use std::sync::Arc;
|
||||
use std::{net::IpAddr, time::Duration};
|
||||
|
||||
use clap::Parser;
|
||||
use easytier::tunnel::websocket::WsTunnelListener;
|
||||
@@ -116,11 +116,19 @@ struct Cli {
|
||||
#[arg(
|
||||
long,
|
||||
env = "ET_HEARTBEAT_MIN_RESPONSE_MS",
|
||||
default_value = "0",
|
||||
default_value = "3500",
|
||||
help = t!("cli.heartbeat_min_response_ms").to_string(),
|
||||
)]
|
||||
heartbeat_min_response_ms: u64,
|
||||
|
||||
#[arg(
|
||||
long,
|
||||
env = "ET_HEARTBEAT_TIMEOUT_MS",
|
||||
default_value = "15000",
|
||||
help = t!("cli.heartbeat_timeout_ms").to_string(),
|
||||
)]
|
||||
heartbeat_timeout_ms: u64,
|
||||
|
||||
#[cfg(feature = "embed")]
|
||||
#[arg(
|
||||
long,
|
||||
@@ -288,7 +296,21 @@ async fn main() {
|
||||
setup_panic_handler();
|
||||
|
||||
let cli = Cli::parse();
|
||||
log::init(&cli, false).unwrap();
|
||||
log::init_with_default_console_targets(&cli, false, &["CORE", "easytier_web"]).unwrap();
|
||||
tracing::info!(
|
||||
version = EASYTIER_VERSION,
|
||||
web_instance_id = ?cli.webhook.web_instance_id,
|
||||
api_address = %cli.api_server_addr,
|
||||
api_port = cli.api_server_port,
|
||||
config_protocol = %cli.config_server_protocol,
|
||||
config_port = cli.config_server_port,
|
||||
heartbeat_min_response_ms = cli.heartbeat_min_response_ms,
|
||||
heartbeat_timeout_ms = cli.heartbeat_timeout_ms,
|
||||
webhook_enabled = cli.webhook.webhook_url.as_deref().is_some_and(|url| !url.trim().is_empty()),
|
||||
rust_log_override = std::env::var_os("RUST_LOG").is_some(),
|
||||
console_log_override = cli.console_log_level.is_some(),
|
||||
"easytier-web starting"
|
||||
);
|
||||
|
||||
// Validate OIDC configuration: check split-deploy specific requirements
|
||||
// Basic OIDC parameter validation is handled in OidcConfig::from_params
|
||||
@@ -326,10 +348,18 @@ async fn main() {
|
||||
cli.webhook.web_instance_id,
|
||||
cli.webhook.web_instance_api_base_url,
|
||||
));
|
||||
let heartbeat_policy = client_manager::HeartbeatPolicy::from_millis(
|
||||
cli.heartbeat_min_response_ms,
|
||||
cli.heartbeat_timeout_ms,
|
||||
)
|
||||
.unwrap_or_else(|error| {
|
||||
eprintln!("Invalid heartbeat configuration: {error}");
|
||||
std::process::exit(2);
|
||||
});
|
||||
let mut mgr = client_manager::ClientManager::new(
|
||||
db.clone(),
|
||||
cli.geoip_db,
|
||||
Duration::from_millis(cli.heartbeat_min_response_ms),
|
||||
heartbeat_policy,
|
||||
feature_flags.clone(),
|
||||
webhook_config.clone(),
|
||||
);
|
||||
|
||||
+112
-9
@@ -308,6 +308,8 @@ pub struct ValidateTokenRequest {
|
||||
pub persisted_config_revision: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub applied_config_revision: Option<String>,
|
||||
pub applied_config_revision_known: bool,
|
||||
pub failed_instance_ids: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
@@ -350,6 +352,26 @@ pub struct NodeDisconnectedRequest {
|
||||
pub binding_version: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub(crate) enum WebhookDeliveryError {
|
||||
#[error("webhook endpoint is invalid: {0}")]
|
||||
Configuration(#[source] anyhow::Error),
|
||||
#[error("webhook request failed: {0}")]
|
||||
Transport(#[source] reqwest::Error),
|
||||
#[error("webhook returned status {0}")]
|
||||
ResponseStatus(reqwest::StatusCode),
|
||||
}
|
||||
|
||||
impl WebhookDeliveryError {
|
||||
pub(crate) fn is_retryable(&self) -> bool {
|
||||
match self {
|
||||
Self::Transport(_) => true,
|
||||
Self::ResponseStatus(status) => status.is_server_error(),
|
||||
Self::Configuration(_) => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --- Webhook client ---
|
||||
|
||||
impl WebhookConfig {
|
||||
@@ -384,7 +406,10 @@ impl WebhookConfig {
|
||||
http_timeout: Duration,
|
||||
) -> anyhow::Result<ValidateTokenResponse> {
|
||||
let url = self.webhook_endpoint("validate-token")?;
|
||||
let started_at = Instant::now();
|
||||
let permit = self.validate_limiter.acquire().await;
|
||||
let queue_elapsed = started_at.elapsed();
|
||||
let http_started_at = Instant::now();
|
||||
let ret = match tokio::time::timeout(http_timeout, async {
|
||||
let resp = self
|
||||
.client
|
||||
@@ -406,25 +431,45 @@ impl WebhookConfig {
|
||||
Err(_) => Err(anyhow::anyhow!("webhook validate-token timed out")),
|
||||
};
|
||||
permit.complete(ret.is_ok());
|
||||
let http_elapsed = http_started_at.elapsed();
|
||||
let elapsed = started_at.elapsed();
|
||||
if queue_elapsed >= Duration::from_secs(1) || http_elapsed >= VALIDATE_TOKEN_SLOW_THRESHOLD
|
||||
{
|
||||
tracing::warn!(
|
||||
machine_id = %req.machine_id,
|
||||
queue_ms = queue_elapsed.as_millis(),
|
||||
http_ms = http_elapsed.as_millis(),
|
||||
elapsed_ms = elapsed.as_millis(),
|
||||
success = ret.is_ok(),
|
||||
"validate-token completed slowly"
|
||||
);
|
||||
}
|
||||
ret
|
||||
}
|
||||
|
||||
/// Notify the webhook receiver that a node has connected.
|
||||
pub async fn notify_node_connected(&self, req: &NodeConnectedRequest) {
|
||||
pub(crate) async fn notify_node_connected(
|
||||
&self,
|
||||
req: &NodeConnectedRequest,
|
||||
) -> Result<(), WebhookDeliveryError> {
|
||||
if !self.is_enabled() {
|
||||
return;
|
||||
return Ok(());
|
||||
}
|
||||
let Ok(url) = self.webhook_endpoint("webhook/node-connected") else {
|
||||
tracing::warn!("skip node-connected webhook because webhook_url is not configured");
|
||||
return;
|
||||
};
|
||||
let _ = self
|
||||
let url = self
|
||||
.webhook_endpoint("webhook/node-connected")
|
||||
.map_err(WebhookDeliveryError::Configuration)?;
|
||||
let response = self
|
||||
.client
|
||||
.post(&url)
|
||||
.header("X-Internal-Auth", self.webhook_auth_secret())
|
||||
.json(req)
|
||||
.send()
|
||||
.await;
|
||||
.await
|
||||
.map_err(WebhookDeliveryError::Transport)?;
|
||||
if !response.status().is_success() {
|
||||
return Err(WebhookDeliveryError::ResponseStatus(response.status()));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Notify the webhook receiver that a node has disconnected.
|
||||
@@ -436,13 +481,35 @@ impl WebhookConfig {
|
||||
tracing::warn!("skip node-disconnected webhook because webhook_url is not configured");
|
||||
return;
|
||||
};
|
||||
let _ = self
|
||||
let started_at = Instant::now();
|
||||
let result = self
|
||||
.client
|
||||
.post(&url)
|
||||
.header("X-Internal-Auth", self.webhook_auth_secret())
|
||||
.json(req)
|
||||
.send()
|
||||
.await;
|
||||
let elapsed = started_at.elapsed();
|
||||
match result {
|
||||
Err(error) => tracing::warn!(
|
||||
machine_id = %req.machine_id,
|
||||
elapsed_ms = elapsed.as_millis(),
|
||||
%error,
|
||||
"node-disconnected webhook delivery failed"
|
||||
),
|
||||
Ok(response) if !response.status().is_success() => tracing::warn!(
|
||||
machine_id = %req.machine_id,
|
||||
status = %response.status(),
|
||||
elapsed_ms = elapsed.as_millis(),
|
||||
"node-disconnected webhook returned failure status"
|
||||
),
|
||||
Ok(_) if elapsed >= VALIDATE_TOKEN_SLOW_THRESHOLD => tracing::warn!(
|
||||
machine_id = %req.machine_id,
|
||||
elapsed_ms = elapsed.as_millis(),
|
||||
"node-disconnected webhook completed slowly"
|
||||
),
|
||||
Ok(_) => {}
|
||||
}
|
||||
}
|
||||
|
||||
fn webhook_auth_secret(&self) -> &str {
|
||||
@@ -461,6 +528,21 @@ mod tests {
|
||||
use axum::{Json, Router, routing::post};
|
||||
use serde_json::json;
|
||||
|
||||
fn node_connected_request() -> NodeConnectedRequest {
|
||||
NodeConnectedRequest {
|
||||
machine_id: uuid::Uuid::new_v4().to_string(),
|
||||
token: "token".to_string(),
|
||||
user_id: Some(1),
|
||||
hostname: String::new(),
|
||||
version: String::new(),
|
||||
os_type: None,
|
||||
os_version: None,
|
||||
os_distribution: None,
|
||||
web_instance_id: None,
|
||||
binding_version: Some(1),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn adaptive_validate_limiter_increases_under_queue_pressure() {
|
||||
let now = Instant::now();
|
||||
@@ -736,6 +818,8 @@ mod tests {
|
||||
web_instance_api_base_url: None,
|
||||
persisted_config_revision: None,
|
||||
applied_config_revision: None,
|
||||
applied_config_revision_known: false,
|
||||
failed_instance_ids: Vec::new(),
|
||||
};
|
||||
validate_webhook
|
||||
.validate_token_with_http_timeout(&req, Duration::from_millis(20))
|
||||
@@ -773,4 +857,23 @@ mod tests {
|
||||
assert!(resp.valid);
|
||||
assert!(resp.config_revision.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn node_connected_transport_error_is_retryable() {
|
||||
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
drop(listener);
|
||||
let webhook = WebhookConfig::new(Some(format!("http://{addr}")), None, None, None, None);
|
||||
|
||||
let error = tokio::time::timeout(
|
||||
Duration::from_secs(1),
|
||||
webhook.notify_node_connected(&node_connected_request()),
|
||||
)
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap_err();
|
||||
|
||||
assert!(matches!(error, WebhookDeliveryError::Transport(_)));
|
||||
assert!(error.is_retryable());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,6 +5,14 @@ use crate::common::config::LoggingConfigLoader;
|
||||
use super::{FileSink, Logger, TargetFilter, install, parse_level};
|
||||
|
||||
pub fn init(config: impl LoggingConfigLoader, reload: bool) -> anyhow::Result<()> {
|
||||
init_with_default_console_targets(config, reload, &[super::LOG_TARGET])
|
||||
}
|
||||
|
||||
pub fn init_with_default_console_targets(
|
||||
config: impl LoggingConfigLoader,
|
||||
reload: bool,
|
||||
default_targets: &[&str],
|
||||
) -> anyhow::Result<()> {
|
||||
let console_config = config.get_console_logger_config();
|
||||
let console_level = console_config
|
||||
.level
|
||||
@@ -12,7 +20,7 @@ pub fn init(config: impl LoggingConfigLoader, reload: bool) -> anyhow::Result<()
|
||||
.map(parse_level)
|
||||
.transpose()
|
||||
.context("invalid console log level")?;
|
||||
let console = TargetFilter::console(console_level)?;
|
||||
let console = TargetFilter::console_with_default_targets(console_level, default_targets)?;
|
||||
let file = FileSink::from_config(config.get_file_logger_config(), reload)?;
|
||||
|
||||
install(Logger::new(console, file))
|
||||
|
||||
@@ -20,7 +20,7 @@ mod file;
|
||||
#[cfg(feature = "management")]
|
||||
mod management;
|
||||
#[cfg(feature = "management")]
|
||||
pub use management::init;
|
||||
pub use management::{init, init_with_default_console_targets};
|
||||
mod tracing_backend;
|
||||
|
||||
use file::FileSink;
|
||||
@@ -107,6 +107,13 @@ struct TargetFilter {
|
||||
|
||||
impl TargetFilter {
|
||||
fn console(level: Option<LevelFilter>) -> anyhow::Result<Self> {
|
||||
Self::console_with_default_targets(level, &[LOG_TARGET])
|
||||
}
|
||||
|
||||
fn console_with_default_targets(
|
||||
level: Option<LevelFilter>,
|
||||
default_targets: &[&str],
|
||||
) -> anyhow::Result<Self> {
|
||||
if level == Some(LevelFilter::Off) {
|
||||
return Ok(Self::off());
|
||||
}
|
||||
@@ -115,7 +122,10 @@ impl TargetFilter {
|
||||
Some(level) => Self::with_default(level),
|
||||
None => Self {
|
||||
default: LevelFilter::Off,
|
||||
targets: vec![(LOG_TARGET.into(), LevelFilter::Info)],
|
||||
targets: default_targets
|
||||
.iter()
|
||||
.map(|target| ((*target).into(), LevelFilter::Info))
|
||||
.collect(),
|
||||
},
|
||||
};
|
||||
Self::from_environment(fallback)
|
||||
@@ -452,6 +462,20 @@ mod tests {
|
||||
assert!(!filter.enabled("other", Level::Error));
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn additional_default_console_targets_are_scoped() {
|
||||
let _env = EnvVarGuard::set(None);
|
||||
let filter =
|
||||
TargetFilter::console_with_default_targets(None, &[LOG_TARGET, "easytier_web"])
|
||||
.unwrap();
|
||||
|
||||
assert!(filter.enabled("CORE::peer", Level::Info));
|
||||
assert!(filter.enabled("easytier_web::client_manager", Level::Info));
|
||||
assert!(!filter.enabled("easytier_web", Level::Debug));
|
||||
assert!(!filter.enabled("other", Level::Error));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rust_log_supports_global_and_target_levels() {
|
||||
let filter = TargetFilter::parse("warn,easytier_core=debug,hyper=off")
|
||||
|
||||
@@ -13,6 +13,12 @@ use easytier_core::{
|
||||
process_runtime::CoreProcessRuntime,
|
||||
};
|
||||
|
||||
#[cfg(feature = "wireguard")]
|
||||
use crate::{
|
||||
common::config::{VpnPortalClientConfig, VpnPortalConfig},
|
||||
tests::three_node::{run_wireguard_client, wireguard_ifname},
|
||||
vpn_portal::wireguard::test_wireguard_keys,
|
||||
};
|
||||
use crate::{
|
||||
common::{
|
||||
config::{ConfigLoader, NetworkIdentity, PeerConfig, TomlConfigLoader},
|
||||
@@ -23,6 +29,12 @@ use crate::{
|
||||
tests::three_node::{generate_secure_mode_config, generate_secure_mode_config_with_key},
|
||||
tunnel::common::tests::wait_for_condition,
|
||||
};
|
||||
#[cfg(feature = "wireguard")]
|
||||
use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD};
|
||||
#[cfg(feature = "wireguard")]
|
||||
use defguard_wireguard_rs::key::Key;
|
||||
#[cfg(feature = "wireguard")]
|
||||
use easytier_core::gateway::vpn_portal::PortalClientState;
|
||||
|
||||
use super::{
|
||||
InstanceTestExt as _, add_ns_to_bridge, create_netns, del_netns, drop_insts, ping_test,
|
||||
@@ -96,13 +108,8 @@ async fn set_prefer_peer_relay(inst: &Instance, prefer_peer_relay: bool) {
|
||||
);
|
||||
}
|
||||
|
||||
fn forwarded_data_packets(inst: &Instance) -> u64 {
|
||||
let labels = LabelSet::new().with_label_type(LabelType::NetworkName(
|
||||
inst.get_global_ctx()
|
||||
.get_network_identity()
|
||||
.network_name
|
||||
.clone(),
|
||||
));
|
||||
fn forwarded_data_packets_for_network(inst: &Instance, network_name: &str) -> u64 {
|
||||
let labels = LabelSet::new().with_label_type(LabelType::NetworkName(network_name.to_owned()));
|
||||
inst.get_core_instance()
|
||||
.metric_snapshots()
|
||||
.into_iter()
|
||||
@@ -112,6 +119,13 @@ fn forwarded_data_packets(inst: &Instance) -> u64 {
|
||||
.map_or(0, |metric| metric.value)
|
||||
}
|
||||
|
||||
fn forwarded_data_packets(inst: &Instance) -> u64 {
|
||||
forwarded_data_packets_for_network(
|
||||
inst,
|
||||
&inst.get_global_ctx().get_network_identity().network_name,
|
||||
)
|
||||
}
|
||||
|
||||
async fn assert_ping_forwarded_by(
|
||||
src_ns: &str,
|
||||
dst_ip: &str,
|
||||
@@ -714,6 +728,191 @@ async fn credential_peers_p2p_to_need_p2p_admin_through_public_server(
|
||||
.await;
|
||||
}
|
||||
|
||||
#[cfg(feature = "wireguard")]
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn credential_peer_reconnects_to_admin_with_portal_client_online() {
|
||||
prepare_credential_network();
|
||||
let process_runtime = CoreProcessRuntime::new();
|
||||
|
||||
let public_server_config = create_public_server_config();
|
||||
let mut public_server_flags = public_server_config.get_flags();
|
||||
public_server_flags.disable_relay_data = true;
|
||||
public_server_config.set_flags(public_server_flags);
|
||||
let mut public_server_inst =
|
||||
Instance::new_with_process_runtime(public_server_config, process_runtime.clone());
|
||||
public_server_inst.run().await.unwrap();
|
||||
|
||||
let admin_config = create_need_p2p_admin_config("udp");
|
||||
admin_config.set_vpn_portal_config(VpnPortalConfig {
|
||||
wireguard_listen: "0.0.0.0:22121".parse().unwrap(),
|
||||
wireguard_private_key: Some(BASE64_STANDARD.encode([42u8; 32])),
|
||||
clients: vec![VpnPortalClientConfig {
|
||||
name: "portal-client".to_owned(),
|
||||
virtual_ip: "10.154.0.10/24".parse().unwrap(),
|
||||
groups: Vec::new(),
|
||||
}],
|
||||
});
|
||||
let mut admin_inst = Instance::new_with_process_runtime(admin_config, process_runtime.clone());
|
||||
admin_inst.run().await.unwrap();
|
||||
admin_inst.add_connector_url("udp://10.1.1.1:11010".parse().unwrap());
|
||||
wait_foreign_network_count(&public_server_inst, 1, Duration::from_secs(10)).await;
|
||||
|
||||
let (_credential_id, credential_secret) = generate_credential_with_options(
|
||||
&admin_inst,
|
||||
Vec::new(),
|
||||
false,
|
||||
Vec::new(),
|
||||
Duration::from_secs(3600),
|
||||
Some("portal-p2p-credential".to_owned()),
|
||||
false,
|
||||
)
|
||||
.await;
|
||||
admin_inst
|
||||
.get_global_ctx()
|
||||
.issue_event(GlobalCtxEvent::CredentialChanged);
|
||||
|
||||
let credential_config = create_public_server_credential_config(
|
||||
&credential_secret,
|
||||
"portal-credential-peer",
|
||||
"portal-credential-peer",
|
||||
"ns_c1",
|
||||
"10.154.0.1",
|
||||
"fd00::1/64",
|
||||
11030,
|
||||
11031,
|
||||
&[],
|
||||
);
|
||||
let mut credential_inst =
|
||||
Instance::new_with_process_runtime(credential_config, process_runtime);
|
||||
credential_inst.run().await.unwrap();
|
||||
credential_inst.add_connector_url("udp://10.1.1.1:11010".parse().unwrap());
|
||||
|
||||
let admin_peer_id = admin_inst.peer_id();
|
||||
let credential_peer_id = credential_inst.peer_id();
|
||||
wait_direct_peer(
|
||||
&credential_inst,
|
||||
admin_peer_id,
|
||||
Duration::from_secs(30),
|
||||
"credential -> admin before portal client",
|
||||
)
|
||||
.await;
|
||||
wait_direct_peer(
|
||||
&admin_inst,
|
||||
credential_peer_id,
|
||||
Duration::from_secs(10),
|
||||
"admin -> credential before portal client",
|
||||
)
|
||||
.await;
|
||||
wait_route_cost(
|
||||
&credential_inst,
|
||||
admin_peer_id,
|
||||
1,
|
||||
Duration::from_secs(10),
|
||||
"credential route to admin before portal client",
|
||||
)
|
||||
.await;
|
||||
|
||||
let portal_config = admin_inst
|
||||
.get_global_ctx()
|
||||
.config
|
||||
.get_vpn_portal_config()
|
||||
.unwrap();
|
||||
let (server_public, client_private) =
|
||||
test_wireguard_keys(&portal_config, "portal-client").unwrap();
|
||||
{
|
||||
let net_ns = crate::common::netns::NetNS::new(Some("ns_c4".to_owned()));
|
||||
let _guard = net_ns.guard();
|
||||
run_wireguard_client(
|
||||
&wireguard_ifname("wg0"),
|
||||
"10.1.1.4:22121".parse().unwrap(),
|
||||
Key::try_from(server_public.as_slice()).unwrap(),
|
||||
Key::try_from(client_private.as_slice()).unwrap(),
|
||||
vec!["10.154.0.0/24".to_owned()],
|
||||
"10.154.0.10".to_owned(),
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
let public_server_forwarded_before =
|
||||
forwarded_data_packets_for_network(&public_server_inst, NEED_P2P_ADMIN_NETWORK_NAME);
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
ping_test("ns_c4", "10.154.0.1", None).await;
|
||||
admin_inst
|
||||
.get_core_instance()
|
||||
.vpn_portal_info()
|
||||
.await
|
||||
.clients
|
||||
.iter()
|
||||
.any(|client| client.state == PortalClientState::Online)
|
||||
},
|
||||
Duration::from_secs(10),
|
||||
)
|
||||
.await;
|
||||
|
||||
let old_conn_ids = credential_inst
|
||||
.get_core_instance()
|
||||
.peer_snapshots()
|
||||
.await
|
||||
.into_iter()
|
||||
.find(|peer| peer.peer_id == admin_peer_id)
|
||||
.map(|peer| peer.directly_connected_conns)
|
||||
.unwrap_or_default();
|
||||
assert!(
|
||||
!old_conn_ids.is_empty(),
|
||||
"credential peer must have a direct admin connection to replace"
|
||||
);
|
||||
for conn_id in &old_conn_ids {
|
||||
credential_inst
|
||||
.get_core_instance()
|
||||
.close_peer_conn(admin_peer_id, conn_id)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
let has_new_connection = credential_inst
|
||||
.get_core_instance()
|
||||
.peer_snapshots()
|
||||
.await
|
||||
.into_iter()
|
||||
.find(|peer| peer.peer_id == admin_peer_id)
|
||||
.is_some_and(|peer| {
|
||||
peer.directly_connected_conns
|
||||
.iter()
|
||||
.any(|conn_id| !old_conn_ids.contains(conn_id))
|
||||
});
|
||||
let has_direct_route = credential_inst
|
||||
.get_core_instance()
|
||||
.route_snapshots()
|
||||
.await
|
||||
.iter()
|
||||
.any(|route| {
|
||||
route.peer_id == admin_peer_id
|
||||
&& route.next_hop_peer_id == admin_peer_id
|
||||
&& route.cost == 1
|
||||
});
|
||||
has_new_connection && has_direct_route
|
||||
},
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await;
|
||||
|
||||
wait_ping_reachability("ns_c1", "10.154.0.10", true, Duration::from_secs(10)).await;
|
||||
for _ in 0..3 {
|
||||
assert!(ping_test("ns_c1", "10.154.0.10", None).await);
|
||||
}
|
||||
assert_eq!(
|
||||
forwarded_data_packets_for_network(&public_server_inst, NEED_P2P_ADMIN_NETWORK_NAME),
|
||||
public_server_forwarded_before,
|
||||
"public server forwarded data despite disable_relay_data"
|
||||
);
|
||||
|
||||
drop_insts(vec![public_server_inst, admin_inst, credential_inst]).await;
|
||||
}
|
||||
|
||||
async fn create_generated_credential_config(
|
||||
admin_inst: &Instance,
|
||||
inst_name: &str,
|
||||
@@ -872,7 +1071,76 @@ async fn wait_stable_single_visible_peer_on_admins(
|
||||
|
||||
assert!(
|
||||
start.elapsed() < timeout,
|
||||
"timed out waiting for a stable single visible peer on both admins: a={:?} c={:?}",
|
||||
"timed out waiting for a stable single visible peer on both admins after {:?}: a_has_a={} a_has_b={} c_has_a={} c_has_b={} a={:?} c={:?}",
|
||||
start.elapsed(),
|
||||
admin_a_has_a,
|
||||
admin_a_has_b,
|
||||
admin_c_has_a,
|
||||
admin_c_has_b,
|
||||
admin_a_routes.iter().map(|r| r.peer_id).collect::<Vec<_>>(),
|
||||
admin_c_routes.iter().map(|r| r.peer_id).collect::<Vec<_>>()
|
||||
);
|
||||
|
||||
tokio::time::sleep(Duration::from_secs(1)).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn wait_stable_failover_visibility_on_admins(
|
||||
admin_a_inst: &Instance,
|
||||
admin_c_inst: &Instance,
|
||||
present_peer_id: u32,
|
||||
absent_peer_id: u32,
|
||||
timeout: Duration,
|
||||
) {
|
||||
let start = std::time::Instant::now();
|
||||
let mut stable_samples = 0;
|
||||
|
||||
loop {
|
||||
let admin_a_routes = admin_a_inst.get_core_instance().route_snapshots().await;
|
||||
let admin_c_routes = admin_c_inst.get_core_instance().route_snapshots().await;
|
||||
|
||||
let admin_a_has_present = admin_a_routes.iter().any(|r| r.peer_id == present_peer_id);
|
||||
let admin_c_has_present = admin_c_routes.iter().any(|r| r.peer_id == present_peer_id);
|
||||
let admin_a_has_absent = admin_a_routes.iter().any(|r| r.peer_id == absent_peer_id);
|
||||
let admin_c_has_absent = admin_c_routes.iter().any(|r| r.peer_id == absent_peer_id);
|
||||
|
||||
let failover_stable = admin_a_has_present
|
||||
&& admin_c_has_present
|
||||
&& !admin_a_has_absent
|
||||
&& !admin_c_has_absent;
|
||||
|
||||
println!(
|
||||
"failover visibility: present={} a_has_present={} c_has_present={} absent={} a_has_absent={} c_has_absent={} stable={} samples={}",
|
||||
present_peer_id,
|
||||
admin_a_has_present,
|
||||
admin_c_has_present,
|
||||
absent_peer_id,
|
||||
admin_a_has_absent,
|
||||
admin_c_has_absent,
|
||||
failover_stable,
|
||||
stable_samples
|
||||
);
|
||||
|
||||
if failover_stable {
|
||||
stable_samples += 1;
|
||||
} else {
|
||||
stable_samples = 0;
|
||||
}
|
||||
|
||||
if stable_samples >= 3 {
|
||||
return;
|
||||
}
|
||||
|
||||
assert!(
|
||||
start.elapsed() < timeout,
|
||||
"timed out waiting for stable failover visibility on both admins after {:?}: present={} a_has_present={} c_has_present={} absent={} a_has_absent={} c_has_absent={} a={:?} c={:?}",
|
||||
start.elapsed(),
|
||||
present_peer_id,
|
||||
admin_a_has_present,
|
||||
admin_c_has_present,
|
||||
absent_peer_id,
|
||||
admin_a_has_absent,
|
||||
admin_c_has_absent,
|
||||
admin_a_routes.iter().map(|r| r.peer_id).collect::<Vec<_>>(),
|
||||
admin_c_routes.iter().map(|r| r.peer_id).collect::<Vec<_>>()
|
||||
);
|
||||
@@ -2399,9 +2667,9 @@ async fn credential_non_reusable_across_two_admins_allows_only_one_peer() {
|
||||
let a_routes = admin_a_inst.get_core_instance().route_snapshots().await;
|
||||
let c_routes = admin_c_inst.get_core_instance().route_snapshots().await;
|
||||
a_routes.iter().any(|r| r.peer_id == admin_c_peer_id)
|
||||
|| c_routes.iter().any(|r| r.peer_id == admin_a_inst.peer_id())
|
||||
&& c_routes.iter().any(|r| r.peer_id == admin_a_inst.peer_id())
|
||||
},
|
||||
Duration::from_secs(10),
|
||||
Duration::from_secs(20),
|
||||
)
|
||||
.await;
|
||||
|
||||
@@ -2519,15 +2787,11 @@ async fn credential_non_reusable_across_two_admins_allows_only_one_peer() {
|
||||
drop_insts(vec![cred_right_inst.take().unwrap()]).await;
|
||||
}
|
||||
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
let admin_a_routes = admin_a_inst.get_core_instance().route_snapshots().await;
|
||||
let admin_c_routes = admin_c_inst.get_core_instance().route_snapshots().await;
|
||||
admin_a_routes.iter().any(|r| r.peer_id == loser_peer_id)
|
||||
&& admin_c_routes.iter().any(|r| r.peer_id == loser_peer_id)
|
||||
&& !admin_a_routes.iter().any(|r| r.peer_id == winner_peer_id)
|
||||
&& !admin_c_routes.iter().any(|r| r.peer_id == winner_peer_id)
|
||||
},
|
||||
wait_stable_failover_visibility_on_admins(
|
||||
&admin_a_inst,
|
||||
&admin_c_inst,
|
||||
loser_peer_id,
|
||||
winner_peer_id,
|
||||
Duration::from_secs(60),
|
||||
)
|
||||
.await;
|
||||
|
||||
@@ -1692,7 +1692,7 @@ use defguard_wireguard_rs::{
|
||||
InterfaceConfiguration, WGApi, WireguardInterfaceApi, host::Peer, key::Key, net::IpAddrMask,
|
||||
};
|
||||
|
||||
fn wireguard_ifname(base: &str) -> String {
|
||||
pub(super) fn wireguard_ifname(base: &str) -> String {
|
||||
if cfg!(target_os = "linux") || cfg!(target_os = "freebsd") {
|
||||
base.to_owned()
|
||||
} else {
|
||||
@@ -1701,7 +1701,7 @@ fn wireguard_ifname(base: &str) -> String {
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn run_wireguard_client(
|
||||
pub(super) fn run_wireguard_client(
|
||||
ifname: &str,
|
||||
endpoint: SocketAddr,
|
||||
peer_public_key: Key,
|
||||
|
||||
Reference in New Issue
Block a user