feat(delivery): add automatic rollback state machine for failed deployments
Introduce a compensating workflow that launches the dedicated AWX rollback template when a deployment cannot be started or fails midway. - add task states: rollback_pending, rolling_back, rolled_back, rollback_failed, rollback_acknowledged - add RollbackJob model to track the compensating AWX run separately from the deploy run, preserving both job IDs for audit - hold resource reservations in 'rollback' status until cleanup succeeds so a failed cleanup cannot be masked by a later delivery - poll rollback jobs with a 2-minute launch timeout; an unknown launch result surfaces as a recoverable failure instead of re-launching - protect finished/register_failed/rolled-back tasks from rollback; register_failed keeps the healthy instance and its resource usage - add DELIVERY_ROLLBACK_TEMPLATE_ID config; without it, failures are marked rollback_failed and require manual cleanup - use unique pending-<task_id> placeholder for executor job IDs
This commit is contained in:
@@ -63,6 +63,7 @@ type Config struct {
|
||||
AWXToken string
|
||||
AWXUsername string
|
||||
AWXPassword string
|
||||
RollbackTemplateID uint64
|
||||
DeliveryServiceToken string
|
||||
DeliverySchedulerEnabled bool
|
||||
DeliveryPollSeconds int
|
||||
@@ -133,6 +134,7 @@ func Load() Config {
|
||||
AWXToken: env("AWX_TOKEN", ""),
|
||||
AWXUsername: env("AWX_USERNAME", ""),
|
||||
AWXPassword: env("AWX_PASSWORD", ""),
|
||||
RollbackTemplateID: uint64(envInt("DELIVERY_ROLLBACK_TEMPLATE_ID", 0)),
|
||||
DeliveryServiceToken: env("DELIVERY_SERVICE_TOKEN", ""),
|
||||
DeliverySchedulerEnabled: envBool("DELIVERY_SCHEDULER_ENABLED", false),
|
||||
DeliveryPollSeconds: envInt("DELIVERY_POLL_SECONDS", 5),
|
||||
|
||||
Reference in New Issue
Block a user