feat(delivery): add automatic rollback state machine for failed deployments

Introduce a compensating workflow that launches the dedicated AWX
rollback template when a deployment cannot be started or fails midway.

- add task states: rollback_pending, rolling_back, rolled_back,
  rollback_failed, rollback_acknowledged
- add RollbackJob model to track the compensating AWX run separately
  from the deploy run, preserving both job IDs for audit
- hold resource reservations in 'rollback' status until cleanup
  succeeds so a failed cleanup cannot be masked by a later delivery
- poll rollback jobs with a 2-minute launch timeout; an unknown launch
  result surfaces as a recoverable failure instead of re-launching
- protect finished/register_failed/rolled-back tasks from rollback;
  register_failed keeps the healthy instance and its resource usage
- add DELIVERY_ROLLBACK_TEMPLATE_ID config; without it, failures are
  marked rollback_failed and require manual cleanup
- use unique pending-<task_id> placeholder for executor job IDs

This commit is contained in:

Hungerdream

2026-07-28 14:29:09 +08:00

parent 7a800b0307

commit f0c1b38c5b

6 changed files with 366 additions and 11 deletions

									
										server/.env.example
									
		+2
		
												View File
												
				@@ -30,6 +30,8 @@ AWX_BASE_URL=

				AWX_TOKEN=

				AWX_USERNAME=

				AWX_PASSWORD=

				# AWX Job Template ID for ansible/mysql-rollback.yml; required for automatic cleanup

				DELIVERY_ROLLBACK_TEMPLATE_ID=0

				DELIVERY_SERVICE_TOKEN=

				CLOUDDM_REGISTER_URL=

				CLOUDDM_API_TOKEN=