feat(delivery): add automatic rollback state machine for failed deployments

Introduce a compensating workflow that launches the dedicated AWX
rollback template when a deployment cannot be started or fails midway.

- add task states: rollback_pending, rolling_back, rolled_back,
  rollback_failed, rollback_acknowledged
- add RollbackJob model to track the compensating AWX run separately
  from the deploy run, preserving both job IDs for audit
- hold resource reservations in 'rollback' status until cleanup
  succeeds so a failed cleanup cannot be masked by a later delivery
- poll rollback jobs with a 2-minute launch timeout; an unknown launch
  result surfaces as a recoverable failure instead of re-launching
- protect finished/register_failed/rolled-back tasks from rollback;
  register_failed keeps the healthy instance and its resource usage
- add DELIVERY_ROLLBACK_TEMPLATE_ID config; without it, failures are
  marked rollback_failed and require manual cleanup
- use unique pending-<task_id> placeholder for executor job IDs

This commit is contained in:

Hungerdream

2026-07-28 14:29:09 +08:00

parent 7a800b0307

commit f0c1b38c5b

6 changed files with 366 additions and 11 deletions

									
										server/internal/database/database.go
									
		+1
		
												View File
												
				@@ -27,6 +27,7 @@ func AutoMigrate(db *gorm.DB) error {

						&model.MySQLInstance{},

						&model.ResourceUsage{},

						&model.ExecutionJob{},

						&model.RollbackJob{},

						&model.TaskEvent{},

					)

				}