Continuous monitoring Azure DevOps Release

Question

I need to create a setup where I deploy a "Webapp for containers" but I want to build in some checks via Azure Monitor. My idea is to deploy the web app and then have a gate that checks an azure monitor alert set with availability. When the availability check fails then it should rollback.

The documentation states "When the release pipeline detects an Application Insights alert, the pipeline can gate or roll back the deployment until the alert is resolved", but I don't know how to configure this in azure DevOps.

I have an AppService Plan and a web app running. I also created an Application Insights instance and I enabled continuous monitoring through the "Azure App Service manage" task. The alert I created is:

az monitor metrics alert create -n 'Availability' -g ${RG_NAME} --scopes "${APP_INSIGHTS_PROD}" \
--condition 'avg availabilityResults/availabilityPercentage < 90' \
--description "created from Azure DevOps"

As Post Deployment-Condition I enabled the Gates and I configured it to check for the Availability alert, which works. When I adjust something to make the app fail on purpose the Gate works and fails the Stage eventually. I also enabled the auto-redeploy to deploy the last successful deployment but that does not do anything because the actual deployment task was successfully finished.. just the gate failing and failing the stage.