End To End Validation

[gatneil]

It would be awesome if the bot could offer some way to check app-level end-to-end success of the deployment, not just azure-level deployment success. For instance, if I set up a web server in my deployment, I might want to check that everything worked correctly by hitting a certain web endpoint and seeing what comes back.

[gatneil]

As far as implementation is concerned, would it make sense to just keep one linux and one windows VM around to run the user-supplied scripts? This should help with security concerns because the VMs won't have any azure subscription info or credentials.

[sedouard]

@gatneil is there a way we can accomplish this by deploying web test resources with the template itself?

[singhkays]

Adding @ahmetalpbalkan as well

@sedouard What are web test resources?

[sedouard]

https://github.com/Azure/azure-quickstart-templates/blob/master/201-dynamic-web-tests/azuredeploy.json#L151

[singhkays]

@sedouard this seems pretty promising. So do we just pipe a DNS or PublicIP from the outputs sections to webtest template that pings the address to verify the application is responding?

[sedouard]

I really have no idea actually :-). I was just bouncing some ideas around to avoid the idea of having to containerize and run arbitrary code from randoms on the internet :-). I suppose you can generate web tests, and ping their status through some ARM api?

Honestly though I did some thinking yesterday and I think I could have done better with the architecture here.

Consider this idea, rather than this server deploying templates itself, just create ephemeral role based access control credentials that have a time-to-live of maybe 90 minutes or so. The credentials will only have a permissions scope of the resource group provided.

Then provide the keys to the client, (ideally, the test framework). Then the client can control the validation using regular CLI commands and decide when to validate that its deployment is 'running' and provide a pass/fail.

Now if we did things like this you can run any validation script client-side and we wouldn't have to worry about the complexities of running arbitrary untrusted code.

Unfortunately the server isn't built this way today, but fortunately there isn't anything stopping having this implementation along side of what's here today. However, this is a sizable task at hand.

[sedouard]

A less expensive approach could be to allow the client to specify a (small) TTL that the deployment can live for after a successful deployment. Ideally since these aren't really trusted deployments it wouldn't be too long before we start deleting the resource group.