live testing plans
==================

During daily and release builds: launch image, do simple tests even before any
image copies.

During release builds: build towards complete test suite: every type, every
region, different block devices, network interface attach/detach, etc

Should be implemented in the debian-cloud-images tool
- way to trigger full matrix of tests
- way to trigger a limited set of cases
- way for users / providers to download & run against a specific target
  - provides a smoke test for downstreams
  - provides pre-release qualification for providers

Possible to use pytest to generate the full set of test cases?

Admin group
Azure - upload, wait for confirmation, then "Go Live"
We can move other clouds to similar workflow
In other words - publish/register privately, test it (at least a bit)
and only if green, publish to public

AWS has this - we can publish as private image
Noah created PR: might require work but good first step

Initial idea (staged testing, per region, etc.) -> too complex,
only makes sense if we expect it to fail

For daily - run simple tests on them
To validate both our build process, and resulting images

Simple (smoke) tests, and then run full matrix of tests

Don't overwhelm gitlab CI; if it gets too complex, we'll use
external tool for running and managing tests

Some external test tools use Jenkins; might be too complicated
for our needs (all Jenkins management)

use packer?

test-infra, build on top of pytest

We already have users/roles, with quite limited capabilities (for publishing images)
We'll need to extend them to be able to run instance, attach some networking devices, and so on

How do we test on OpenStack?
zigo will provide account so we can register (upload) images there
and run tests (similar to other providers)
waldi will manage keys for that