During tests we discovered that python.apt throws exceptions when Currently - unittest and custom code ros is using py.test with conditional testing and host parametrisation * module testinfra - not packaged in Debian, but all rdeps are in Debian, we should just package it * ability to run unittests from py.test (? - not sure here, but would be good start) Generate new SSH key for test - not to use private (user) or security-conscious Security groups - currently by name, but need to use by-id to be able to run in non-standard VPC Advanced topic - test ENI, maximum bandwidth, pipeline support, DPDK-functionality MSI - separate, but related; MSI-X Hardware support for enhanced networking, virtualisation Related to Amazon Linux testing https://fedoramagazine.org/running-fedora-kernel-regression-tests/ Fedora uses AI; they learn what is normal For UI-related, screenshots, etc. - use OpenQA? https://debconf17.debconf.org/talks/69/ - talk from DebConf17, and video User-related tests Am I using all available memory? SMBIOS type - what I'm reading. sysctl parameters - are they all saved (even those temporarly changed), or only those changed explicitly GCE (by Max) Not much kernel, end-to-end boot, and user functionality Add-ons for users functions Login, agents Integration testing: * create instance * log-in * test functions Externally build image Make sure that SSH keys are not there Generalized tests vs. vendor-specific Specific to cloud User-story testing Login, ssh key management startup and startup scripts shutdown scripts: at GCE Boot sequence and shutdown sequence e.g. generated messages should appear - so networking cannot be down before they are generated Time limits for such tests? networking functionality: forwarding, load balancing, route setup Many NICs (verify their presence, that nothing is removing them, routing tables) Backward compatibility (? is it needed) Project vs. Instance config The most important: boot test: boot, reboot, run scp. Reboot from outside Security test; and check kernel settings Check machine sizes Memory - try to alloc or just check sys-values (from kernel) Number of cores? Cloud test vs instance tests - image uses services offered by the cloud, and expects for them to be working tests updates (apt upgrade) - mirrors used to have problems, especially old redirector Who is monitoring those tests? Who is AoD/carries pager, and can/will react when test fails Tests will have flakes Failure needs to be repeated failure Do we want to automatise? Or more correctly - how do we want to automatise it? Login as root - we get only message but we enable root login in sshd.conf and our test give wrong result During tests we discovered that python.apt throws exceptions when run inside of unittest URLs of mirrors - will probably depend on the cloud provider On AWS we have http://cdn-aws.deb.debian.org/ Provisional git repo: https://gitlab.com/rvandegrift/debian-cloud-tests List of open ports. How do we deal e.g. with image with PostgreSQL will have 5432 open. We might want to have some configuration file with list of allowed ports, to allow for variations Configuration file - like Python, so we have easier way to import (and have variables) Or YAML? But then need to parse it Day 2 ===== Starting instance and SSH into it Helen is writing code that will run gce-cli tools to start image and get SSH connection string so we can use it in Paramiko There might be easier way, but for now it should be enough For AWS we get AMI id for running instance with this image For GCE - we might run from registered image, or from disk image (not sure about that) At which step of this process should we run tests? Item 5) SSH root: check the message it output, or remove root access in the cloud iamge We are not produce images that are specialize (e.g. with postgre) Separate tests by file (by class name), to work as a plugin, so we can have a set of tests per image Item 12) we could check the ssd config file or try to login as root. There are Debian predefined users, maybe check that there is no adicional user Item 18) efi partition needs to be tested Build logs needs to be available but not necessary in the image, and probably deal with reproducibility Also, the logs in the image can confuse people that build images from other images and it won't match what the person have We can just point where the logs are Cover tests with different types of virtualized hardware, solve this with some kind of plugin system so we can add different tests per cloud provider Also launch instances with different options Day 3 ===== Additional users on GCE depends on the account metadata configuration (ssh keys are also affected) URLs for sources.list depends on the platform but they should be unified TODO: check why tests are not regocnizing stretch on gce fai image (use assert to print what is the version it recognizes) NOTE: this is related to URLs and aditional google repositories that is configured in GCE TODO: use assert to print which other ports are open TODO: check why it doesn't find git package TODO: check that all installed packages are from the repository TEST-SUGGESTION: maybe check if users have uid and gid set to zero (as it has root priviledges) TEST-SUGGESTION: check specific key ids listed in apt-key TEST-SUGGESTION: check if there is no duplicated users/uid (mainly uid zero) TODO: write a (-policy-) requirements about the image (e.g. which ports are allowed, which users are allowed, which tests we should make) based on the current test suites