Validation methods

piuparts
ci.debian.net
Archive rebuilds
LAVA

Validation targets

Individual packages
Collections of related packages
  - could be done with a special source package in ci.debian.net
    (e.g. debian-functional-tests ?)
  - could this scale up to whole metapackages having tests that run whole
    desktop environments?
Installer & bootstraps
  - jenkins.d.n already runs graphical d-i test installs for Debian Edu
Kernel and bootloaders
Dirty systems
  How to anonymise systems?
    - autopkgtest supports trashing the testbed (for testbed backends that
      support reverthing that), so it is possible to script making the system
      dirty
    - IDEA: run tests on desktop systems (even if clean nevertheless)
      NEEDS: someone to do the work

Integration

Deterministic test frequency
  within N hours / days of upload
Ensuring developers are aware of breakages
Make notifications clear and descriptive
  direct to maintainers by email?
  use BTS?
    take care with frequency
    Maybe the unit test files can define a notification method/frequency
      ~~ but can't change it except by another upload :(
Avoid overload or flooding
Auto-filing bugs?
  Severity escalation?
Multiple architecture testing

IDEA: gerrit-like workflow
- patches submitted for package X
- packages is built and thrown at testing infrastructure
- if tests pass, patch gets a +1
- if a developer gives a +2, the patch could be "accepted" AKA merged
- QUESTION: how will such patch end up in the package VCS?
  Possible Answer: merge the change into the git repos via "Submit" after being accepted
  NOTE: there's a team of people investigating in a PoC for Gerrit/Code review usage inside Debian, feel free to ping mika, ch and/or jimmy

NEED: automatically test packages _before_ they enter the archive
- need to be able to test a source package from any arbitrary location
  - PPA's could be a solution
  - a special upload queue	
- uploading to experimental (and having experimental in ci.debian.net)
  could be a solution
- to run reverse deps' unit tests, against your proposed new package (maybe from experimental)
    this is meant for behavioural changes or unnoticed ABI changes?  (API changes require a rebuild)