Possible agenda BoF on gating migration from unstable to testing by
  autopkgtest results

* My ambition: use autopkgtest regression to prevent breakage in testing

* Current status and some technical background: what needs to be done:
  - autopkgtest
  - debci
  - britney2

* Choices
  - Influence aging and/or blocking?
  - BTS control?
  	-> don't file them immediately + up the age
  	- Don't automatically file bugs or be extermely careful about much of errors
  	- 

* Raised concerns
  - social impact; what about poor reverse dependency tests
    - maintainers would *have* to work together to figure out exactly is causing failures
  - tests with different severity
  - avoid adding load on the RT (for overriding corner cases)?
    - sometimes hints will be needed in order to override hard/complicated/problematic blockage of packages due to ci failures
  - Retry test a couple of times
  - handle a larger batches of tests (e.g. a perl transition) with lower priority to avoid starving out everyone else


- [xnox] AWS does not provide nested KVM, to check if nesting avaiable
- [xnox] Have an aws-ssh runner, to launch aws VM and run tests inside it over ssh
- [xnox] AWS use containers currently

- [xnox] maybe email "your package is stuck in unstable for X days now, britney excuses"

Links:
  https://wiki.debian.org/debci/britneyIntegration
  https://bugs.debian.org/cgi-bin/pkgreport.cgi?archive=both;tag=autopkgtest;users=ubuntu-devel@lists.ubuntu.com
  http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html

Miscelleneous ideas:
  - future use by security team to test all reverse dependencies before releasing

General remarks:

does someone have a pointer to the d-devel discussion?
"Auto reject if autopkgtest of reverse dependencies fail or cause FTBFS" ?
https://www.mail-archive.com/debian-devel@lists.debian.org/msg345232.html
thanks

Currently this is being used by Ubuntu and there are a lot of bugs that are 
caught by new things being broken.  There are some things that don't get 
detected and then show up in further tests, but that's the minority.

The release team has agreed to use the autopkgtests to modify the aging of 
packages (i.e. packages with tests that pass migrate faster, packages without 
tests migrate more slowly, packages with failing tests migrate even slower)

Once this is working for testing this could be used for stable / security as 
well.

Bugs filed when packages don't pass tests should inform both the maintainer
of the package that failed and the maintainer that wrote the test.

On Ubuntu they have "hints" for ignoring tests.  This allows for monitoring that
packages don't regress further.

We could also rely on the BTS, i.e. file the bugs and then block the packages. 
It would be an RC bug if there's a regression in the tests (tests passed before 
and are not passing now)

It would be important not to file bugs immediately, because there might be 
transient issues.  So, bugs should not be filed until the tests are proved not 
to be transient.

Currently Ubuntu only retries manually.