* Who is here? pabs lucas buxy skainz Rhonda tobi * Status of our major testing tools & infrastructure: ** maintained? (on d.o?) ** reliable (false positives)? easy to run? ** who is filing bugs? # run: gobby infinote://gobby.debian.org/debconf16/bof/qa https://wiki.debian.org/qa.debian.org manual package review (mentors, ftp-master, secteam) static analysis (lintian, duck, check-all-the-things (and the many tools run by it)) build reproducibility (tests.reproducible-builds.org/debian) build log checks (qa.d.o/bls) packaging testing (piuparts, adequate) * maint: holger, andreas beckman * also testing upgrades of packages * filing bugs as-installed testing (autopkgtest, ci.debian.net) * only amd64/arm64 * would be useful to test on an infrastructure where ocaml doesn't have a native code compiler (mips, mipsel, ...) * similarly, with big-endian infrastructure install testing (jenkins.debian.net) user-testing of installs (manual by cd team) whole-system automatic testing (lava.d.n, jenkins.d.n) qa.debian.org (various) archive rebuilds for FTBFS testing * reproducible builds people are filing some bugs (not sure about coverage) * lucas set up the AWS rebuild infrastructure again, and plans to do rebuild (including custom) and file bugs * What should we test for stretch? ** That we already tested in the past ** That we haven't tested yet +testing packages *before* they enter the archive? (NEW queue, mentors) - testing of maintainer scripts + testing by piuparts + static analysis by lintian + talk by ralf tomorrow morning https://debconf16.debconf.org/talks/63/ - builds that access the internet - maintainer scripts, autopkgtest that access the internet (maybe piuparts does it for maint scripts) * Process question: should mass bug filings remain our main way of reporting problems, or should we plug tools into britney? devscripts: /usr/bin/mass-bug (can't attach files: #801926) + bug filing: seems efficient, allows to keep track of progress * Infrastructure is a mess combination of git & svn split qa.debian.org into multiple machines/domains/services? consolidate multiple services onto the same machine? - data transfer issues * Common scheduler for all QA services? - ci.d.n uses rabbitmq?? - duck.d.n uses daily cron - reproducible builds uses jenkins - fedmsg - died a bit * Add more things to lintian auto-rejects? - File RC bugs for remaining auto-reject tags - Non-automated stuff: * Accurately gathering user issue reports * Doing user testing based on particular tasks * Usability testing of individual packages * Reportbug not particularly usable * email maintainers with stuff about their packages after new uploads * email on changes to their packages * opt-in site to reject on any lintian/piuparts/etc warnings * speed packages pass through the system * Other topics (feel free to add)