Running autopkgtest for your package Questions: Where can I start? Q: if you're just testing --help ... will that affect testing migration? seems a low bar A: In the test/control file, you can set "Restrictions: superficial" to express that the test is just a smoke-test, so it can prevent migration, but not make it quicker Q: Where can I find a howto for "Dummies"? Can you show a simple example how to add autotest to a package? Q & DISCUSSION BETWEEN A FEW: It can be tricky to compile the unit tests against the installed package (testing the pkg-config integration as well). A: See for instance liblouisutdml, I apply a patch to make the package link against the installed library rather than the just-built library. Basically s_../lib/libfoo.la_-lfoo_ and possibly try to avoid even building inside lib/ (both to save time, and to make sure one doesn't use it by accident) Running it at build-time might otherwise be easier, unless it's a long test. Sure, but sometimes the tests need internet. Also, tests sometimes break because of another package, so we want to run them against newer deps. I managed to coerce Automake into building tests against the installed library a couple of times. It would be usefult to collect such tricks at some central place. (autopkgtest wiki), I guess, these aren't exactly trivial. Note: take care of rpath :) Rpath is discouraged. :) Yes but libtool does it, I rather not patch libtool :) Debhelper patches it for you. Actually Debian libtool is patched AFAIK. Yes, but debhelper runs after the build+test That's why you don't rebuild the source before the autopkgtest but rather use the installed library. the library, yes, but the tests are not in .debs, so you have to build them. And *they* might get an rpath due to libtool, or at least be actually a script that looks for the lib in the build tree Yes, this is tricky stuff, that's why I recommended collcting these at some central location. From irc: [11:13:55] there are some cmake test suites that embed the build path of the tests ... which obviously won't work. Q: Is there a location in policy for test-only binaries? A: If these test binaries aren't generally useful, it's probably better to not package them but build as part of the autopkgtest. The source tree is available so you can have the code in the so you can run/build the test from there. We have had this thought also in Lintian. For us it would be a speed issue. We have 1,250 test packages. Also see https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests - /usr/libexec/installed-tests is used there, e.g. see pango1.0-tests, and /usr/share/installed-tests for data Q: For which package is it practical? Q: Is there an easy way to run tests in CMake packages? It depends: usually the build system will try to use the build binaries/libraries... https://salsa.debian.org/mbanck/dd-autopkgtest - script to run autopkgtest on porter boxes or in local schroots Q: Is there something similar for normal packages to start? (similar as for porter boxes also for normal arch? please elaborate (e.g. define normal packages :)) (barriere.debian.org is a i386/amd64 porterbox, FYI) Q: When will ci.debian.net start supporting isolation-machine? How can one help with providing isolation-machine support? A: come by IRC and "we can investigate that together" Q: What is considered superficial? Some concrete examples would help. Running the program with --help, certainly. But what about building an application using an installed library for example? Should I declare that superficial? A: it looks superficial to me indeed. A test should at least run the library :) A: the rule-of-thumb I use is that if autopkgtest is not doing enough stuff for me to be confident in uploading the package, I make it superficial Q: How would you test GUI packages? A: I run them throught xvfb-run, e.g. in at-spi2-atk -> how would you "click" somewhere and check behavior? A: bambam is a nice example, it calls xdotool, and verify doing screenshots, though that second part is only checking for colors. A: dogtail can also be used for testing automation (going through the accessibility stack) Suse (and Fedora) have OpenQA for testing GUIs. Not sure if that can be used in this context. Qt packages have integrated support for GUI testing that can be run in xvfb-run (e.g. in keepassxc) - QTest::mouseClick() and QTest::keyClicks() - you write the tests in C++ using standard QTest helpers, you can then even access the GUI internal implementation Q: what about "standard" Testuite beside autopkgtest? For example: - debian/tests/cmake/foo -> it would try to create (and run) a simple CMakeLists.txt to check for the packages/versions listed in foo - debian/tests/pkg-config/foo -> it would try that `pkg-config --libs/--cflags/etc foo` work I see many packages started to add this as basic autopkgtest, IMHO it would be nice to have them standardized, and have Testsuite properly filled at source build. Seems like this should land in autodep8 Q: How can we help? (in development or otherwise!) There is a request for help thread, autopkgtest is in Python and debianci in Ruby, Swing by the IRC or the ML Q: I have seen cross-arch-autopkgtest patches fly around :) (e.g. on the at-spi2-atk package) Possibly useful to document (and factorize?) What's your plan with https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/69 (Cross arch testing, which is of course a misnomer.) I think cross-testing is currently an Ubuntu-specific patch set? Looks so, #946292. Yes, and they are pressing for changes supporting that to reduce their delta with Debian. Yes we do :) smcv seems to have some unstated objections. --> bof discussed up until this point in video <-- Q: Hi, why does autopkgtest complete fine for Lintian as part of Salsa CI, while it fails on the regular CI framework, please? Can I make changes to d/tests/control that will restore Deb CI, or do I have to revert the access to the process table? Q: Could you please export to the tests whether internet is available for them? (Instead of having the needs-internet restriction. Sometimes upstream test frameworks produce a single binary running lots of tests, some of which might require internet but can be skipped by setting an environmental variable for example.) The predictability requirement mentioned just now may figure in here. Q: see IRC #dc20-talks regarding continuous integration autopkg test., (resolved with qemu install?) plus new questions in IRC.