* Debian in production Group Thera^W^WBoF

gobby infinote://gobby.debian.org/minidebconf-vaumarcus/debian_in_prod.org

"we" means a group of people keeping systems running for an employer

** Tension between getting things done and doing things right

 - Debian Developers
 - Should package stuff for Debian...
 - ... but policy adherence and going through NEW can take a while
 - Compromise: upload existing package updates to Debian, do "quick and dirty"
   backports for the rest

** Debian as a grey box
 - Debian is a strong, stable base
 - I trust point releases and security updates (mostly) blindly
 - unattended-upgrades everywhere
   - also from proposed-updates on the staging infra (still TODO)
 - monitoring of all services
 - systemd automatic restarts for critical services
 - manual updates for our own software
 - manual service restarts / reboots
 - use policy-rc.d to override dpkg restarting services on upgrade

** Deploying stuff "the Debian way"

*** Upstream packaging
 - all the stuff we write is Python
 - automatic PyPI packaging from tags (using jenkins)

*** Debian packaging
 - automatic Debian (unstable) packaging from PyPI releases (also in jenkins)
   (not always working, because Python3.8 is arriving and this is messing)
   (but still this is some fun coming in unstable \o/)
 - automatic backports (stable and oldstable) from the unstable packages
 - all of this to a private repository

*** Deployment
 - all done through puppet manifests
 - puppet doesn't do package updates
 - version pinning for clustered systems done via apt preferences, updated via
   puppet

** Some annoying defaults

 - Services start on install, restart on upgrade
 - `debconf` messing with config files
   - race conditions between conf management and service installation
     (it _should_ work as specified in debian policy, we have some unreported rc bugs...)
 - policy-rc.d to deny dpkg to run services everywhere? this is used as a workaround.

** Installing a new server
 - base install:
   - debian installer (manual) for bare metal
   - debian installer (preseeded) for old virtual machines
   - for more recent stuff, via terraform
     - template + cloud-init on proxmox
     - base image (by Debian) on azure
 - then a puppet run

** multiple systems management
 - we use clustershell (clush)
 - most used command:

     clush -w <hostnames> \
       env DEBIAN_FRONTEND=noninteractive \
         apt-get \
           -o Dpkg::Options::="--force-confdef" \
           -o Dpkg::Options::="--force-confold" \
           -y dist-upgrade


** Cluster support in Debian

 - No provisions for clustered systems, e.g. rabbitmq
 - restarting a single node is not enough, and can break your cluster horribly
 - kafka needs to be upgraded one node at a time
 - hard to make a policy about clusters: everything has a different way of working as a cluster

** #904558 (tech-ctte): what should happen when a maintscript fails to restart a service
 - agreement that the status quo is not optimal
 - disagreement whether the packaging system should touch the service state at all, or whether it should make sure the state is the same, or restart unconditionnally...
 - in some cases it's often not preferred to auto-start a service by default, like game servers on desktop
 - whether you want to restart automatically can also depend on the type of fix that you've just applied (security?)


https://tracker.debian.org/pkg/policy-rcd-declarative
#911290 policy-rc.d is a bad interface, please replace it


Issues with the way configuration ais managed: no consistency in between the way packages are configured, which makes automation hard.


Kubernetes? has a unified configuration layer, but is now a behemoth

systemd + fleetd? fleetd is dead upstream now


How to upgrade openstack?
 - stop some core services on all nodes
 - upgrade the code on all nodes
 - upgrade the database
 - restart the core services on all nodes

some people do it with ansible (there's good examples doing it for Ceph clusters); openstack is harder; rabbitmq + corosync + pacemaker + galera is done by hand :D


are there alternatives to kubernetes? (no good answer)

Debian packaging for deployment? Putting a debian package on a repository is lots of steps to just push code.

Maybe some kind of fasttrack repository could make the process smoother. Needs solid QA in-house so your own stuff doesn't break.

Doing stable updates to help doing upgrades is not always easy. Auditing upstream diffs is sometimes a nightmare.

(olasd heard interesting things :o 
(CAN WE STOP RECORDING????)
(Didier WILL DEFINITELY ORGANIZE ANOTHER MINIDEBCONF NEXT YEAR (he doesn't know it yet but he will (really, one should tell him)))