09:41:42 <decauseDevConf> #startmeeting DevConf-D105-DanWalsh
09:41:43 <zodbot> Meeting started Fri Feb  5 09:41:42 2016 UTC.  The chair is decauseDevConf. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:41:43 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
09:41:43 <zodbot> The meeting name has been set to 'devconf-d105-danwalsh'
09:41:57 <decauseDevConf> I started working 2 years ago to get systemd to work well with containers
09:42:05 <decauseDevConf> it has been an interesting task, to say the least
09:42:15 <decauseDevConf> we have solman, and lennart
09:42:24 <decauseDevConf> neither one is willing ot compromise much
09:42:27 <decauseDevConf> *laughs*
09:42:31 <decauseDevConf> so I get stuck int he middle
09:42:48 <decauseDevConf> #topic DockervsSystemd
09:42:57 <decauseDevConf> "can't we all just get along"
09:43:03 <decauseDevConf> we have to look at diff components
09:43:17 <decauseDevConf> dockerd runs, eventually we get systemd running in there
09:43:23 <decauseDevConf> "who own's restart?"
09:43:30 <decauseDevConf> systemd wants to own it, docker wants to own it
09:43:36 <decauseDevConf> in many ways, docker dreams of being systemd
09:43:48 <decauseDevConf> it wants to build in much in
09:44:01 <decauseDevConf> I want to run apache in a unifile, who owns restart?
09:44:18 <decauseDevConf> the probelm is, restarting dockerd, takes down all of your containers
09:44:41 <decauseDevConf> when docker goes down, systemd doesn't know either, cause it only sees the client
09:44:51 <decauseDevConf> #topic sd_notify
09:45:06 <decauseDevConf> when you build a service that other services depend on, sahing "service started" is not enough
09:45:12 <decauseDevConf> mariadb, with a web front end
09:45:22 <decauseDevConf> mariadb has to be fully up before front-end
09:45:27 <decauseDevConf> so, sd_notify is there
09:45:34 <decauseDevConf> you can have it in your mariadb code
09:45:38 <decauseDevConf> what happens when you run that in a conatiner?
09:45:51 <decauseDevConf> dockerd does sd_notify now, but i'm jumping ahead
09:46:02 <decauseDevConf> #topic Socket Activation
09:46:14 <decauseDevConf> you can tell systemd to listen on ports to start services
09:46:23 <decauseDevConf> you dn't have to have services running at boot time, all of htem
09:46:31 <decauseDevConf> dockerd can do socket activation
09:46:37 <decauseDevConf> but there is a problem
09:46:44 <decauseDevConf> but we disable it becuase it breaks auto-restart
09:47:02 <decauseDevConf> that means docker will not start unless someone connects, but htat will only be the admin so we disabled it
09:47:25 <decauseDevConf> sd_notify works inside docker, but I'd like it to work inside my application.
09:47:35 <decauseDevConf> the problem is, docker is a client/server operation
09:47:49 <decauseDevConf> systemd requests from docker client, docker client, talks to dockerd.
09:47:55 <decauseDevConf> container would then mimic systemd
09:48:06 <decauseDevConf> then then server gets notice form container
09:48:12 <decauseDevConf> then the server sends sdnotify backt o tclient
09:48:15 <decauseDevConf> then back to system d
09:48:25 <decauseDevConf> hows that for a complex way of doing thigns?
09:48:36 <decauseDevConf> *patches were not accepted upstream"
09:49:00 <decauseDevConf> #topic Socket Activation docker containers
09:49:06 <decauseDevConf> systemd will accept the call
09:49:18 <decauseDevConf> then, on normal services, it would like athe file descriptor to the application
09:49:25 <decauseDevConf> apache could be socket activated,
09:49:40 <decauseDevConf> systemd would listen on port 80, then leak the
09:50:01 <decauseDevConf> passing the open socket someone to the daemon, and then leak it? we didn't even try it.
09:50:21 <decauseDevConf> #topic cgroup config of docker conatiners
09:50:28 <decauseDevConf> docker wants to run cgroups
09:50:34 <decauseDevConf> it wants to be systemd
09:50:42 <decauseDevConf> docker run in unit file
09:50:51 <decauseDevConf> the container will get the dockerd cgropu constaints
09:51:07 <decauseDevConf> it affects the client, not the process inside the container
09:51:13 <decauseDevConf> #topic runc
09:51:21 <decauseDevConf> this is the implementation for the spec
09:51:28 <decauseDevConf> runc also uses libcontainer
09:51:31 <decauseDevConf> so does docker
09:51:39 <decauseDevConf> in theory, docker will exec runc command
09:51:43 <decauseDevConf> we're packaging it in Fedora
09:51:49 <decauseDevConf> and evnetually it will show up in RHEL
09:51:59 <decauseDevConf> runc does not work in client/server model
09:52:14 <decauseDevConf> we got patches merged to get sd_notify working in runc
09:52:22 <decauseDevConf> sock activation merged too
09:52:32 <decauseDevConf> modifyin cgroup will affec the container
09:52:38 <decauseDevConf> and you can use systemctl to modify it
09:52:46 <decauseDevConf> you can use runc as a potential alternative
09:52:55 <decauseDevConf> #topic docker logs journald support
09:53:08 <decauseDevConf> anyhing in a container logging to standard out
09:53:16 <decauseDevConf> but if you lose a container, the logs get wiped out
09:53:25 <decauseDevConf> I'd like to know what the cotainer did on my system
09:53:40 <decauseDevConf> as of docker 1.9, you can specify a log driver, so that they stay permanently
09:53:51 <decauseDevConf> tools that offload logs will be
09:53:59 <decauseDevConf> journald willb e default log-driver in F24
09:54:17 <decauseDevConf> #topic systemd in the base image (Fedora/RHEL/CentOS)
09:54:21 <decauseDevConf> we want minimal size
09:54:33 <decauseDevConf> the problem with base images, that apps always bring in systemd
09:54:43 <decauseDevConf> they require systemd control
09:55:12 <decauseDevConf> it is a packaging issue that we always need systemd, and it brings a lot of stuff with it, like udev, and other things required for the kernel, and not ystemd
09:55:18 <decauseDevConf> we wanted a version that didn't do that
09:55:29 <decauseDevConf> the first version, fakesystemd
09:55:51 <decauseDevConf> that satisfied us, but lennart and friends hated it
09:56:06 <decauseDevConf> if you wanted systemd in a container, you had to run a weird yum command to remove the fake for th real
09:56:13 <decauseDevConf> we didn't ship this in Fedora
09:56:21 <decauseDevConf> but we did in 7.0
09:56:27 <decauseDevConf> then changed our minds in 7.1
09:56:36 <decauseDevConf> v2 systemd-container
09:56:41 <decauseDevConf> problem was, lennart hated it
09:56:47 <decauseDevConf> it was real systemd, but remixed
09:56:59 <decauseDevConf> so RHEL7.2 came out
09:57:04 <decauseDevConf> and v3 is systemd
09:57:16 <decauseDevConf> in Fedora/CentOS, we're shipping real ysstemd
09:57:26 <decauseDevConf> it's in the base container, an deveyone is happy, right?...
09:57:34 <decauseDevConf> may5, 2014
09:57:45 <decauseDevConf> article explained how to run systemd in a container
09:57:51 <decauseDevConf> much of the stuff in there is no longer true
09:58:01 <decauseDevConf> but the key thing is, it has to be runnign privileged
09:58:14 <decauseDevConf> #topic What is the benefit of running systemd in a conatiner?
09:58:27 <decauseDevConf> docker requires pid1
09:58:35 <decauseDevConf> lennart thinks this is crazy
09:58:44 <decauseDevConf> pid1 means certain things
09:58:52 <decauseDevConf> you need an api
09:59:00 <decauseDevConf> solman thinks this is crazy
09:59:28 <decauseDevConf> systemd, proper pid1, must cleanup zombies
09:59:48 <decauseDevConf> pid1 is supposed to reap children
10:00:03 <decauseDevConf> default docker containers write to syslog == /dev/null
10:00:17 <decauseDevConf> anytime an app runs in a docker container, messages go to /dev/null
10:00:35 <decauseDevConf> journald support, get syslog messages on the host with journalctl
10:00:42 <decauseDevConf> running apps, as designed by the packager
10:00:53 <decauseDevConf> you'll see lots of weird wacky startupscripts
10:01:47 <decauseDevConf> the big downfall is that you have more things running (systemd, journald)
10:01:52 <decauseDevConf> #topic Have I Failed?
10:02:05 <decauseDevConf> when systemd comes up, it expects teh world to be it's way
10:02:16 <decauseDevConf> /etc/machineid created
10:03:23 <decauseDevConf> #topic Jessie Frazelle - "I say not to systemd specific pull requests"
10:03:32 <decauseDevConf> #topic latest attempt
10:03:42 <decauseDevConf> runc has a feature called dockerhooks
10:03:48 <decauseDevConf> pre/post-start hook
10:04:29 <decauseDevConf> we have a patch, it got closed, but we're shipping in 1.10
10:04:40 <decauseDevConf> carrying a 5 line patch will not be painful
10:04:50 <decauseDevConf> systemd has machinectl
10:05:00 <decauseDevConf> if you run an endspawn, it will tell names
10:05:11 <decauseDevConf> if you run VM's, libvrit will tell it
10:05:45 <decauseDevConf> you can use machinectl to tell your system about all your machines, accept docker, because docker refuses to play with systemd
10:05:53 <decauseDevConf> oci-register-machine hook
10:06:28 <decauseDevConf> machinectl can start and stop containers, and with our patch, docker included
10:06:35 <decauseDevConf> docker 1.10 gets shipped today
10:06:48 <decauseDevConf> we're going to get these pathces into fedor asoon
10:06:58 <decauseDevConf> #topic running systemd without --privilege
10:07:20 <decauseDevConf> if you have /run/httpd inside a container image, ou put a tmpfs on top, it will untar
10:07:34 <decauseDevConf> i got a patch accepted for --tmpfs accepted
10:07:52 <decauseDevConf> creats a /var/log/journald/UUID
10:07:59 <decauseDevConf> mounts it in container
10:08:12 <decauseDevConf> creates /etc/machine-id in container with containers UUID
10:08:23 <decauseDevConf> #topic Making Docker Stop Work
10:08:32 <decauseDevConf> we can't get systemd to end properly in a container
10:08:43 <decauseDevConf> it sends sigterm to pid for systemd
10:09:02 <decauseDevConf> which to systemd, means 'restart' if you are pid1, according to lennart
10:09:25 <decauseDevConf> docker now supports custom stop signals
10:09:36 <decauseDevConf> #topic demo
10:13:17 <decauseDevConf> #topic questions
10:13:28 <decauseDevConf> Q: what user is systemd running on?
10:13:34 <decauseDevConf> A: Root, but it could be not root
10:19:32 <decauseDevConf> #endmeeting