16:30:01 <dustymabe> #startmeeting fedora_coreos_meeting
16:30:01 <zodbot> Meeting started Wed Sep 25 16:30:01 2019 UTC.
16:30:01 <zodbot> This meeting is logged and archived in a public location.
16:30:01 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:01 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:06 <dustymabe> #topic roll call
16:30:10 <dustymabe> .hello2
16:30:10 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:11 <slowrie> .hello2
16:30:13 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:15 <bgilbert> .hello2
16:30:17 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:18 <ajeddeloh> .hello2
16:30:20 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:30:32 <strigazi> .hello2
16:30:33 <zodbot> strigazi: strigazi 'Spyros Trigazis' <strigazi@gmail.com>
16:30:48 <jlebon> .hello2
16:30:49 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:09 <red_beard> .hello redbeard
16:31:12 <zodbot> red_beard: redbeard 'Brian 'redbeard' Harrington' <bharring@redhat.com>
16:31:24 <geoff-> geoff-: 'Geoff Levand' <geoff@infradead.org>
16:32:04 <lorbus> .hello2
16:32:05 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:32:23 <walters> .hello2
16:32:24 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:32:47 <dustymabe> #chair slowrie bgilbert ajeddeloh strigazi jlebon red_beard geoff- lorbus walters
16:32:47 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jlebon lorbus red_beard slowrie strigazi walters
16:33:19 <dustymabe> wow - what a beautiful group of FCOS enthusiasts today!
16:33:26 <kaeso[m]> .hello lucab
16:33:27 <zodbot> kaeso[m]: lucab 'Luca Bruno' <lucab@redhat.com>
16:33:48 <dustymabe> #chair kaeso[m]
16:33:48 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jlebon kaeso[m] lorbus red_beard slowrie strigazi walters
16:34:21 <red_beard> streams for everyone
16:34:21 <lorbus> dustymabe: true dat! :)
16:34:29 <dustymabe> #topic Action items from last meeting
16:34:38 <dustymabe> looks like last meeting we didn't have any action items!!
16:34:56 <jdoss> .hello2
16:34:56 <red_beard> 100% completion
16:34:56 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:35:02 <dustymabe> :)
16:35:05 <dustymabe> #chair jdoss
16:35:05 <zodbot> Current chairs: ajeddeloh bgilbert dustymabe geoff- jdoss jlebon kaeso[m] lorbus red_beard slowrie strigazi walters
16:35:35 <dustymabe> #topic FCOS as Kubernetes / OKD node
16:35:41 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/93
16:36:02 <dustymabe> I added this to the agenda
16:36:13 <dustymabe> it is a broad topic but I'd like to slice out a piece of it
16:36:39 <lorbus> perfect. I have some thoughts on this from the OKD WG point of view
16:36:42 <dustymabe> strigazi: and the Openstack Magnum team have used FAH and system containers to run kubernetes in the past
16:36:52 <red_beard> and i have some from the _non_ k8s point of view :)
16:37:00 <dustymabe> they are starting to play around with FCOS
16:37:10 <strigazi> we used a bunch of atomic system containers
16:37:15 <brtknr> o/
16:37:34 <dustymabe> background: Atomic CLI (used to set up system containers) is going away (python based and now not really maintained by that group)
16:37:50 <dustymabe> so they are wondering on the path forward
16:38:08 <red_beard> for clarification, what was the technology for "system" containers?
16:38:13 <dustymabe> It seems to me that podman has grown some "Atomic CLI" like features with regards to system containers
16:38:24 <strigazi> ostree for image storage
16:38:44 <strigazi> templated systemd unit with runc as exec
16:38:47 <strigazi> red_beard: ^^
16:38:55 <red_beard> :thumbs_up:
16:38:58 <red_beard> thanks
16:39:02 <dustymabe> correct ^^ it was a tool that helps set that up easily for a user
16:40:12 <dustymabe> `podman generate systemd` seems like it will help
16:40:16 <dustymabe> but I don't know what the other gaps are
16:40:34 <strigazi> systemd unit with podman run could work. Does podman use a daemon to run containers?
16:40:49 <dustymabe> no
16:40:49 <jdoss> No daemons
16:41:01 <red_beard> strigazi: is there anything funky which needs to be brought in which isn't supported by current namespace mechanisms?
16:42:00 <strigazi> nothing from the top of my head up to k8s v1.15, though k8s 1.16 feels like a new beast
16:42:37 <lorbus> The dependencies for running Kubernetes or OKD are not in the current FCOS compose
16:42:38 <red_beard> oh, so this is kubernetes specific.  this isn't a more generic mechanism.
16:42:53 <lorbus> kubectl (oc), hyperkube and cri-o namely
16:43:08 <dustymabe> would a worst case scenario be that one would use ignition to download the kubelet and kubectl and land a systemd unit to start the kubelet ?
16:43:15 <dustymabe> strigazi: ^^ ?
16:43:18 <lorbus> it would add about half a gb of size to the compose
16:43:35 <dustymabe> lorbus: let's close off the discussion with strigazi real quick
16:44:02 <strigazi> dustymabe: atm we run kubelet in a container which is pulled at node first boot
16:44:33 <strigazi> dustymabe: we can do the same with ignition, we would prefer to run it in a container though, even super privileged
16:44:37 <dustymabe> strigazi: right. For some reason I think that running the kubelet containerized is not supported upstream any longer
16:45:25 <dustymabe> but yes.. running in a faux container (i.e. basically just chroot) may still work as you desired
16:45:45 * dustymabe is reminded that he'd like to try systemd portable services one day
16:46:05 <dustymabe> strigazi: do you have some options you can try until we meet again ?
16:46:12 <strigazi> dustymabe: that would be ideal ^^ but building and distributing the images was strange
16:46:30 <dustymabe> strigazi: you mean portable services ?
16:46:35 <strigazi> sure, will try with podman
16:46:40 <strigazi> dustymabe: yes
16:46:45 <kaeso[m]> (portable services are still containers, just not run by runc)
16:46:59 <strigazi> ^ +1
16:47:10 <dustymabe> honestly if portable services get you what you want you could just download the directory structure from an OCI image using podman :)
16:47:15 <strigazi> but no podman/docker/buildah build
16:47:19 <dustymabe> lots of different ways to solve problems, some more elegant than others
16:47:53 <strigazi> we will give that one more go
16:48:07 <dustymabe> #info strigazi will trying using podman to replace atomic CLI for system containers to run the kubelet for Openstack Magnum
16:48:11 <dustymabe> thanks strigazi !
16:48:27 <strigazi> thank you all again
16:48:33 <dustymabe> lorbus: you had an item (and red_beard too, after lorbus )
16:48:55 <lorbus> yes =)
16:49:17 <strigazi> one the reasons we fight it with fcos is this meeting and the community :)
16:49:28 <dustymabe> strigazi: :) we love you too!
16:49:45 <dustymabe> except you are the community - so you love you!
16:49:46 <red_beard> for clarification, i have _opinions_ which can easily get filed in the bit bucket
16:49:48 <lorbus> so OKD needs to have a specific set of okd-clients, okd-hyperkube and cri-o in the base compose to support running a 4.x cluster
16:50:18 <dustymabe> lorbus: correct
16:50:32 <lorbus> We cannot layer them in after the fact. They are not upstream. And they add a lot of size to the ostree compose
16:51:15 <dustymabe> lorbus: let us know when you're finished and then we can ask questions
16:51:38 <lorbus> Taking into account all of these downsides, I wanted to bring up the idea of making an OKD COS Compose again
16:52:41 <lorbus> sticking to the OpenShift/OKD version of everything will not make K8s upstream users happy, but I guess its a plus from current situation, where there is no K8s in the base at all
16:53:36 <lorbus> I'm still figuring out the RPM CI buils on OpenShift's Prow, so I'd ideally like to plan ahead now, in order to know what to do with them when we have them
16:54:17 <lorbus> (unfoirtunately I'm also on sick leave this week, because flu :( )
16:54:58 <jlebon> re. "the add a lot of size to the ostree compose", do you have an estimate?
16:55:19 <lorbus> half a gb
16:56:58 <dustymabe> lorbus: done?
16:57:38 <lorbus> yeah. if you could all give it some thought, whether you'd like it in FCOS or create a separate ostree compose for OKD, thatd be great
16:57:43 <bgilbert> my opinion is still that if any k8s distribution needs its own compose, we're doing something very wrong
16:57:49 <strigazi> imo, adding the deps in the base os is a half measure. When we started running everything in containers the velocity of our project reallly took off
16:57:50 <red_beard> lorbus: bgilbert bbreard and i talked about this back in May while in Boston.  Personally, I'm a fan of a compose/spin/etc *especially* because it ensures that the workflow will be maintained for other users who need to have opinionated FCOS builds.
16:58:46 <red_beard> but, as bgilbert stated, if it becomes that kindof maintenance burden... we're asking folks running those composes/spins to become _distro maintainers_
16:58:49 <bgilbert> if everyone has their own FCOS build, the fragmentation leads to a lot of additional support burden etc.
16:58:50 <red_beard> which is a HUGE ask
16:58:52 <ajeddeloh> red_beard: that means that we're going to get skew between FCOS and those unless the maintainers are _really_ up to date with us
16:59:11 <lorbus> red_beard: That sort of reflects my thinking :)
17:00:05 <red_beard> ajeddeloh: which is why academically i fall on the side of bgilbert
17:00:06 <bgilbert> personally, if the cost of avoiding fragmentation is 500 MB more in the image, I'm okay paying it
17:00:07 <lorbus> Not everybody has an OKD to support tho. And even if, it'd be great if we had a way to create those spins.
17:00:31 <red_beard> i just know that (from practical experience) 90% of my usage of CoreOS/Container Linux does _not_ involve kubernetes
17:00:58 <lorbus> also, FCOS would then be hard-bound to OKDs version of everything
17:01:01 <dustymabe> lorbus: spins are easy to POC, hard to maintain over time unfortunatley
17:01:26 <bgilbert> a big part of the benefit of FCOS is the update model, including the backport/release/rollout decisions.  any downstream that doesn't inherit those decisions more-or-less in real time loses the benefit.
17:01:28 <red_beard> yeah, a spin works OK because it still uses the same RPM repos
17:01:40 <kaeso[m]> the problem with "k8s-in-OS" is that is couples the updates lifecycles, which either means "the OS dictates which k8s version you run" or "k8s dictates which OS version you run"
17:01:41 <red_beard> bgilbert +1
17:01:51 <lorbus> I was envisioning something that lives right next to the main FCOS compose. It would get rebuilt each time as well
17:01:58 <dustymabe> kaeso[m]: +1 - that one is going to be a hard one to solve
17:02:06 <lorbus> OKD on FCOS will not support Zincati as update agent. at least initially
17:02:35 <lorbus> the coupled lifecycle is actually a feature in OpenShift 4, that we want to keep in OKD.
17:03:15 <lorbus> (I forgot to mention the OpenShift specific machine-config-daemon rpm, which will also be needed on OKD hosts)
17:03:25 <bgilbert> what do we need to do now to keep things moving?
17:03:39 <dustymabe> lorbus: we should probably schedule a vFAD (virtual Fedora Activity Day) where we talk about all of this
17:03:50 <bgilbert> everyone talking here has been through several of these conversations already :-)
17:03:53 <dustymabe> There is a lot to bite off and chew
17:04:12 <lorbus> bgilbert: Fedora RPM builds on OpenShift's Prow (we need an entire new pipeline, base image etc for that too)
17:04:32 <lorbus> and then a - any - *COS compose including them
17:04:55 <bgilbert> that was already the plan, no?
17:05:25 <lorbus> that has not changed, yes. I just want to make sure we are on one page reagrding the compose that comes after
17:06:14 <bgilbert> I'm not sure talking about it further is going to get us there, unfortunately
17:06:23 <dustymabe> lorbus: ehh I don't know if I know enough to say - I need to fully be in the context for that and I think we need some dedicated time to talk about it
17:06:30 <bgilbert> I'm +1 to including in the base image, at least as a test
17:06:52 <dustymabe> yeah but there are some implications there regarding where rpms come from etc..
17:07:04 <bgilbert> if that turns out not to be a good idea, we'll then have technical reasons to make a different decision
17:07:06 <lorbus> ok. more time for this topic sounds reasonable
17:07:06 <dustymabe> all this is to say: let's schedule some other time to talk about it - probably high bandwidth ?
17:07:38 <dustymabe> ack/nack?
17:07:42 <lorbus> ack. please note I'll be on PTO till Oct 12
17:07:49 <bgilbert> sgtm
17:07:53 <ajeddeloh> +1
17:08:02 <jlebon> +1
17:08:03 <lorbus> and sick this week, which is really shitty timing
17:08:20 <dustymabe> #topic Set up annex OSTree repo
17:08:27 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/266
17:08:48 <dustymabe> #info please add your votes to the proposal in the ticket: https://github.com/coreos/fedora-coreos-tracker/issues/266#issuecomment-535072644
17:09:13 <dustymabe> this was mostly to draw attention - we probably don't need to discuss much here unless someone has a question they want answered
17:09:34 * dustymabe will leave the floor open on this topic for another minute and then move on
17:10:07 <jlebon> dustymabe: did you find out about pruning? we're looking at a lot composes going in there once all the streams are set up
17:10:57 <dustymabe> jlebon: we're not currently pruning - I wrote a script a long time ago to prune but it got stuck in code review
17:11:11 <dustymabe> as part of this effort I've added a task to revive that and also run it in openshift now
17:11:30 <jlebon> just shortly on this topic: what is nice about this approach is that it allows us to place prod content much closer to its final destination, reducing the number of things that could go wrong during the release step
17:11:58 <jlebon> plus the fact we don't have to maintain another OSTree repo means +1 for me in general
17:12:40 <dustymabe> ok will move to next ticket here in 30s unless discussion continues
17:13:38 <dustymabe> #topic python3 getting pulled in by crypto-policies
17:13:43 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/280
17:13:52 <dustymabe> our favorite subject!! \o/
17:14:16 <kaeso[m]> /o\
17:14:38 <jlebon> we should have a CI/kola test that checks whether python sneaks back in
17:14:57 <jlebon> it's known to be sneaky like that
17:14:57 <bgilbert> I skimmed the diff.  rewriting that script in Python does seem like an improvement
17:14:58 <dustymabe> looks like update-crypto-policies was rewritten in python
17:15:22 <dustymabe> bgilbert: +1
17:15:37 <dustymabe> a couple things here:
17:15:43 <dustymabe> 1 - how do we handle this specific case
17:15:48 <bgilbert> does anyone have ideas on how to ask maintainers to stop replacing their hacky shell scripts with nice clean Python?
17:15:52 <dustymabe> 2 - how do we handle cases like this in the future
17:15:54 <bgilbert> dustymabe: +1
17:16:30 <dustymabe> 3 - what do we do when something we really need insists on using python
17:16:39 <ajeddeloh> pardon my ignorance, but what does that script do?
17:16:51 * dustymabe in the same boat as ajeddeloh
17:17:09 <lorbus> maybe we can as the packaging committee to make a guideline stating one shall not pull in unnecessary runtimes?
17:17:18 <slowrie> I think this is going to be a hard sell. If I was a maintainer and I re-wrote something I hated in bash in a much cleaner python solution and someone came to me asking me to go back I'd probably show them the door.
17:17:29 <bgilbert> slowrie: +1
17:17:39 <dustymabe> slowrie: yeah I agree
17:17:55 <slowrie> And I don't think we have nearly enough manpower to support custom commits on top ourselves to backport things to bash
17:17:57 <dustymabe> the only thing that would help is if we volunteered to rewrite it for them in another language
17:17:59 <lorbus> I dont really agree here
17:18:02 <kaeso[m]> bgilbert: throw the systemd-eating-sockpuppet at it?
17:18:22 <ajeddeloh> We should make sure the script does something that makes sense for us. Like if it's handling configuration that should be handled via just writing config files, we could ask them to split it out
17:18:23 <dustymabe> volunteering to rewrite doesn't scale
17:19:24 <dustymabe> do I hear a proposal for solving #1 ?
17:19:58 <ajeddeloh> I think we need to understand what the script does first
17:20:15 <jlebon> my understanding is it tweaks various knobs that different libraries/apps respond to to make sure the crypto policies are consistent across the board
17:20:18 <dustymabe> ajeddeloh: there are a few outcomes to that investigation
17:20:28 <dustymabe> which we can predict
17:20:39 <dustymabe> and plan accordingly
17:21:37 <ajeddeloh> skimming it, it looking like it's just doing configuration
17:21:50 <bgilbert> maintainers probably don't even know this is a thing.  we could do a devel-announce post pointing out that we're trying to keep Python out of FCOS
17:22:04 <bgilbert> I'm sure the downthread reaction would be... interesting
17:22:05 <ajeddeloh> like writing out symlinks for configs
17:22:35 <ajeddeloh> and running scripts to do configuration is an antipattern on fcos
17:22:39 <lorbus> I think the ML announce is a good idea. we need to be offensive about communicating this
17:22:42 <dustymabe> yeah I don't suspect people will much care since python is so pervasive in this world we're in - but might be worth a shot
17:22:55 <lorbus> people will have opinions either way, let them vent
17:24:06 <dustymabe> so we did talk about a system python option in the past (one that only we can use) - would it be better for us to implement that instead of fighting each one of these battles individually ?
17:24:32 <dustymabe> or investigate implementing it (techincally it might be challenging)
17:24:48 <bgilbert> dustymabe: I'm semi-resigned to doing it eventually, but since we're Python-free right now, I think it's worth fighting a couple more of these battles
17:25:06 <dustymabe> action items I see
17:25:09 <ajeddeloh> Can we instead request that people make helper management scripts into subpackages?
17:25:14 <bgilbert> we don't really know how hard it is to hold the line
17:25:32 <dustymabe> A - investigate crypto policies and see how we can move forward
17:25:46 <dustymabe> B - start ML discussion about python in Fedora
17:25:55 <kaeso[m]> dustymabe: in *this specific case* it would yes, as I don't see any addition python-library dependency
17:26:38 <dustymabe> do we agree with those two action items?
17:26:42 <kaeso[m]> dustymabe: in the general case, the web of libraries/dependencies is what we are fearing
17:26:42 <ajeddeloh> +1
17:26:44 <dustymabe> anyone want to volunteer for them
17:26:48 <bgilbert> +1
17:26:54 <ajeddeloh> I can take A
17:27:01 <dustymabe> thanks ajeddeloh
17:27:14 <dustymabe> any takers for B? I really don't have the juice to run that one right now
17:27:25 * ajeddeloh has been peeking at it just now, wonders if we can do something that reads their .pol files and generates ign config snippets
17:27:35 <bgilbert> we can kick that one down the road right now
17:27:39 <ajeddeloh> which would be pretty slick
17:27:44 <lorbus> thanks andrew. If nobody else is up for B, I'll do it, but pbrobaly note before the weekend
17:27:45 <bgilbert> I don't think we need to do B before stable, and if it'll be a timesink, we should focus on stable
17:27:47 <dustymabe> #action ajeddeloh to investigate the addition of python in update-crypto-policies #280
17:27:59 <dustymabe> bgilbert: +1
17:28:06 <dustymabe> we'll kick the ML discussion down the road
17:28:11 <lorbus> dustymabe: action me for B, next week
17:28:12 <bgilbert> +1
17:28:45 <lorbus> I really think this is something to be at least made more public as early as possible.
17:28:45 <bgilbert> lorbus: honestly, I think I'd rather defer
17:28:51 <dustymabe> lorbus: for B we need to be careful about how we communicate with the rest of the Fedora community
17:28:59 <bgilbert> dustymabe++
17:29:06 <dustymabe> we don't want to seem abrasive and we need to explain the benefits
17:29:11 <lorbus> ok then
17:29:34 <dustymabe> it will take some finesse (spelling) :)
17:29:42 <bgilbert> we should do those things, but we don't need to be drawn into a major comms effort atm
17:29:43 <dustymabe> so we probably don't want to rush it
17:29:50 <dustymabe> #topic open floor
17:29:57 <dustymabe> 10 whole seconds for open floor
17:29:59 <lorbus> this can be seen as part of the ongoing Fedora Minimization effort.
17:30:03 <ajeddeloh> let do a gdoc so we can revise it, make sure everyone thinks it's good?
17:30:10 <lorbus> but eh, later will do :)
17:30:11 <ajeddeloh> quick open floor thing
17:30:50 <dustymabe> ajeddeloh: it's your floor :)
17:30:52 <ajeddeloh> strigazi: wrt openstack+heat+ignition and multipart, ignition has it's own mime type, can that be set correctly?
17:31:13 <bgilbert> ajeddeloh: +1
17:31:19 <ajeddeloh> it would make finding the ignition config in the set of multipart parts easier
17:31:26 <ajeddeloh> and less error prone
17:32:38 <bgilbert> my client shows him as idle
17:32:55 <dustymabe> discuss in ticket?
17:32:58 <ajeddeloh> I left a comment in the bug anyway, we can close out
17:33:04 <dustymabe> he is EU based so may have left early
17:33:32 <dustymabe> #endmeeting