16:29:51 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:51 <zodbot> Meeting started Wed Sep 19 16:29:51 2018 UTC.
16:29:51 <zodbot> This meeting is logged and archived in a public location.
16:29:51 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:29:51 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:51 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:55 <dustymabe> #topic roll call
16:30:02 <dustymabe> .hello2 dustymabe
16:30:04 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:06 <yzhang> .hello2
16:30:07 <zodbot> yzhang: yzhang 'Yu Qi Zhang' <jzehrarnyg@gmail.com>
16:30:09 <geoff-> geoff-: Geoff Levand <geoff@infradead.org>
16:30:19 <slowrie> .hello2
16:30:19 <bhavin192> .hello2
16:30:19 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:22 <ksinny> .hello sinnykumari
16:30:22 <zodbot> bhavin192: bhavin192 'Bhavin Gandhi' <bhavin7392@gmail.com>
16:30:27 <mnguyen_> .hello mnguyen
16:30:28 <zodbot> ksinny: sinnykumari 'Sinny Kumari' <ksinny@gmail.com>
16:30:30 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:30:31 <miabbott> .hello miabbott
16:30:33 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:30:45 <rfairley> .hello rfairleyredhat
16:30:46 <zodbot> rfairley: rfairleyredhat 'Robert Fairley' <rfairley@redhat.com>
16:31:03 <ajeddeloh> .hello2
16:31:04 <zodbot> ajeddeloh: ajeddeloh 'Andrew Jeddeloh' <andrew.jeddeloh@redhat.com>
16:31:07 * dustymabe waves at everyone
16:31:16 * ajeddeloh waves back
16:31:34 <dustymabe> #chair yzhang geoff- slowrie bhavin192 ksinny mnguyen_ miabbott rfairley ajeddeloh
16:31:34 <zodbot> Current chairs: ajeddeloh bhavin192 dustymabe geoff- ksinny miabbott mnguyen_ rfairley slowrie yzhang
16:31:49 <mskarbek> .hello2
16:31:50 <zodbot> mskarbek: mskarbek 'None' <redhat@skarbek.name>
16:31:53 <jlebon> .hello2
16:31:56 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:58 <dustymabe> i think walters said last week he has a conflict for the first 30 minutes
16:32:06 <jbrooks> .fas jasonbrooks
16:32:07 <zodbot> jbrooks: jasonbrooks 'Jason Brooks' <jbrooks@redhat.com>
16:32:25 <dustymabe> haha jlebon
16:32:30 <dustymabe> $ curl -L jlebon.com
16:32:31 <dustymabe> 404
16:32:38 <jlebon> :)
16:32:44 <davdunc> .hello davdunc
16:32:45 <zodbot> davdunc: davdunc 'David Duncan' <davdunc@amazon.com>
16:32:50 <dustymabe> #chair mskarbek jbrooks jlebon davdunc
16:32:50 <zodbot> Current chairs: ajeddeloh bhavin192 davdunc dustymabe geoff- jbrooks jlebon ksinny miabbott mnguyen_ mskarbek rfairley slowrie yzhang
16:33:13 <dustymabe> ok we can get started
16:33:20 <dustymabe> #topic Action items from last meeting
16:33:25 <kaeso> .himynameis lucab
16:33:25 <dustymabe> last meeting we had:
16:33:28 <zodbot> kaeso: lucab 'Slim Shady' <lucab@redhat.com>
16:33:35 <dustymabe> nice!
16:33:41 <dustymabe> that's awesome
16:34:11 <dustymabe> * dustymabe to gather more information on frequency of python (and
16:34:13 <dustymabe> related library) security issues in Fedora
16:34:20 <dustymabe> ^^ that was the only action item from last meeting
16:34:28 <dustymabe> I spent the last 30 minutes looking at bodhi updates
16:35:12 <dustymabe> #info dustymabe investigated python and python library security issues and fedora and wrote up a summary: https://github.com/coreos/fedora-coreos-tracker/issues/32#issuecomment-422867659
16:35:29 <dustymabe> we can discuss it further today if we get to that ticket in the meeting items
16:36:01 <dustymabe> if no one objects I'll move on to meeting items (and try to prioritize meeting items we didn't get to last week)
16:36:18 <davdunc> nice list dustymabe
16:36:21 <kaeso> dustymabe: yes, please
16:36:48 <dustymabe> #topic Do we separate /boot and the ESP
16:36:55 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/43
16:37:20 <dustymabe> ajeddeloh: want to give us some background for discussion?
16:37:34 <ajeddeloh> sure
16:37:59 <ajeddeloh> so CL has /boot and /esp as the same partition, fedora does not
16:38:40 <dustymabe> ESP == EFI system partition in case anyone didn't know
16:38:42 <ajeddeloh> on the upside of combining them, there's one less partition to deal with, on the downside, it means /boot must be fat32
16:39:09 <ajeddeloh> this is also somewhat dependent on what we do for automatic rollback
16:39:50 <ajeddeloh> I'm in favor of the combined one, but not super strongly so
16:39:58 <dustymabe> agree. so the best we can do today is come up with what we think we want to do, which may change based on the automatic rollback discussion ?
16:40:10 <ajeddeloh> sgtm
16:40:32 <dustymabe> does anyone have a strong opinion for OR against combined /boot + ESP ?
16:40:33 <ajeddeloh> my thought is combine them since that's (in my mind) the simplest route
16:40:42 <ajeddeloh> and if we hit some blocker, revisit
16:41:06 <mskarbek> /boot + ESP conflicts with selinux for example - no way to add labels on fat32
16:41:41 <dustymabe> mskarbek: that's a good point
16:41:53 <kaeso> mskarbek: do you mean "/boot on ESP"?
16:41:58 <mskarbek> yes
16:42:08 <ashcrow> .hello smilner
16:42:09 <zodbot> ashcrow: smilner 'None' <smilner@redhat.com>
16:42:43 <ajeddeloh> hrm yeah that is a good point
16:43:08 <jlebon> hmm, yeah hadn't thought about that. another issue i know Colin mentions often is that fat32 doesn't do atomic replacements
16:43:31 <ajeddeloh> jlebon: depending on how automatic rollback is handled, we may not need that
16:43:50 <ajeddeloh> (hence the "this is kinda dependent on what we do for that
16:43:54 <dustymabe> mskarbek: do you mind making the comment about selinux in the ticket ?
16:43:56 <jlebon> ahh understood
16:44:38 * dustymabe was hoping to go with combined boot/ESP. wish we could have a decent filesystem support for ESP :)
16:44:40 <mskarbek> to be honest I'm currently using this setup with Fedora Workstation to avoid duplication on initrd files on ESP partition becouse I'm using systemd-boot not grub and I simply ignore selinux warnings during dracut generation process but I don't think those warning will be simply ignored in normal scenarios
16:45:13 <kaeso> jlebon: is rpm-ostree managing files under /boot? do the hardlink forest approach work there?
16:45:42 <mskarbek> dustymabe: sure
16:46:01 <jlebon> kaeso: ostree drops the new kernel/initramfs there, yeah. IIRC it tries to hardlink if it can (/boot on /)
16:46:31 <kaeso> jlebon: ah, right, the latter part of the question doesn't matter anyway in our case
16:46:40 <ajeddeloh> hmm that's a thought actually... do we need a seperate /boot at all?
16:46:42 <dustymabe> dumb question.. could we just leave /boot in / ?
16:46:47 <dustymabe> ajeddeloh: you beat me
16:46:49 <ajeddeloh> dustymabe: hahaa
16:47:02 <kaeso> ajeddeloh, dustymabe: encrypted rootfs?
16:47:09 <ajeddeloh> damn. right
16:47:43 <jlebon> i'm also not sure what the state of that is for grub
16:47:54 <kaeso> which has a similar problem to selinux, uid/gid/perms of files over there are fixed at mount time
16:48:00 <dustymabe> yeah whats funny is i actually have /boot/ in / but i configured grub to prompt me for a password and decrypt them
16:48:00 <jlebon> i know there's been community work to have it supported with uboot
16:48:50 <ajeddeloh> jlebon: the state of what
16:48:56 <kaeso> dustymabe: sure, interactively may work, but we were hoping to make it automatic via cloud-HSM
16:49:17 <dustymabe> ok, so i think the sentiment is we'd like to have boot and ESP combined but given the limitations of fat32 we don't think we'll be able to do that
16:49:23 <jlebon> ajeddeloh: the state of /boot on /, walters would know
16:49:26 <ajeddeloh> ah
16:49:43 <ajeddeloh> dustymabe: sgtm
16:49:44 <dustymabe> jlebon: it works :)
16:49:44 <kaeso> overall I think we should aim at making them combined
16:50:05 <ajeddeloh> kaeso: /boot and ESP or /boot and /
16:50:09 <dustymabe> but anaconda will tell you you're wrong and laugh at you if you try it
16:50:21 <kaeso> /boot == ESP
16:50:22 <jlebon> dustymabe: ahh, sweet!
16:51:47 <dustymabe> #proposed our goal is to make a combined boot and ESP partition, but we don't have super strong opinions here and foresee possible issues with fat32 not supporting xattrs. fallback plan is separate /boot and ESP
16:52:09 <ajeddeloh> +1
16:52:21 <dustymabe> kaeso: ^^ WDYT?
16:52:32 <jlebon> SGTM
16:52:54 <kaeso> +1
16:53:15 <geoff-> sgtm, I guess we'll need to split them...
16:53:24 <dustymabe> mskarbek: since you were the one who brought up selinux issue WDYT ^^
16:53:47 <sayan> .hello sayanchowdhury
16:53:48 <zodbot> sayan: sayanchowdhury 'Sayan Chowdhury' <sayan.chowdhury2012@gmail.com>
16:53:49 <dustymabe> quick someone go change the spec and tell EFI to support XFS
16:54:17 <mskarbek> +1 for combining but somebody will have to quiet selinux warnings in logs :P
16:54:21 <ajeddeloh> switch off of utf-16 while you're at it
16:54:48 <dustymabe> cool. i'll bring this up in the ticket
16:54:59 <dustymabe> #agreed proposed our goal is to make a combined boot and ESP partition, but we don't have super strong opinions here and foresee possible issues with fat32 not supporting xattrs. fallback plan is separate /boot and ESP
16:55:06 <dustymabe> moving on to next ticket
16:55:21 <dustymabe> #topic Determine how to handle automatic rollback
16:55:28 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/47
16:55:47 <dustymabe> ajeddeloh: :) - you again
16:56:13 <ajeddeloh> sure
16:56:50 <ajeddeloh> We want automatic rollback. CL uses gpt partition attributes but we can't do that since we're not using A/B partitions
16:57:06 <ajeddeloh> we want to ensure bad kernels are handled sa well
16:57:22 <ajeddeloh> that means grub/grub config needs the logic to pick what to boot
16:57:43 <ajeddeloh> CL today has a static grub config which knows how to pick which to boot
16:58:28 <ajeddeloh> grub has the "grubenv" which is 1024 byes of environment variables that persist across boots
16:58:48 <ajeddeloh> that's probably where we want to store the metadata about boot sucess/failure/etc
17:00:07 <ajeddeloh> imo the static grub config is nice, I'd like to preserve that if we can. All ostree would need to do is drop files in /boot and change the grubenv to say "hey, this is new, try it", then some other service would need to mark boots sucessful
17:00:37 <kaeso> lorbus[m]: if you are around ^^^
17:00:38 <ashcrow> ajeddeloh: would an other service be like greenboot? Or would it be something new need to add?
17:01:41 <ajeddeloh> ashcrow: yeah, I (still) haven't looked into greenboot in depth
17:01:46 <jlebon> that might be an orthogonal decision. greenboot is about how to mark a boot successful
17:02:00 <bgilbert> .hello2
17:02:01 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
17:02:06 <bgilbert> sorry I'm late
17:02:28 <jlebon> i really like the idea of a static grub config, though i don't have much experience with it
17:02:28 <ajeddeloh> but yeah, having something like greenboot would be good so users can have their own definitions of "success"
17:03:08 <ajeddeloh> although if a user is reporting metrics back, we should have another target of "did the system come up at all" as well
17:03:10 <dustymabe> #chair bgilbert
17:03:10 <zodbot> Current chairs: ajeddeloh bgilbert bhavin192 davdunc dustymabe geoff- jbrooks jlebon ksinny miabbott mnguyen_ mskarbek rfairley slowrie yzhang
17:03:29 <ajeddeloh> so users that have brittle setups don't cloud the metrics
17:03:36 <dustymabe> ajeddeloh: i responded to the "list of goals" in the ticket
17:04:54 <ajeddeloh> I agree with that  response
17:05:21 <ajeddeloh> except the last one
17:05:21 <dustymabe> i know lorbus[m] has been AFK trying to rest - he should be back soon and I think he'll have some good input here and we might even be able to get him to work on it :)
17:05:59 <ajeddeloh> When doing an upgrade we should keep 3 around until the upgrade is marked successful
17:06:28 <dustymabe> ajeddeloh: that sounds nice but can you think of a good reason why ?
17:06:39 <ajeddeloh> the TCP bug comes to mind
17:06:41 <dustymabe> it honestly makes the "logic" a lot harder if you have 3 IMHO
17:07:06 <ajeddeloh> basically don't overwrite your fallback
17:07:19 <dustymabe> but we would always have the fallback
17:07:33 <bgilbert> i.e., being able to upgrade out of a bug without overwriting the non-buggy version
17:07:38 <kaeso> dustymabe: a seemingly good update which turns out to be subtly broken
17:08:11 <dustymabe> kaeso: that's an interesting case
17:08:19 <bgilbert> I guess it's less of an issue here.  CL clobbers the previous version at the beginning of an update, rpm-ostree does it at the end
17:08:27 <ajeddeloh> I'd prefer to not spin up another nginx server that kills itself every 5 minutes
17:08:54 <dustymabe> real quick scenario
17:09:20 <kaeso> dustymabe: I think ajeddeloh is concerned if one update is broken, and the next one is slightly-broken, you may end with none good-enough to get out of that
17:09:35 <dustymabe> I currently have A, B. A is an old good update, B is the currently running update (which we think is good)
17:09:53 <dustymabe> i run upgrade to C which is bad
17:09:59 <dustymabe> so now I have B, C
17:10:04 <dustymabe> but C fails to boot properly
17:10:07 <dustymabe> so we rollback to B
17:10:17 <dustymabe> but you're saying B is broken, but we didn't know it was broken ?
17:10:23 <bgilbert> the scenario was:
17:10:45 <bgilbert> A was good, B was good enough to boot but not good enough to download an update properly.  in CL, the act of pushing C would overwrite A
17:11:14 <bgilbert> so we couldn't preserve the option to tell users "manually roll back to A"
17:11:24 <jlebon> hmm, in the ostree case though, if B is not good enough to update, it wouldn't get to the point of garbage collecting A
17:11:37 <jlebon> ahh ok bgilbert said this higher up
17:11:56 <bgilbert> so I'm less concerned with this particular scenario here
17:11:58 <ajeddeloh> as long as we don't delete A until C is ready to boot, we're fine
17:12:39 <ajeddeloh> it sounds like ostree already handles this correctly
17:12:41 <dustymabe> ok. ajeddeloh can you make sure when we get closer to a finished Fedora 30, this concern is addressed ?
17:12:53 <ajeddeloh> +1
17:13:17 <dustymabe> I don't want to leave it behind, because I think it's valid. we do have the option of keeping 3 deployments (the functionality already exists)
17:13:31 <dustymabe> but I know in greenboot we simplified our logic quite a bit by assuming only two
17:13:39 <dustymabe> https://pagure.io/fedora-iot/issue/12#comment-523560
17:14:26 <dustymabe> ajeddeloh: if you don't mind can you capture relevant parts of this conversation for the ticket ?
17:14:45 <ajeddeloh> yup
17:14:58 <dustymabe> ok next topic is one left over from last week
17:15:17 <dustymabe> #topic Allow binaries written in Python via a "platform python" style approach
17:15:21 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/32
17:16:12 <dustymabe> as I said earlier I added some rough data of python dep security issues in the ticket: https://github.com/coreos/fedora-coreos-tracker/issues/32#issuecomment-422867659
17:16:53 <dustymabe> i just took what is currently in the image spit out by https://github.com/coreos/fedora-coreos-config repo
17:17:17 <dustymabe> once we try to weed out our python deps that list should be shorter, but it's a good starting point
17:17:34 <jlebon> dustymabe: nice investigation
17:17:53 <ksinny> indeed
17:17:55 <ajeddeloh> looking at the dep list, what is authconfig?
17:18:22 <ajeddeloh> and atomic-registries
17:18:40 <kaeso> I know I am biased, but there seems to be general consensus that we should not ship python interpreters and that we are implicitly moving into the "how" part
17:18:40 <jlebon> it's a way to customize the PAM stack using configs
17:18:51 <jlebon> though actually it's been obsoleted by authselect
17:19:03 <ajeddeloh> nfs-tools and xfs-progs are actually optional dependencies. We ship those on CL without python
17:19:41 <jlebon> atomic-registries i think is used by podman for parsing out registry configs
17:20:13 <jlebon> in /etc/containers/registries.conf
17:20:14 <ajeddeloh> the selinux tools you can do without python, but its more awkward iirc. Plus you're probably not going to be messing with selinux much as a user
17:21:03 <dustymabe> kaeso: i think it's we'd like to not ship python, unless we want to for X,Y,Z tools that would make our life easier. There is a list of reasons why we don't want to ship python but we've analyzed each one to see which ones can be achieved by just shipping a "system (not for users at all) python"
17:21:16 <dustymabe> that list is here: https://github.com/coreos/fedora-coreos-tracker/issues/32#issuecomment-420808195
17:21:27 <jlebon> did we address ansible already? how do we think about config mgmt in FCOS? purely through ignition?
17:21:56 <dustymabe> jlebon: we have not considered ansible up until this point.. I think the general thoughts so far is that's not something we are explicitly supporting
17:22:01 <slowrie> imo it should be purely through ignition. otherwise we leave the immutable host realm
17:22:16 <kaeso> jlebon: it's quite uncomfortable, but you can bring your own python for ansible/salt
17:22:17 <slowrie> and that comes with a lot of trade-offs for upgradability, maintenence, etc.
17:22:52 <kaeso> jlebon: you'll likely having a mismatch/miss on some library with the OS one anyway
17:23:39 <dustymabe> so "system python" in this case would not be to support ansible IMHO
17:23:49 <ashcrow> dustymabe: correct.
17:23:54 <dustymabe> it would be only to support things that we've chosen we need (i.e. firewalld if we were to choose that)
17:24:09 <dustymabe> and if we remove all of the things we need (like firewalld) then we could just remove system python then
17:24:19 <dustymabe> i.e. if firewalld were rewritten
17:24:27 <dustymabe> or we chose to use another (new) tool
17:25:09 <ajeddeloh> has anyone investigated running firewalld in a conatiner?
17:25:10 <kaeso> I noticed in RHEL7 that several python packages were providing both pylibs and binaries in PATH
17:25:11 <dustymabe> kaeso: ^^ does that help explain it. I think you missed the meeting last week
17:25:39 <kaeso> dustymabe: yes, it's clear, I was speaking about the ansible/salt scenario
17:26:03 <jlebon> i think purely through ignition works, though it'll be a learning curve for folks coming from FAH
17:27:05 <dustymabe> ajeddeloh: i'm sure we could run firewalld in a container, but my experience with things like that is it's not maintained properly (i.e. the rpm maintainer doesn't really consider the container that needs to get built) and it also doesn't get maintained properly on the host
17:27:27 <dustymabe> i.e. we upgrade the host but the container is forgotten about somehow
17:28:06 * dustymabe notices we have only a few minutes left
17:29:00 <dustymabe> does anyone have any thoughts on the frequency of security updates that were found in the analysis
17:29:26 <kaeso> so what do we do here: we start by removing all of those packaged and slowly add back split ones, or do we split/remove one by one?
17:29:56 <dustymabe> given that we hope to decrease those dependencies and the frequency listed in the analysis, could we draw any conclusions about #4
17:30:17 <bgilbert> dustymabe: the CVE rate is not thrilling but I think we can probably live with it?
17:30:20 <kaeso> dustymabe: that's somehow a false indicator, usually "less CVEs" just means "nobody had a reason to actively investigate/report CVEs"
17:30:38 <bgilbert> also not every CVE needs an out-of-cycle release
17:31:00 <dustymabe> kaeso: yeah, but if they don't report it how would we ever know we needed an out-of-cycle release?
17:31:50 <dustymabe> ok quickly i'll make a proposal
17:31:51 <kaeso> dustymabe: I see, my comment is unrelated to your question then, sorry
17:33:08 <dustymabe> #proposed we'd really really like to remove python, but if we choose that we want to keep a tool or a few tools in fedora coreos that use python then we think the system python approach given that it solves #1 and #3, which are our primary concerns, would be reasonable.
17:33:37 <dustymabe> feel free to nack that
17:33:42 <dustymabe> and we'll revisit next week
17:34:23 <dustymabe> actually I probably should have started with this
17:34:37 <dustymabe> #undo
17:34:37 <zodbot> Removing item from minutes: <MeetBot.items.Link object at 0x7f56e061c290>
17:34:51 <bgilbert> #redo
17:34:55 <bgilbert> foo
17:34:56 * kaeso grabs a pointer
17:35:21 <dustymabe> #proposed we think security issues for python (#4) aren't significant enough to cause us great hardships during releases
17:36:00 <bgilbert> #link https://github.com/coreos/fedora-coreos-tracker/issues/32
17:36:22 <bgilbert> +0.5
17:36:30 <dustymabe> bgilbert: suggested edit?
17:36:55 <bgilbert> wording is fine.  I just haven't carefully gone through the CVE list
17:37:10 <bgilbert> so, weak agreement
17:37:11 <dustymabe> cool in that case we'll wait til next time and retry again with the above two
17:37:31 <dustymabe> sorry i ran us over and no open floor today
17:37:33 <dustymabe> #topic open floor
17:37:50 <dustymabe> did someone have something they wanted to say that won't take much time
17:38:37 <kaeso> concern: coreos-assembler is already ~400 LoC and shell and I'd like for it not to grown into the ~5000 of `coreos/scripts`
17:38:53 <kaeso> s/and shell/of sh/
17:39:23 <jlebon> but shell has so many guns you can shoot yourself with!
17:39:26 <dustymabe> yeah I think we'd like to rewrite it at some point in a nicer language.. there was an open issue and we discussed it and I think we settled on python
17:39:53 <jlebon> well, we settled on "not Rust", not sure if we hard settled on Python?
17:40:02 <ashcrow> walters: ^
17:40:05 <kaeso> ack, I missed that ticket then
17:40:21 <dustymabe> https://github.com/coreos/coreos-assembler/pull/84
17:40:31 <dustymabe> for some reason I thought that was an "issue" and not a PR
17:40:35 <dustymabe> but I found it ^^
17:40:47 <geoff-> I think shell is OK.  The problem I saw with CL scripts is that it got so big it was hard to refactor.
17:41:11 <dustymabe> kaeso: maybe weigh in on that PR
17:41:17 <kaeso> yup
17:41:21 <dustymabe> or start a new issue somewhere for us to discuss
17:41:32 <dustymabe> ok all closing it out.. sorry 10 minutes over
17:41:36 <dustymabe> #endmeeting