16:29:51 #startmeeting fedora_coreos_meeting 16:29:51 Meeting started Wed Sep 19 16:29:51 2018 UTC. 16:29:51 This meeting is logged and archived in a public location. 16:29:51 The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:29:51 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:29:51 The meeting name has been set to 'fedora_coreos_meeting' 16:29:55 #topic roll call 16:30:02 .hello2 dustymabe 16:30:04 dustymabe: dustymabe 'Dusty Mabe' 16:30:06 .hello2 16:30:07 yzhang: yzhang 'Yu Qi Zhang' 16:30:09 geoff-: Geoff Levand 16:30:19 .hello2 16:30:19 .hello2 16:30:19 slowrie: slowrie 'Stephen Lowrie' 16:30:22 .hello sinnykumari 16:30:22 bhavin192: bhavin192 'Bhavin Gandhi' 16:30:27 .hello mnguyen 16:30:28 ksinny: sinnykumari 'Sinny Kumari' 16:30:30 mnguyen_: mnguyen 'Michael Nguyen' 16:30:31 .hello miabbott 16:30:33 miabbott: miabbott 'Micah Abbott' 16:30:45 .hello rfairleyredhat 16:30:46 rfairley: rfairleyredhat 'Robert Fairley' 16:31:03 .hello2 16:31:04 ajeddeloh: ajeddeloh 'Andrew Jeddeloh' 16:31:07 * dustymabe waves at everyone 16:31:16 * ajeddeloh waves back 16:31:34 #chair yzhang geoff- slowrie bhavin192 ksinny mnguyen_ miabbott rfairley ajeddeloh 16:31:34 Current chairs: ajeddeloh bhavin192 dustymabe geoff- ksinny miabbott mnguyen_ rfairley slowrie yzhang 16:31:49 .hello2 16:31:50 mskarbek: mskarbek 'None' 16:31:53 .hello2 16:31:56 jlebon: jlebon 'None' 16:31:58 i think walters said last week he has a conflict for the first 30 minutes 16:32:06 .fas jasonbrooks 16:32:07 jbrooks: jasonbrooks 'Jason Brooks' 16:32:25 haha jlebon 16:32:30 $ curl -L jlebon.com 16:32:31 404 16:32:38 :) 16:32:44 .hello davdunc 16:32:45 davdunc: davdunc 'David Duncan' 16:32:50 #chair mskarbek jbrooks jlebon davdunc 16:32:50 Current chairs: ajeddeloh bhavin192 davdunc dustymabe geoff- jbrooks jlebon ksinny miabbott mnguyen_ mskarbek rfairley slowrie yzhang 16:33:13 ok we can get started 16:33:20 #topic Action items from last meeting 16:33:25 .himynameis lucab 16:33:25 last meeting we had: 16:33:28 kaeso: lucab 'Slim Shady' 16:33:35 nice! 16:33:41 that's awesome 16:34:11 * dustymabe to gather more information on frequency of python (and 16:34:13 related library) security issues in Fedora 16:34:20 ^^ that was the only action item from last meeting 16:34:28 I spent the last 30 minutes looking at bodhi updates 16:35:12 #info dustymabe investigated python and python library security issues and fedora and wrote up a summary: https://github.com/coreos/fedora-coreos-tracker/issues/32#issuecomment-422867659 16:35:29 we can discuss it further today if we get to that ticket in the meeting items 16:36:01 if no one objects I'll move on to meeting items (and try to prioritize meeting items we didn't get to last week) 16:36:18 nice list dustymabe 16:36:21 dustymabe: yes, please 16:36:48 #topic Do we separate /boot and the ESP 16:36:55 #link https://github.com/coreos/fedora-coreos-tracker/issues/43 16:37:20 ajeddeloh: want to give us some background for discussion? 16:37:34 sure 16:37:59 so CL has /boot and /esp as the same partition, fedora does not 16:38:40 ESP == EFI system partition in case anyone didn't know 16:38:42 on the upside of combining them, there's one less partition to deal with, on the downside, it means /boot must be fat32 16:39:09 this is also somewhat dependent on what we do for automatic rollback 16:39:50 I'm in favor of the combined one, but not super strongly so 16:39:58 agree. so the best we can do today is come up with what we think we want to do, which may change based on the automatic rollback discussion ? 16:40:10 sgtm 16:40:32 does anyone have a strong opinion for OR against combined /boot + ESP ? 16:40:33 my thought is combine them since that's (in my mind) the simplest route 16:40:42 and if we hit some blocker, revisit 16:41:06 /boot + ESP conflicts with selinux for example - no way to add labels on fat32 16:41:41 mskarbek: that's a good point 16:41:53 mskarbek: do you mean "/boot on ESP"? 16:41:58 yes 16:42:08 .hello smilner 16:42:09 ashcrow: smilner 'None' 16:42:43 hrm yeah that is a good point 16:43:08 hmm, yeah hadn't thought about that. another issue i know Colin mentions often is that fat32 doesn't do atomic replacements 16:43:31 jlebon: depending on how automatic rollback is handled, we may not need that 16:43:50 (hence the "this is kinda dependent on what we do for that 16:43:54 mskarbek: do you mind making the comment about selinux in the ticket ? 16:43:56 ahh understood 16:44:38 * dustymabe was hoping to go with combined boot/ESP. wish we could have a decent filesystem support for ESP :) 16:44:40 to be honest I'm currently using this setup with Fedora Workstation to avoid duplication on initrd files on ESP partition becouse I'm using systemd-boot not grub and I simply ignore selinux warnings during dracut generation process but I don't think those warning will be simply ignored in normal scenarios 16:45:13 jlebon: is rpm-ostree managing files under /boot? do the hardlink forest approach work there? 16:45:42 dustymabe: sure 16:46:01 kaeso: ostree drops the new kernel/initramfs there, yeah. IIRC it tries to hardlink if it can (/boot on /) 16:46:31 jlebon: ah, right, the latter part of the question doesn't matter anyway in our case 16:46:40 hmm that's a thought actually... do we need a seperate /boot at all? 16:46:42 dumb question.. could we just leave /boot in / ? 16:46:47 ajeddeloh: you beat me 16:46:49 dustymabe: hahaa 16:47:02 ajeddeloh, dustymabe: encrypted rootfs? 16:47:09 damn. right 16:47:43 i'm also not sure what the state of that is for grub 16:47:54 which has a similar problem to selinux, uid/gid/perms of files over there are fixed at mount time 16:48:00 yeah whats funny is i actually have /boot/ in / but i configured grub to prompt me for a password and decrypt them 16:48:00 i know there's been community work to have it supported with uboot 16:48:50 jlebon: the state of what 16:48:56 dustymabe: sure, interactively may work, but we were hoping to make it automatic via cloud-HSM 16:49:17 ok, so i think the sentiment is we'd like to have boot and ESP combined but given the limitations of fat32 we don't think we'll be able to do that 16:49:23 ajeddeloh: the state of /boot on /, walters would know 16:49:26 ah 16:49:43 dustymabe: sgtm 16:49:44 jlebon: it works :) 16:49:44 overall I think we should aim at making them combined 16:50:05 kaeso: /boot and ESP or /boot and / 16:50:09 but anaconda will tell you you're wrong and laugh at you if you try it 16:50:21 /boot == ESP 16:50:22 dustymabe: ahh, sweet! 16:51:47 #proposed our goal is to make a combined boot and ESP partition, but we don't have super strong opinions here and foresee possible issues with fat32 not supporting xattrs. fallback plan is separate /boot and ESP 16:52:09 +1 16:52:21 kaeso: ^^ WDYT? 16:52:32 SGTM 16:52:54 +1 16:53:15 sgtm, I guess we'll need to split them... 16:53:24 mskarbek: since you were the one who brought up selinux issue WDYT ^^ 16:53:47 .hello sayanchowdhury 16:53:48 sayan: sayanchowdhury 'Sayan Chowdhury' 16:53:49 quick someone go change the spec and tell EFI to support XFS 16:54:17 +1 for combining but somebody will have to quiet selinux warnings in logs :P 16:54:21 switch off of utf-16 while you're at it 16:54:48 cool. i'll bring this up in the ticket 16:54:59 #agreed proposed our goal is to make a combined boot and ESP partition, but we don't have super strong opinions here and foresee possible issues with fat32 not supporting xattrs. fallback plan is separate /boot and ESP 16:55:06 moving on to next ticket 16:55:21 #topic Determine how to handle automatic rollback 16:55:28 #link https://github.com/coreos/fedora-coreos-tracker/issues/47 16:55:47 ajeddeloh: :) - you again 16:56:13 sure 16:56:50 We want automatic rollback. CL uses gpt partition attributes but we can't do that since we're not using A/B partitions 16:57:06 we want to ensure bad kernels are handled sa well 16:57:22 that means grub/grub config needs the logic to pick what to boot 16:57:43 CL today has a static grub config which knows how to pick which to boot 16:58:28 grub has the "grubenv" which is 1024 byes of environment variables that persist across boots 16:58:48 that's probably where we want to store the metadata about boot sucess/failure/etc 17:00:07 imo the static grub config is nice, I'd like to preserve that if we can. All ostree would need to do is drop files in /boot and change the grubenv to say "hey, this is new, try it", then some other service would need to mark boots sucessful 17:00:37 lorbus[m]: if you are around ^^^ 17:00:38 ajeddeloh: would an other service be like greenboot? Or would it be something new need to add? 17:01:41 ashcrow: yeah, I (still) haven't looked into greenboot in depth 17:01:46 that might be an orthogonal decision. greenboot is about how to mark a boot successful 17:02:00 .hello2 17:02:01 bgilbert: bgilbert 'Benjamin Gilbert' 17:02:06 sorry I'm late 17:02:28 i really like the idea of a static grub config, though i don't have much experience with it 17:02:28 but yeah, having something like greenboot would be good so users can have their own definitions of "success" 17:03:08 although if a user is reporting metrics back, we should have another target of "did the system come up at all" as well 17:03:10 #chair bgilbert 17:03:10 Current chairs: ajeddeloh bgilbert bhavin192 davdunc dustymabe geoff- jbrooks jlebon ksinny miabbott mnguyen_ mskarbek rfairley slowrie yzhang 17:03:29 so users that have brittle setups don't cloud the metrics 17:03:36 ajeddeloh: i responded to the "list of goals" in the ticket 17:04:54 I agree with that response 17:05:21 except the last one 17:05:21 i know lorbus[m] has been AFK trying to rest - he should be back soon and I think he'll have some good input here and we might even be able to get him to work on it :) 17:05:59 When doing an upgrade we should keep 3 around until the upgrade is marked successful 17:06:28 ajeddeloh: that sounds nice but can you think of a good reason why ? 17:06:39 the TCP bug comes to mind 17:06:41 it honestly makes the "logic" a lot harder if you have 3 IMHO 17:07:06 basically don't overwrite your fallback 17:07:19 but we would always have the fallback 17:07:33 i.e., being able to upgrade out of a bug without overwriting the non-buggy version 17:07:38 dustymabe: a seemingly good update which turns out to be subtly broken 17:08:11 kaeso: that's an interesting case 17:08:19 I guess it's less of an issue here. CL clobbers the previous version at the beginning of an update, rpm-ostree does it at the end 17:08:27 I'd prefer to not spin up another nginx server that kills itself every 5 minutes 17:08:54 real quick scenario 17:09:20 dustymabe: I think ajeddeloh is concerned if one update is broken, and the next one is slightly-broken, you may end with none good-enough to get out of that 17:09:35 I currently have A, B. A is an old good update, B is the currently running update (which we think is good) 17:09:53 i run upgrade to C which is bad 17:09:59 so now I have B, C 17:10:04 but C fails to boot properly 17:10:07 so we rollback to B 17:10:17 but you're saying B is broken, but we didn't know it was broken ? 17:10:23 the scenario was: 17:10:45 A was good, B was good enough to boot but not good enough to download an update properly. in CL, the act of pushing C would overwrite A 17:11:14 so we couldn't preserve the option to tell users "manually roll back to A" 17:11:24 hmm, in the ostree case though, if B is not good enough to update, it wouldn't get to the point of garbage collecting A 17:11:37 ahh ok bgilbert said this higher up 17:11:56 so I'm less concerned with this particular scenario here 17:11:58 as long as we don't delete A until C is ready to boot, we're fine 17:12:39 it sounds like ostree already handles this correctly 17:12:41 ok. ajeddeloh can you make sure when we get closer to a finished Fedora 30, this concern is addressed ? 17:12:53 +1 17:13:17 I don't want to leave it behind, because I think it's valid. we do have the option of keeping 3 deployments (the functionality already exists) 17:13:31 but I know in greenboot we simplified our logic quite a bit by assuming only two 17:13:39 https://pagure.io/fedora-iot/issue/12#comment-523560 17:14:26 ajeddeloh: if you don't mind can you capture relevant parts of this conversation for the ticket ? 17:14:45 yup 17:14:58 ok next topic is one left over from last week 17:15:17 #topic Allow binaries written in Python via a "platform python" style approach 17:15:21 #link https://github.com/coreos/fedora-coreos-tracker/issues/32 17:16:12 as I said earlier I added some rough data of python dep security issues in the ticket: https://github.com/coreos/fedora-coreos-tracker/issues/32#issuecomment-422867659 17:16:53 i just took what is currently in the image spit out by https://github.com/coreos/fedora-coreos-config repo 17:17:17 once we try to weed out our python deps that list should be shorter, but it's a good starting point 17:17:34 dustymabe: nice investigation 17:17:53 indeed 17:17:55 looking at the dep list, what is authconfig? 17:18:22 and atomic-registries 17:18:40 I know I am biased, but there seems to be general consensus that we should not ship python interpreters and that we are implicitly moving into the "how" part 17:18:40 it's a way to customize the PAM stack using configs 17:18:51 though actually it's been obsoleted by authselect 17:19:03 nfs-tools and xfs-progs are actually optional dependencies. We ship those on CL without python 17:19:41 atomic-registries i think is used by podman for parsing out registry configs 17:20:13 in /etc/containers/registries.conf 17:20:14 the selinux tools you can do without python, but its more awkward iirc. Plus you're probably not going to be messing with selinux much as a user 17:21:03 kaeso: i think it's we'd like to not ship python, unless we want to for X,Y,Z tools that would make our life easier. There is a list of reasons why we don't want to ship python but we've analyzed each one to see which ones can be achieved by just shipping a "system (not for users at all) python" 17:21:16 that list is here: https://github.com/coreos/fedora-coreos-tracker/issues/32#issuecomment-420808195 17:21:27 did we address ansible already? how do we think about config mgmt in FCOS? purely through ignition? 17:21:56 jlebon: we have not considered ansible up until this point.. I think the general thoughts so far is that's not something we are explicitly supporting 17:22:01 imo it should be purely through ignition. otherwise we leave the immutable host realm 17:22:16 jlebon: it's quite uncomfortable, but you can bring your own python for ansible/salt 17:22:17 and that comes with a lot of trade-offs for upgradability, maintenence, etc. 17:22:52 jlebon: you'll likely having a mismatch/miss on some library with the OS one anyway 17:23:39 so "system python" in this case would not be to support ansible IMHO 17:23:49 dustymabe: correct. 17:23:54 it would be only to support things that we've chosen we need (i.e. firewalld if we were to choose that) 17:24:09 and if we remove all of the things we need (like firewalld) then we could just remove system python then 17:24:19 i.e. if firewalld were rewritten 17:24:27 or we chose to use another (new) tool 17:25:09 has anyone investigated running firewalld in a conatiner? 17:25:10 I noticed in RHEL7 that several python packages were providing both pylibs and binaries in PATH 17:25:11 kaeso: ^^ does that help explain it. I think you missed the meeting last week 17:25:39 dustymabe: yes, it's clear, I was speaking about the ansible/salt scenario 17:26:03 i think purely through ignition works, though it'll be a learning curve for folks coming from FAH 17:27:05 ajeddeloh: i'm sure we could run firewalld in a container, but my experience with things like that is it's not maintained properly (i.e. the rpm maintainer doesn't really consider the container that needs to get built) and it also doesn't get maintained properly on the host 17:27:27 i.e. we upgrade the host but the container is forgotten about somehow 17:28:06 * dustymabe notices we have only a few minutes left 17:29:00 does anyone have any thoughts on the frequency of security updates that were found in the analysis 17:29:26 so what do we do here: we start by removing all of those packaged and slowly add back split ones, or do we split/remove one by one? 17:29:56 given that we hope to decrease those dependencies and the frequency listed in the analysis, could we draw any conclusions about #4 17:30:17 dustymabe: the CVE rate is not thrilling but I think we can probably live with it? 17:30:20 dustymabe: that's somehow a false indicator, usually "less CVEs" just means "nobody had a reason to actively investigate/report CVEs" 17:30:38 also not every CVE needs an out-of-cycle release 17:31:00 kaeso: yeah, but if they don't report it how would we ever know we needed an out-of-cycle release? 17:31:50 ok quickly i'll make a proposal 17:31:51 dustymabe: I see, my comment is unrelated to your question then, sorry 17:33:08 #proposed we'd really really like to remove python, but if we choose that we want to keep a tool or a few tools in fedora coreos that use python then we think the system python approach given that it solves #1 and #3, which are our primary concerns, would be reasonable. 17:33:37 feel free to nack that 17:33:42 and we'll revisit next week 17:34:23 actually I probably should have started with this 17:34:37 #undo 17:34:37 Removing item from minutes: 17:34:51 #redo 17:34:55 foo 17:34:56 * kaeso grabs a pointer 17:35:21 #proposed we think security issues for python (#4) aren't significant enough to cause us great hardships during releases 17:36:00 #link https://github.com/coreos/fedora-coreos-tracker/issues/32 17:36:22 +0.5 17:36:30 bgilbert: suggested edit? 17:36:55 wording is fine. I just haven't carefully gone through the CVE list 17:37:10 so, weak agreement 17:37:11 cool in that case we'll wait til next time and retry again with the above two 17:37:31 sorry i ran us over and no open floor today 17:37:33 #topic open floor 17:37:50 did someone have something they wanted to say that won't take much time 17:38:37 concern: coreos-assembler is already ~400 LoC and shell and I'd like for it not to grown into the ~5000 of `coreos/scripts` 17:38:53 s/and shell/of sh/ 17:39:23 but shell has so many guns you can shoot yourself with! 17:39:26 yeah I think we'd like to rewrite it at some point in a nicer language.. there was an open issue and we discussed it and I think we settled on python 17:39:53 well, we settled on "not Rust", not sure if we hard settled on Python? 17:40:02 walters: ^ 17:40:05 ack, I missed that ticket then 17:40:21 https://github.com/coreos/coreos-assembler/pull/84 17:40:31 for some reason I thought that was an "issue" and not a PR 17:40:35 but I found it ^^ 17:40:47 I think shell is OK. The problem I saw with CL scripts is that it got so big it was hard to refactor. 17:41:11 kaeso: maybe weigh in on that PR 17:41:17 yup 17:41:21 or start a new issue somewhere for us to discuss 17:41:32 ok all closing it out.. sorry 10 minutes over 17:41:36 #endmeeting