16:29:39 #startmeeting fedora_coreos_meeting 16:29:39 Meeting started Wed Sep 12 16:29:39 2018 UTC. 16:29:39 This meeting is logged and archived in a public location. 16:29:39 The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:29:39 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:29:39 The meeting name has been set to 'fedora_coreos_meeting' 16:29:42 #topic roll call 16:29:45 .hello2 16:29:46 slowrie: slowrie 'Stephen Lowrie' 16:29:53 .hello mnguyen 16:29:54 mnguyen_: mnguyen 'Michael Nguyen' 16:29:55 .hello2 16:29:57 ajeddeloh: ajeddeloh 'Andrew Jeddeloh' 16:30:27 .hello2 16:30:29 yzhang: yzhang 'Yu Qi Zhang' 16:30:35 .hello rfairleyredhat 16:30:36 rfairley: rfairleyredhat 'Robert Fairley' 16:31:04 .hello 16:31:04 jlebon: (hello ) -- Alias for "hellomynameis $1". 16:31:05 .hello lorbus 16:31:07 lorbus_: lorbus 'Christian Glombek' 16:31:09 .hello jlebon 16:31:10 jlebon: jlebon 'None' 16:31:35 .hello sayanchowdhury 16:31:35 sayan: sayanchowdhury 'Sayan Chowdhury' 16:31:37 .hello sinnykumari 16:31:38 ksinny: sinnykumari 'Sinny Kumari' 16:31:40 geoff-: Geoff Levand 16:31:54 * lorbus_ isn`t sure why his nick is temporarily unavailable?! 16:32:23 .hello smilner 16:32:24 ashcrow: smilner 'None' 16:32:25 .hello2 16:32:26 sanja: sanja 'Sanja Bonic' 16:32:34 .hello miabbott 16:32:35 miabbott: miabbott 'Micah Abbott' 16:33:08 #chair slowrie mnguyen_ ajeddeloh yzhang rfairley jlebon lorbus_ sayan ksinny geoff- ashcrow sanja miabbott 16:33:08 Current chairs: ajeddeloh ashcrow dustymabe geoff- jlebon ksinny lorbus_ miabbott mnguyen_ rfairley sanja sayan slowrie yzhang 16:33:22 do we have walters or bgilbert around today? 16:34:12 .fas jasonbrooks 16:34:13 jbrooks: jasonbrooks 'Jason Brooks' 16:34:22 will move into previous meeting action items 16:34:42 * cverna wages 16:34:56 #topic Action items from last meeting 16:35:02 #chair cverna jbrooks 16:35:02 Current chairs: ajeddeloh ashcrow cverna dustymabe geoff- jbrooks jlebon ksinny lorbus_ miabbott mnguyen_ rfairley sanja sayan slowrie yzhang 16:35:12 * ajeddeloh to tag closed FCOS issues into design-doc PRs 16:35:37 ajeddeloh: that was from two weeks ago.. not sure if we should consider it done or not 16:35:52 I think so, the only two that made sense to include are now merged 16:36:45 It'll kinda become an ongoing thing, but that was specifically talking about the backlog 16:36:47 cool 16:37:10 #info ajeddeloh closed FCOS issues into design-doc PRs 16:37:13 thanks 16:37:39 #topic created a bunch of new issues in FCOS tracker 16:37:51 #link https://github.com/coreos/fedora-coreos-tracker/issues 16:38:27 I created a bunch of new issues related to either work we need to do or discussions we need to have in order to hammer down our strategy for many different parts of Fedora CoreOS 16:39:23 I also added some priority labels `high` `medium` etc.. so we can try to prioritize ones we think are most important and/or ones that we think will take a long time to complete 16:39:54 any questions about that before we move into meeting tickets ? 16:40:24 looks good 16:40:46 lgtm 16:41:07 .hello2 16:41:08 bhavin192: bhavin192 'Bhavin Gandhi' 16:41:24 k. some background, we have a lot of meeting tickets today so we may not get to them all 16:41:32 https://github.com/coreos/fedora-coreos-tracker/issues?q=is%3Aissue+is%3Aopen+label%3Ameeting 16:41:39 #chair bhavin192 16:41:39 Current chairs: ajeddeloh ashcrow bhavin192 cverna dustymabe geoff- jbrooks jlebon ksinny lorbus_ miabbott mnguyen_ rfairley sanja sayan slowrie yzhang 16:41:49 #topic ostree delivery model 16:41:59 #link https://github.com/coreos/fedora-coreos-tracker/issues/23 16:43:06 There is some discussion in the ticket about this, but basically we have a choice on how we choose to deliver our OSTree payloads to clients 16:44:00 one is by using the OSTree repository (like Git repo) model using an HTTPS server (this is what Atomic Host, Silverblue and Fedora IoT use today) 16:44:14 I'm in favor of the vanilla https payloads. KISS. https servers are pretty well figured out by now. 16:44:16 the other is a new delivery model using RPMs 16:44:45 which would allow us to leverage our existing Yum repo mirror network, but would require some integration with build tools 16:44:58 using rpm's feels like we're shoehorning it into that because we have them, not because they're the best choice 16:45:14 https servers can also be done with s3, gce cloud storage, etc 16:45:48 I don't disagree, thanks for the input ajeddeloh 16:46:00 anyone else have any thoughts on this topic ? 16:46:07 ajeddeloh: One thing to think about though is that it's not easy to teach different delivery systems about ostree, but many already understand rpms. But it I don't disagree really. 16:46:34 From a different perspective, if I saw an RPM I'd wonder if it's just the fs tree that's being distributed or if there's other "rpm magic" going on 16:46:39 It's shoehorning it in not just because we have them, but because of what others have -- like the mirror infra, or infra w/in orgs to deal w/ rpms 16:46:53 With that said, I'm fine w/ the current status quo 16:47:03 ashcrow: what do you mean by "different delivery systems" 16:47:38 if we can reliably mirror the ostree content, i'm in favor of the https/KISS approach. for example, even though we are fronting the ostree bits for AH/Silverblue via CloudFront, it still seems slow 16:47:44 the alternatives mentioned in the ticket are: ostree, rojig, container; only that last one is not httpd friendly 16:47:48 one pain point I should also mention is that when someone package layers something now there can be a mismatch between what is in the yum repo and what is in the base tree on the system that causes some conflicts, I think colin's idea was that rojig would solve this somehow, but not sure on that 16:48:20 ajeddeloh: similar to what jbrooks noted. As an example, if someone wants to mirror ostree content that isn't easy to do with many open or closed products. However, many udnerstand how to mirror rpm content. True, rsync is always an option, but usually people are looking for something smarter. It's not a requirement though. 16:48:45 walters: ^^ 16:49:14 Is there a reason we can't do both? 16:49:49 to know how big of a problem this is (for FCOS) we should ask ourselves how often we think people are going to try to store our ostree content locally and redistribute vs just pulling from our mirrors 16:49:49 i.e. start with the https KISS method and later add support for rpm if we determine we want it 16:50:05 Nothing that is a show stopper. The normal development/maint overhead. 16:50:11 I'm cool with that 16:50:30 ajeddeloh: +1 we can always pivot later if we want to change our strategy 16:50:32 Start with KISS, iterate if/as needed 16:50:47 That'd be good because we can keep the common case simple but still support the other cases if we need to 16:51:00 I should also add that we should pull in Iot/Silverblue because it would be great if we did the same thing for all distributed OSTree content 16:51:25 +1 16:52:50 #proposed The general sentiment is that we should target plain OSTree repo + enhanced mirroring support (not implemented today) and then augment with rojig in the future if needed. We will try to get buy in from IoT/Silverblue teams so that we can have a common strategy for all of Fedora delivered OSTree content. 16:53:13 though note transitioning later on would be much more painful than starting in rojig mode right off the bat if that's where we want to go 16:53:49 jlebon: I agree. but I'm not sure what the benefits are exactly, other than taking advantage of the mirror network 16:54:07 do you have any off the top of your head? 16:54:22 .hello2 16:54:23 bgilbert: bgilbert 'Benjamin Gilbert' 16:54:30 #chair bgilbert 16:54:30 Current chairs: ajeddeloh ashcrow bgilbert bhavin192 cverna dustymabe geoff- jbrooks jlebon ksinny lorbus_ miabbott mnguyen_ rfairley sanja sayan slowrie yzhang 16:54:58 if we chose rojig, wouldn't we also have to change the retention of the RPMs needed for each commit? i.e. to preserve commit history, we would need all those rpm versions 16:54:58 yeah, mirroring is definitely the big one. as miabbott mentioned above, ostree updates are still pretty slow even today 16:55:36 why can we not use for mirroring. It's just a bunch of files served over https? 16:55:45 yep. I think we've been putting off "mirroring efforts" because we weren't sure between that or rojig 16:55:59 ajeddeloh: that's an option, yes 16:56:10 ajeddeloh: it's possible. we use cloudfront now for atomic host/workstation 16:56:11 and amazon has offered us credits for stuff like this 16:56:23 so we just need to implement it 16:56:33 We've been using google cloud storage for CL and it's worked well 16:56:38 and I think once we make this decision work can start on that front and we can prioritize it with fedora infra 16:56:43 s/workstation/silverblue/ :P 16:56:56 the main idea with rojig is that is allows users to reuse that whole ecosystem of tools that know how to handle RPMs 16:57:09 anyone want to ack/nack the #proposed statement ? 16:57:48 ack 16:58:06 ack 16:58:10 +1 16:58:14 ack 16:58:14 ack 16:58:17 ack 16:58:20 ack 16:58:26 \o/ 16:58:43 #agreed The general sentiment is that we should target plain OSTree repo + enhanced mirroring support (not implemented today) and then augment with rojig in the future if needed. We will try to get buy in from IoT/Silverblue teams so that we can have a common strategy for all of Fedora delivered OSTree content. 16:58:56 ack 16:58:58 moving on to next topic 16:59:10 #topic Allow binaries written in Python via a "platform python" style approach 16:59:17 #link https://github.com/coreos/fedora-coreos-tracker/issues/32 16:59:41 I placed this one as high priority because other discussions hinge on it (like the firewall discussion) 17:00:09 I laid out 4 goals related to python in FEdora CoreOS in the ticket 17:00:36 1. prevent users from running scripts directly on the host 17:00:38 2. prevent shipping/maintaining python 17:00:40 3. prevent issues where user's python script needs library X that isn't installed 17:00:42 4. prevent security issues in python requiring a respin 17:01:03 are there any others that I missed? 17:01:37 There's a (tiny) 5th: less space used on disk + less data transmitted for updates 17:01:50 +1 17:02:03 plus an even smaller 6th: better perception we're minimal 17:02:19 :) 17:02:21 (which is something I've seen as a selling point for CL) 17:02:43 +1 17:02:44 the two options being discussed are 17:02:51 - no python at all 17:03:13 - a shipped python that is not accessible to users (an attempted goal) 17:03:40 If we have nothing that _hard_ requires python (i.e. firewalld) then imo its worth breaking up the packages that think they require it but dont 17:03:44 out of the ~6 goals we laid out above, which are the most important and which ones are not addressed by the 2nd option ? 17:03:53 I think "no python at all" is the ideal. But I think that is a large mountain to climb given a specific time frame. 17:04:02 Note: I'm a big fan of python as a developer :-) 17:04:12 haha ashcrow same :) 17:04:27 +1 to no python at all 17:04:28 2,4,5,6 17:04:58 can we run firewalld in a container or systemd PS? 17:05:15 ok let's discuss 2,4,5,6 real quick 17:05:31 I feel like this is becoming a catchphrase but "this seems like a job for systemd portable services" 17:05:43 2. i'm not too worried about because it's already being maintained in fedora 17:05:50 does anyone agree/disagree with that ? 17:06:06 yeah, agreed 17:06:08 I don't disagree. 17:06:13 agreed 17:06:41 I do, to some extent 17:06:45 +1 17:06:53 do disagree, sorry 17:07:01 bgilbert: :) I was going t oask 17:07:03 we still have to be concerned about the package set 17:07:15 bgilbert: i.e. security issues ? 17:07:43 most of the work is done elsewhere, yes, but there's always a development cost to carrying things 17:08:17 bgilbert: is that concern that the rest of fedora would drop it and we'd be maintaining it ourselves? 17:08:43 I think it's more so we have to keep an eye on new deps and what changes. Sizes may go up and down into what is pulled in or dropped by upstream. 17:08:50 bgilbert: you're talking about the cost of splitting packages? 17:09:18 not really. but for any package we carry, there's always the possibility of having to dig into it to root-cause an issue. 17:09:28 * ashcrow nods 17:09:33 because we tickle some case that upstream Fedora doesn't 17:09:48 so, the fewer large things we carry, the better. /shrug 17:10:00 +1 I agree with that 17:10:22 so caveat there that just because the work is being done somewhere else, doesn't mean it won't ever cause us pain 17:10:56 i.e. firewalld (example) doesn't work in FCOS because of the way we have the system set up and it's a bug in python-xyz package that doesn't show up on vanilla Fedora 17:11:11 bgilbert: ^^ accurate ? 17:11:45 yep 17:11:54 ok so I think we are done with 2. 17:12:02 .hello2 17:12:03 walters: walters 'Colin Walters' 17:12:06 4. prevent security issues in python requiring a respin 17:12:15 this is a legitimate concern 17:12:57 hopefully our tooling/processes allow us to respin things easily, but there is always costs associated and disruptions as a result 17:13:23 if we ship python I don't think we'll get away from #4, but do wonder how often python itself has security issues 17:13:27 anyone know ^^ 17:13:32 just as a rough measure, I was scanning https://bodhi.fedoraproject.org/updates/?packages=python3 to see how many updates are marked as security 17:13:46 there was one 5 months ago, another 9 months ago 17:14:32 jlebon: and depending on the severity of those two, maybe wouldn't have even caused us to respin 17:14:34 the next one after that was in f23 17:14:57 dustymabe: true 17:15:09 well it's not just python we have to worry about. If firewalld depends on X Y and Z python packages we need to worry about them as well 17:15:09 that's probably not the full picture, though. as related libraries could have had security issues and needed updating 17:15:17 so we'd probably need to do a little more investigation 17:15:23 though OTOH, this is *only* the main python3 package. including python3 means it's easier to include other packages which in turn can have sec erratas 17:15:39 unlike the kernel, who do not have CVE per dozen every month :p 17:16:14 i think if we do go the "system python" route we still want to be very careful about anything that we include that is python based, we only include what we really really want OR need 17:17:14 @bgilbert should we try to get more information about security issues in python3 and related libraries to get more info for this point? 17:17:40 wouldn't hurt. 17:17:51 ok I can try to do that or ask someone else to 17:17:58 there are certainly other packages that we have to respin for much more often 17:18:20 #action dustymabe to gather more information on frequency of python (and related library) security issues in Fedora 17:18:38 ok points 5/6 (which I'll group together as smaller/minimal footprint) 17:18:49 shipping python will make us less minimal, yes 17:19:34 alternatively though, rewriting the few things that we do need in python in Go (large binary) might not actually save that much disk space ?? 17:19:40 Somewhat OT: (I don't think it should change the discussion) IF using ansible to do things is planned you will have to have quite a few libraries added in to system python and have instructions for using ansible with the distribution. 17:19:51 dustymabe: true. Until it's looked at there is no way to really know. 17:20:26 afaik we're not supporting ansible, right? 17:20:35 ashcrow: maybe naive of me but I don't think we should consider ansible at this time (or maybe force that into a containerized route) 17:20:40 (at least on host) 17:20:40 +1 17:20:57 ajeddeloh: agreed, mostly 17:21:33 well, not only that, I wonder if there is some selinux tooling in python too :/ 17:21:34 ajeddeloh: FWIW ansible needs the libs on the host when run from an external system. But +1 that we punt on that. 17:21:40 so after discussion points 1-6 - do we have a feel for the ones that are most important to us? 17:21:45 misc: there definitely is 17:22:16 For me 1 is the most important :-) But I'll defer to others. 17:22:33 the way I see it is 1,3 are most important to us 17:23:02 iirc (and very much may be wrong on this) the selinux python tools are higher level and you can get by with the ones not in python, but it's been a while since I looked at that 17:23:02 2 I think we can rely on the rest of Fedora for that, but note bgilbert's caveat that says there won't be zero pain 17:23:42 4 we have an action item to gather more information, but it doesn't look like there have been my primary python package security issues in the past year 17:24:00 ajeddeloh: well, the experience of debugging selinux issue would be likely more annoying and push users to disable it. That's kinda not something we want 17:24:08 i feel like at some point ansible probably will find it beneficial in general to support a containerized mode 17:24:24 5/6 - minimal is good, but people who choose not to use Fedora CoreOS specifically because we chose to ship system python probably want more control over their package set anyway and will build their own 17:25:01 dustymabe: I don't think we're going to get a lot of "build it on your own" people 17:25:18 the whole point of a *coreos is that you don't need to do anything for updates 17:25:28 ajeddeloh: so if they choose "not FCOS" because of this, you think they'd choose another offering ? 17:26:01 yeah, you'd have to run your own build infra for that 17:26:12 right 17:26:33 with coreos-assembler that will be easier :), but I agree, it would be work they'd have to do 17:26:46 ok I'll try to take all of these points and put them in the ticket 17:27:02 thats not something I expect many people to do, and they'd either have to automate looking for our updates or manually kick off builds 17:27:07 after I gather more information maybe we can get together next week and agree on a strategy 17:27:31 ok, look at the time 17:27:31 probably worth noting that the inaccessible-Python approach means that we don't need to make a permanent decision up front. 17:27:52 we can try it, explicitly document that users shouldn't use it, and revisit down the road. 17:27:58 bgilbert: I disagree 17:28:00 yes, the decision here is between "no python" and "inaccessible-Python" 17:28:13 that python is in service of _something_ that is user facing 17:28:25 ajeddeloh: concerned about compat constraints? 17:28:26 presumably removing it means removing the user facing thing 17:28:34 yeah 17:28:41 fair 17:29:04 ok i'll open up for open floor now 17:29:06 * bgilbert creates github/coreos/coreos-firewalld 17:29:12 #topic open floor 17:29:32 this is a topic i wanted to get to because I think it will be short but I'll just ask people to weigh in on the ticket so we can close it out: 17:29:49 Default filesystem choices for FCOS https://github.com/coreos/fedora-coreos-tracker/issues/33 17:30:07 basically it seems like we are leaning toward XFS ^^ please weigh in in the ticket with +1 or -1 17:30:16 anyone else with anything for open floor ? 17:30:27 i'm not sure if this was brought up in a previous meeting, though https://github.com/coreos/fedora-coreos-config is worth a look if you haven't already 17:30:40 along with https://github.com/coreos/coreos-assembler 17:30:50 jlebon: +1 17:31:01 i guess one could call it "proto-proto-FCOS" 17:31:03 re fs choice: can I +0? I don't care much tbh 17:31:20 there was also a long/detailed question written on the discourse last night that I tried to answer with a lot of good information 17:31:27 ajeddeloh: sure that's still good information to have 17:31:42 re: discourse see https://discussion.fedoraproject.org/t/getting-involved/386/2 17:32:05 anyone with anything else for open floor? 17:32:50 Nothing from me 17:33:37 ok closing out 17:33:39 #endmeeting