16:30:42 #startmeeting fedora_coreos_meeting 16:30:42 Meeting started Wed Apr 8 16:30:42 2020 UTC. 16:30:42 This meeting is logged and archived in a public location. 16:30:42 The chair is jlebon. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:30:42 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:30:42 The meeting name has been set to 'fedora_coreos_meeting' 16:30:49 #topic roll call 16:31:57 .hello2 16:31:57 .hello lucab 16:31:58 jlebon: jlebon 'None' 16:32:01 kaeso[m]: lucab 'Luca Bruno' 16:32:04 .hello2 16:32:05 dustymabe: dustymabe 'Dusty Mabe' 16:32:16 .hello sohank2602 16:32:17 skunkerk: sohank2602 'Sohan Kunkerkar' 16:32:28 #chair kaeso[m] dustymabe skunkerk 16:32:28 Current chairs: dustymabe jlebon kaeso[m] skunkerk 16:32:30 .hello mnguyen 16:32:31 mnguyen_: mnguyen 'Michael Nguyen' 16:32:36 #chair mnguyen_ 16:32:36 Current chairs: dustymabe jlebon kaeso[m] mnguyen_ skunkerk 16:33:32 .hello2 16:33:33 bgilbert: bgilbert 'Benjamin Gilbert' 16:33:37 #chair bgilbert 16:33:37 Current chairs: bgilbert dustymabe jlebon kaeso[m] mnguyen_ skunkerk 16:33:44 gonna get started in 30s 16:34:16 * dustymabe reminds anyone that wants to bring up a topic to add the meeting label to it https://github.com/coreos/fedora-coreos-tracker/issues 16:34:21 #topic Action items from last meeting 16:34:30 ENOTFOUND 16:34:43 yay for no action items 16:34:43 alrighty then, moving on! :) 16:35:10 hmm, no issues with meeting label either 16:35:43 if that's the case, i'm going to bring up one thing 16:36:34 so yesterday, i pushed f32-based FCOS into the next-devel branch 16:36:52 jlebon: maybe add #topic blurb 16:37:14 #topic revisiting FCOS streams 16:37:49 this caused a whole host of issues. but i'd like to get into one in particular 16:37:59 https://github.com/coreos/fedora-coreos-config/pull/335 16:38:30 essentially, the bodhi-updates has been the stream that tracks FCOS based on the latest Fedora stable+updates packages 16:38:38 so it by definition is unlocked 16:39:15 however, for $reasons, we were including the coreos-pool repo as a source for bodhi-updates itself 16:39:45 this was always meant to be temporary because it muddles the intent of that stream 16:40:13 * lorbus arrives late 16:40:19 now with f32 packages in the pool, we have to sever that link 16:40:31 .hello2 16:40:31 otherwise bodhi-updates would get f32 packages 16:40:31 lorbus: lorbus 'Christian Glombek' 16:40:36 #chair lorbus 16:40:36 Current chairs: bgilbert dustymabe jlebon kaeso[m] lorbus mnguyen_ skunkerk 16:41:20 jlebon: yep. it's good to sever the coreos-pool link in bodhi updates I think 16:41:36 and now comes the issue: bodhi-updates can't compose currently with purely f31 packages in stable+updates 16:42:18 the reason is that we've had an override in testing-devel for a long time to bring in crypto-policies from f32, so that we can drop python out 16:42:32 and now we have a hard check in rpm-ostree to make sure it doesn't come back in 16:42:57 I added a comment with a suggestion: use urls to rpms directly that we want to fast track into bodhi-updates 16:43:17 hopefully overriding things in bodhi should be the exception 16:43:19 i think the issue is higher-level than that though 16:44:15 either testing-devel overrides are truly meant to be temporary "diffs" from bodhi-updates, in which case we should probably have backported the python-less patch to f31 instead of holding it until f32 16:45:04 OR bodhi-updates isn't actually providing much value by allowing it to either be broken for a long time, or not truly represent pure f31 stable+updates 16:45:52 i think the proper fix if we want to maintain the current streams would've been to backport 16:46:07 but i'm wondering if we should just nuke bodhi-updates entirely 16:46:46 IOW we don't feed testing-devel from bodhi updates but from some other thing? 16:46:53 the difference between testing-devel and bodhi-updates is nuanced, and we have enough streams to care for already 16:47:09 dustymabe: we definitely have to break that relationship anyway 16:47:25 that's https://github.com/coreos/fedora-coreos-tracker/issues/293 16:47:32 which we'll also need for next-devel 16:48:10 thoughts? 16:48:24 ok, yeah I think maybe let's get https://github.com/coreos/fedora-coreos-tracker/issues/293 in place and then we can re-evaluate the usefulness of the bodhi-updates stream 16:48:49 maybe we keep it but in some limited form (i.e. only building qemu and running qemu kola tests) 16:48:51 IOW, just leave it broken for now and revisit later? 16:49:30 no, let's try to figure out a way to fix the immediate problem for now 16:49:35 from my perspective, the "FCOS - FCOS" nature of bodhi-updates is a bit odd 16:49:55 bgilbert: I'm not sure I follow 16:49:57 dustymabe: right putting 293 aside here. that has to be fixed 16:50:01 IIRC the original justification was to have something to point users at, to repro certain bugs 16:50:20 FCOS minus FCOS. our build system but not our overrides. 16:50:57 FWIW i think its cousin bodhi-updates-testing is useful. but i'd rebrand it as being about testing-devel + updates-testing 16:51:05 to the extent we've wanted to reduce scope of our efforts, dropping it makes sense to me 16:51:48 jlebon: sure, could do 16:51:54 i'm not 100% opposed to dropping it, but would like to re-evaluate once we have #293 in place 16:51:56 bgilbert: that's my conclusion as well 16:53:01 dustymabe: sure, that WFM. if we fast forward e.g. 3 months and it's still broken and unused, then let's nuke it 16:53:16 so should we focus on the immediate problem? 16:53:44 dustymabe: i didn't want to muddle the discussion, but we can switch topic if you'd like 16:54:36 if there's more to discuss on #293 + bodhi updates then let's do it 16:54:41 once 293 is fixed, bodhi-updates will no longer be in our critical path 16:54:41 if not let's switch to the immediate problem 16:55:08 sure 16:55:49 (are you talking about https://github.com/coreos/fedora-coreos-tracker/issues/454 ? ) 16:55:54 * dustymabe wished excludepkgs/includepkgs worked 16:55:57 https://github.com/coreos/rpm-ostree/issues/1447 16:56:45 jlebon: we should be able to keep bodhi-updates stream where it's dropped coreos-pool right 16:56:52 dustymabe: let me clarify, i don't think we should work towards fixing bodhi-updates. i think we should just do 293 16:57:08 jlebon: how long do you think #293 would take? 16:57:44 i was assuming it wasn't going to be something we get in the next 7 days 16:57:46 i think the basic git push functionality with PRs should be pretty easy 16:57:51 s/with/without/ 16:58:18 but i've underestimated these seemingly simple things before :) 16:58:55 ok so one proposal is we sprint to #293 (goal being next wednesday) which gives us a week before the next set of releases? 16:59:32 and we need it for our next-devel stream anyway 16:59:49 dustymabe: ahh i see, you're suggesting just adding the URL in the lockfile? 17:00:03 well adding the URL in the manifest 17:00:09 hmm yeah, that should work 17:00:13 which is an open PR upstream 17:00:19 but we'd have to do it for all of our overrides 17:00:29 not just the crypto-policies one 17:00:35 which is OK in the short term 17:00:37 ahhh right, i thought that was merged already 17:00:46 ok so either way, there's coding to do and things to pipe through 17:01:16 right - if we get that PR merged upstream I can backport it 17:01:41 how about we timebox 293 and if we go over we can do a dirty hack to unblock bodhi-updates? 17:02:00 e.g. just host a side repo on fedorapeople or something with just crypto-policies f32 in it 17:02:02 that's fine. so we're agreed we're going to let bodhi updates keep failing for now 17:02:24 yeah, that's my inclination 17:02:33 +1 17:02:43 who wants to take the action on #293 17:03:03 side note: keeping python out of the base is a PITA 17:03:40 maybe we can discuss allocation for 293 after? 17:03:41 * dustymabe hopes other packages don't pick up new deps :) 17:03:52 jlebon: +1 17:04:11 i feel bad we took up all this time on an obscure releng-related thing that most people don't really care about :) 17:04:32 ok cool, let's move on then! 17:04:49 anyone have anything else to discuss? 17:04:57 i guess this is merging into open floor now 17:05:08 +1 for open floor 17:05:11 #topic Open Floor 17:06:08 #info image uploading to GCP has made a lot of progress - the next releases should be available once we get everything wired up and released 17:06:28 I'm guessing we'll need some websites changes for that ^^ 17:06:48 nice! 17:06:54 dustymabe++ 17:07:08 when you say next release you mean the one we're doing right now, right? 17:07:48 jlebon: assuming we decide to actually "release" it 17:07:50 :) 17:08:07 dustymabe: ahh right, that's another fun topic heh 17:08:26 also there is some interesting finds from our `next` stream exploration 17:08:29 https://github.com/coreos/fedora-coreos-tracker/issues/452 17:09:05 we need to figure out how to not pull in that 33M of stuff 17:09:48 that's all from me 17:10:41 hmm, yeah we don't explicitly have samba in the manifest 17:10:57 we do have nfs though 17:10:59 but do note we have `setsebool -P -N virt_use_samba on # RHBZ#1754825` 17:11:21 but we don't have e.g. ceph or gluster 17:11:37 but samba is more akin to nfs than those 17:11:42 I'm somehow not concerned by having `libicudata` in 17:11:52 looks like i'm just arguing with myself now :) 17:12:31 because it will eventually be dragged it by anything that wants to do proper unicode handling 17:13:22 *dragged in 17:13:50 yeah. I'd prefer to not have it if we can help, but if it must happen it must happne 17:13:59 i'd rather spend those 33M on something more useful :) 17:14:07 i guess that's what we need to drill into. it didn't use to get pulled in, but it does now. and presumably we've had unicode problems for a while :) 17:14:32 jlebon: so you think this will fix other problems? 17:14:50 dustymabe: i just mean unicode handling isn't a new problem 17:15:09 kk 17:15:28 I'm somehow surprised this is not already pulled in by any regexp library 17:15:41 but yeah, if it's not an easy dep to shed anyway and we see more apps depending on it, then... probably not worth fighting 17:15:57 *cough* python *cough* 17:16:19 I kid 17:16:24 dustymabe: oh does libicu have a full-fledged REPL? :) 17:16:40 that was bad sarcasm, sorry 17:16:57 haha i'm just saying your argument there applies appropriately to python 17:17:01 jlebon: for interactive browsing of unicode tables. That would be great! 17:17:16 dustymabe: but you can't write apps on top of it 17:17:36 jlebon: that's what the "system python" proposal was hoping to solve 17:18:27 right yeah. i just mean the motives between the two are not the same :) 17:18:30 in fedora there is a new python minimization effort too 17:18:44 so we should probably follow that 17:18:51 am I allowed to circle back to the "are we releasing today?" topic? 17:19:04 yup, sure 17:19:54 dustymabe, kaeso[m], and I have been chasing a regression in update testing on aws: https://github.com/coreos/coreos-assembler/issues/1301 17:20:31 we're making progress and think it might be related to an ostree/rpm-ostree bump, but more investigation is needed 17:20:34 and we somehow expected this to show up in the stable release we are doing today 17:20:51 which indeed happened, https://github.com/coreos/fedora-coreos-streams/issues/79#issuecomment-611068739 17:21:36 my vote is to hold the release for now until we get to the bottom of it 17:22:00 is this my understanding we already ignored it in a previous testing release, right? 17:22:13 s/is this/it is/ 17:22:20 correct, in a previous `testing` release 17:22:26 but not for stable 17:22:50 and my testing canary on AWS already updated once past that 17:22:51 yes. we didn't dig into it at the time, and it seemed more like a flake than a consistent failure 17:23:04 and nobody else reported any update failures 17:23:07 right. the flake vs consistent failure is important 17:23:29 kaeso[m]: not that I know of, but I realize a lot less people are probably running testing 17:24:06 ack 17:24:12 i definitely remember rerunning kola aws tests which used to fail and then passed. so there was some flakiness at some point in time 17:24:45 what do we want to do at this point? 17:24:49 I think we should definitely do some more investigation before we decide which way we want to go with the currently built release candidates 17:24:55 keep investigating :) 17:25:13 kaeso[m]: it's late in your day so we can take it 17:25:29 i.e. pausing both releases for now? 17:25:36 yes 17:25:40 IMHO 17:25:57 yup, agreed. i think we're close to getting to the bottom of it 17:26:10 my testing canary upgraded fine last week, but it's got 8 cores and 16G memory 17:26:30 ok 17:27:13 anything else for open floor? 17:27:17 but I have a feeling current testing may already have the bug 17:27:51 ehm, even the last two testing releases actually 17:28:01 kaeso[m]: correct 17:28:15 breaking testing is not nearly as bad as breaking stable though 17:28:32 jlebon: I think we're good on open floor 17:28:35 yeah... we'll see. maybe there's an easy manual workaround for those. or maybe it only manifests in our tests 17:28:51 yes, I was wondering whether to finish testing anyway and just pause stable 17:28:51 ok, will close in 45s 17:29:05 but let's wait tomorrow I guess 17:29:10 ! 17:29:16 Hello you guys 17:29:23 Sorry, I'm here again to bother you 17:29:29 x3mboy: hi! :) 17:30:38 Hi! 17:30:54 I have this issue: https://pagure.io/design/issue/642 17:31:25 And I will be more than happy to have it finished to launch those infographics this month 17:32:05 dustymabe: looks like you were tagged in there 17:32:21 x3mboy: i'm time strapped - your best bet is to schedule a video meeting with me for 30m or 1 hour where we can extemporaneously brain storm on the issue 17:32:32 if we don't do that I'll never get to it 17:33:23 Let me know when and I will send you an invitation 17:33:44 Or send me and invite and make this happens 17:33:49 x3mboy: pm me and we can coordinate 17:33:53 Ok 17:33:55 Thanks 17:33:57 eom 17:33:57 and thanks for being persistent 17:34:07 Well, it's part of my job 17:34:08 :D 17:34:40 thanks x3mboy! 17:34:51 alrighty, closing in 20s 17:35:18 #endmeeting