16:30:42 <jlebon> #startmeeting fedora_coreos_meeting
16:30:42 <zodbot> Meeting started Wed Apr  8 16:30:42 2020 UTC.
16:30:42 <zodbot> This meeting is logged and archived in a public location.
16:30:42 <zodbot> The chair is jlebon. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:42 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:30:42 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:30:49 <jlebon> #topic roll call
16:31:57 <jlebon> .hello2
16:31:57 <kaeso[m]> .hello lucab
16:31:58 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:01 <zodbot> kaeso[m]: lucab 'Luca Bruno' <lucab@redhat.com>
16:32:04 <dustymabe> .hello2
16:32:05 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:32:16 <skunkerk> .hello sohank2602
16:32:17 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:32:28 <jlebon> #chair kaeso[m] dustymabe skunkerk
16:32:28 <zodbot> Current chairs: dustymabe jlebon kaeso[m] skunkerk
16:32:30 <mnguyen_> .hello mnguyen
16:32:31 <zodbot> mnguyen_: mnguyen 'Michael Nguyen' <mnguyen@redhat.com>
16:32:36 <jlebon> #chair mnguyen_
16:32:36 <zodbot> Current chairs: dustymabe jlebon kaeso[m] mnguyen_ skunkerk
16:33:32 <bgilbert> .hello2
16:33:33 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:33:37 <jlebon> #chair bgilbert
16:33:37 <zodbot> Current chairs: bgilbert dustymabe jlebon kaeso[m] mnguyen_ skunkerk
16:33:44 <jlebon> gonna get started in 30s
16:34:16 * dustymabe reminds anyone that wants to bring up a topic to add the meeting label to it https://github.com/coreos/fedora-coreos-tracker/issues
16:34:21 <jlebon> #topic Action items from last meeting
16:34:30 <jlebon> ENOTFOUND
16:34:43 <dustymabe> yay for no action items
16:34:43 <jlebon> alrighty then,  moving on! :)
16:35:10 <jlebon> hmm, no issues with meeting label either
16:35:43 <jlebon> if that's the case, i'm going to bring up one thing
16:36:34 <jlebon> so yesterday, i pushed f32-based FCOS into the next-devel branch
16:36:52 <dustymabe> jlebon: maybe add #topic blurb
16:37:14 <jlebon> #topic revisiting FCOS streams
16:37:49 <jlebon> this caused a whole host of issues.  but i'd like to get into one in particular
16:37:59 <jlebon> https://github.com/coreos/fedora-coreos-config/pull/335
16:38:30 <jlebon> essentially, the bodhi-updates has been the stream that tracks FCOS based on the latest Fedora stable+updates packages
16:38:38 <jlebon> so it by definition is unlocked
16:39:15 <jlebon> however, for $reasons, we were including the coreos-pool repo as a source for bodhi-updates itself
16:39:45 <jlebon> this was always meant to be temporary because it muddles the intent of that stream
16:40:13 * lorbus arrives late
16:40:19 <jlebon> now with f32 packages in the pool, we have to sever that link
16:40:31 <lorbus> .hello2
16:40:31 <jlebon> otherwise bodhi-updates would get f32 packages
16:40:31 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:40:36 <jlebon> #chair lorbus
16:40:36 <zodbot> Current chairs: bgilbert dustymabe jlebon kaeso[m] lorbus mnguyen_ skunkerk
16:41:20 <dustymabe> jlebon: yep. it's good to sever the coreos-pool link in bodhi updates I think
16:41:36 <jlebon> and now comes the issue: bodhi-updates can't compose currently with purely f31 packages in stable+updates
16:42:18 <jlebon> the reason is that we've had an override in testing-devel for a long time to bring in crypto-policies from f32, so that we can drop python out
16:42:32 <jlebon> and now we have a hard check in rpm-ostree to make sure it doesn't come back in
16:42:57 <dustymabe> I added a comment with a suggestion: use urls to rpms directly that we want to fast track into bodhi-updates
16:43:17 <dustymabe> hopefully overriding things in bodhi should be the exception
16:43:19 <jlebon> i think the issue is higher-level than that though
16:44:15 <jlebon> either testing-devel overrides are truly meant to be temporary "diffs" from bodhi-updates, in which case we should probably have backported the python-less patch to f31 instead of holding it until f32
16:45:04 <jlebon> OR bodhi-updates isn't actually providing much value by allowing it to either be broken for a long time, or not truly represent pure f31 stable+updates
16:45:52 <jlebon> i think the proper fix if we want to maintain the current streams would've been to backport
16:46:07 <jlebon> but i'm wondering if we should just nuke bodhi-updates entirely
16:46:46 <dustymabe> IOW we don't feed testing-devel from bodhi updates but from some other thing?
16:46:53 <jlebon> the difference between testing-devel and bodhi-updates is nuanced, and we have enough streams to care for already
16:47:09 <jlebon> dustymabe: we definitely have to break that relationship anyway
16:47:25 <jlebon> that's https://github.com/coreos/fedora-coreos-tracker/issues/293
16:47:32 <jlebon> which we'll also need for next-devel
16:48:10 <jlebon> thoughts?
16:48:24 <dustymabe> ok, yeah I think maybe let's get https://github.com/coreos/fedora-coreos-tracker/issues/293 in place and then we can re-evaluate the usefulness of the bodhi-updates stream
16:48:49 <dustymabe> maybe we keep it but in some limited form (i.e. only building qemu and running qemu kola tests)
16:48:51 <jlebon> IOW, just leave it broken for now and revisit later?
16:49:30 <dustymabe> no, let's try to figure out a way to fix the immediate problem for now
16:49:35 <bgilbert> from my perspective, the "FCOS - FCOS" nature of bodhi-updates is a bit odd
16:49:55 <dustymabe> bgilbert: I'm not sure I follow
16:49:57 <jlebon> dustymabe: right putting 293 aside here. that has  to be fixed
16:50:01 <bgilbert> IIRC the original justification was to have something to point users at, to repro certain bugs
16:50:20 <bgilbert> FCOS minus FCOS.  our build system but not our overrides.
16:50:57 <jlebon> FWIW i think its cousin bodhi-updates-testing is useful. but i'd rebrand it as being about testing-devel + updates-testing
16:51:05 <bgilbert> to the extent we've wanted to reduce scope of our efforts, dropping it makes sense to me
16:51:48 <bgilbert> jlebon: sure, could do
16:51:54 <dustymabe> i'm not 100% opposed to dropping it, but would like to re-evaluate once we have #293 in place
16:51:56 <jlebon> bgilbert: that's my conclusion as well
16:53:01 <jlebon> dustymabe: sure, that WFM.  if we fast forward e.g. 3 months and it's still broken and unused, then let's nuke it
16:53:16 <dustymabe> so should we focus on the immediate problem?
16:53:44 <jlebon> dustymabe: i didn't want to muddle the discussion, but we can switch topic if you'd like
16:54:36 <dustymabe> if there's more to discuss on #293 + bodhi updates then let's do it
16:54:41 <jlebon> once 293 is fixed, bodhi-updates will no longer be in our critical path
16:54:41 <dustymabe> if not let's switch to the immediate problem
16:55:08 <jlebon> sure
16:55:49 <jlebon> (are you talking about https://github.com/coreos/fedora-coreos-tracker/issues/454 ? )
16:55:54 * dustymabe wished excludepkgs/includepkgs worked
16:55:57 <dustymabe> https://github.com/coreos/rpm-ostree/issues/1447
16:56:45 <dustymabe> jlebon: we should be able to keep bodhi-updates stream where it's dropped coreos-pool right
16:56:52 <jlebon> dustymabe: let me clarify, i don't think we should work towards fixing bodhi-updates. i think we should just do 293
16:57:08 <dustymabe> jlebon: how long do you think #293 would take?
16:57:44 <dustymabe> i was assuming it wasn't going to be something we get in the next 7 days
16:57:46 <jlebon> i think the basic git push functionality with PRs should be pretty easy
16:57:51 <jlebon> s/with/without/
16:58:18 <jlebon> but i've underestimated these seemingly simple things before :)
16:58:55 <dustymabe> ok so one proposal is we sprint to #293 (goal being next wednesday) which gives us a week before the next set of releases?
16:59:32 <dustymabe> and we need it for our next-devel stream anyway
16:59:49 <jlebon> dustymabe: ahh i see, you're suggesting just adding the URL in the lockfile?
17:00:03 <dustymabe> well adding the URL in the manifest
17:00:09 <jlebon> hmm yeah, that should work
17:00:13 <dustymabe> which is an open PR upstream
17:00:19 <dustymabe> but we'd have to do it for all of our overrides
17:00:29 <dustymabe> not just the crypto-policies one
17:00:35 <dustymabe> which is OK in the short term
17:00:37 <jlebon> ahhh right, i thought that was merged already
17:00:46 <jlebon> ok so either way, there's coding to do and things to pipe through
17:01:16 <dustymabe> right - if we get that PR merged upstream I can backport it
17:01:41 <jlebon> how about we timebox 293 and if we go over we can do a dirty hack to unblock bodhi-updates?
17:02:00 <jlebon> e.g. just host a side repo on fedorapeople or something with just crypto-policies f32 in it
17:02:02 <dustymabe> that's fine. so we're agreed we're going to let bodhi updates keep failing for now
17:02:24 <jlebon> yeah, that's my inclination
17:02:33 <dustymabe> +1
17:02:43 <dustymabe> who wants to take the action on #293
17:03:03 <dustymabe> side note: keeping python out of the base is a PITA
17:03:40 <jlebon> maybe we can discuss allocation for 293 after?
17:03:41 * dustymabe hopes other packages don't pick up new deps :)
17:03:52 <dustymabe> jlebon: +1
17:04:11 <jlebon> i feel bad we took up all this time on an obscure releng-related thing that most people don't really care about :)
17:04:32 <jlebon> ok cool, let's move on then!
17:04:49 <jlebon> anyone have anything else to discuss?
17:04:57 <jlebon> i guess this is merging into open floor now
17:05:08 <dustymabe> +1 for open floor
17:05:11 <jlebon> #topic Open Floor
17:06:08 <dustymabe> #info image uploading to GCP has made a lot of progress - the next releases should be available once we get everything wired up and released
17:06:28 <dustymabe> I'm guessing we'll need some websites changes for that ^^
17:06:48 <jlebon> nice!
17:06:54 <jlebon> dustymabe++
17:07:08 <jlebon> when you say next release you mean the one we're doing right now, right?
17:07:48 <dustymabe> jlebon: assuming we decide to actually "release" it
17:07:50 <dustymabe> :)
17:08:07 <jlebon> dustymabe: ahh right, that's another fun topic heh
17:08:26 <dustymabe> also there is some interesting finds from our `next` stream exploration
17:08:29 <dustymabe> https://github.com/coreos/fedora-coreos-tracker/issues/452
17:09:05 <dustymabe> we need to figure out how to not pull in that 33M of stuff
17:09:48 <dustymabe> that's all from me
17:10:41 <jlebon> hmm, yeah we don't explicitly have samba in the manifest
17:10:57 <jlebon> we do have nfs though
17:10:59 <dustymabe> but do note we have `setsebool -P -N virt_use_samba on  # RHBZ#1754825`
17:11:21 <jlebon> but we don't have e.g. ceph or gluster
17:11:37 <jlebon> but samba is more akin to nfs than those
17:11:42 <kaeso[m]> I'm somehow not concerned by having `libicudata` in
17:11:52 <jlebon> looks like i'm just arguing with myself now :)
17:12:31 <kaeso[m]> because it will eventually be dragged it by anything that wants to do proper unicode handling
17:13:22 <kaeso[m]> *dragged in
17:13:50 <dustymabe> yeah. I'd prefer to not have it if we can help, but if it must happen it must happne
17:13:59 <dustymabe> i'd rather spend those 33M on something more useful :)
17:14:07 <jlebon> i guess that's what we need to drill into. it didn't use to get pulled in, but it does now. and presumably we've had unicode problems for a while :)
17:14:32 <dustymabe> jlebon: so you think this will fix other problems?
17:14:50 <jlebon> dustymabe: i just mean unicode handling isn't a new problem
17:15:09 <dustymabe> kk
17:15:28 <kaeso[m]> I'm somehow surprised this is not already pulled in by any regexp library
17:15:41 <jlebon> but yeah, if it's not an easy dep to shed anyway and we see more apps depending on it, then... probably not worth fighting
17:15:57 <dustymabe> *cough* python *cough*
17:16:19 <dustymabe> I kid
17:16:24 <jlebon> dustymabe: oh does libicu have a full-fledged REPL?  :)
17:16:40 <jlebon> that was bad sarcasm, sorry
17:16:57 <dustymabe> haha i'm just saying your argument there applies appropriately to python
17:17:01 <kaeso[m]> jlebon: for interactive browsing of unicode tables. That would be great!
17:17:16 <jlebon> dustymabe: but you can't write apps on top of it
17:17:36 <dustymabe> jlebon: that's what the "system python" proposal was hoping to solve
17:18:27 <jlebon> right yeah.  i just mean the motives between the two are not the same :)
17:18:30 <dustymabe> in fedora there is a new python minimization effort too
17:18:44 <dustymabe> so we should probably follow that
17:18:51 <kaeso[m]> am I allowed to circle back to the "are we releasing today?" topic?
17:19:04 <jlebon> yup, sure
17:19:54 <jlebon> dustymabe, kaeso[m], and I have been chasing a regression in update testing on aws: https://github.com/coreos/coreos-assembler/issues/1301
17:20:31 <jlebon> we're making progress and think it might be related to an ostree/rpm-ostree bump, but more investigation is needed
17:20:34 <kaeso[m]> and we somehow expected this to show up in the stable release we are doing today
17:20:51 <kaeso[m]> which indeed happened, https://github.com/coreos/fedora-coreos-streams/issues/79#issuecomment-611068739
17:21:36 <jlebon> my vote is to hold the release for now until we get to the bottom of it
17:22:00 <kaeso[m]> is this my understanding we already ignored it in a previous testing release, right?
17:22:13 <kaeso[m]> s/is this/it is/
17:22:20 <dustymabe> correct, in a previous `testing` release
17:22:26 <dustymabe> but not for stable
17:22:50 <kaeso[m]> and my testing canary on AWS already updated once past that
17:22:51 <jlebon> yes.  we didn't dig into it at the time, and it seemed more like a flake than a consistent failure
17:23:04 <kaeso[m]> and nobody else reported any update failures
17:23:07 <dustymabe> right. the flake vs consistent failure is important
17:23:29 <dustymabe> kaeso[m]: not that I know of, but I realize a lot less people are probably running testing
17:24:06 <kaeso[m]> ack
17:24:12 <jlebon> i definitely remember rerunning kola aws tests which used to fail and then passed. so there was some flakiness at some point in time
17:24:45 <kaeso[m]> what do we want to do at this point?
17:24:49 <dustymabe> I think we should definitely do some more investigation before we decide which way we want to go with the currently built release candidates
17:24:55 <jlebon> keep investigating :)
17:25:13 <dustymabe> kaeso[m]: it's late in your day so we can take it
17:25:29 <kaeso[m]> i.e. pausing both releases for now?
17:25:36 <dustymabe> yes
17:25:40 <dustymabe> IMHO
17:25:57 <jlebon> yup, agreed. i think we're close to getting to the bottom of it
17:26:10 <dustymabe> my testing canary upgraded fine last week, but it's got 8 cores and 16G memory
17:26:30 <kaeso[m]> ok
17:27:13 <jlebon> anything else for open floor?
17:27:17 <kaeso[m]> but I have a feeling current testing may already have the bug
17:27:51 <kaeso[m]> ehm, even the last two testing releases actually
17:28:01 <dustymabe> kaeso[m]: correct
17:28:15 <dustymabe> breaking testing is not nearly as bad as breaking stable though
17:28:32 <dustymabe> jlebon: I think we're good on open floor
17:28:35 <jlebon> yeah... we'll see. maybe there's an easy manual workaround for those. or maybe it only manifests in our tests
17:28:51 <kaeso[m]> yes, I was wondering whether to finish testing anyway and just pause stable
17:28:51 <jlebon> ok, will close in 45s
17:29:05 <kaeso[m]> but let's wait tomorrow I guess
17:29:10 <x3mboy> !
17:29:16 <x3mboy> Hello you guys
17:29:23 <x3mboy> Sorry, I'm here again to bother you
17:29:29 <jlebon> x3mboy: hi! :)
17:30:38 <x3mboy> Hi!
17:30:54 <x3mboy> I have this issue: https://pagure.io/design/issue/642
17:31:25 <x3mboy> And I will be more than happy to have it finished to launch those infographics this month
17:32:05 <jlebon> dustymabe: looks like you were tagged in there
17:32:21 <dustymabe> x3mboy: i'm time strapped - your best bet is to schedule a video meeting with me for 30m or 1 hour where we can extemporaneously brain storm on the issue
17:32:32 <dustymabe> if we don't do that I'll never get to it
17:33:23 <x3mboy> Let me know when and I will send you an invitation
17:33:44 <x3mboy> Or send me and invite and make this happens
17:33:49 <dustymabe> x3mboy: pm me and we can coordinate
17:33:53 <x3mboy> Ok
17:33:55 <x3mboy> Thanks
17:33:57 <x3mboy> eom
17:33:57 <dustymabe> and thanks for being persistent
17:34:07 <x3mboy> Well, it's part of my job
17:34:08 <x3mboy> :D
17:34:40 <jlebon> thanks x3mboy!
17:34:51 <jlebon> alrighty, closing in 20s
17:35:18 <jlebon> #endmeeting