15:02:20 <davdunc> #startmeeting  fedora cloud meeting
15:02:20 <zodbot> Meeting started Thu Jul  8 15:02:20 2021 UTC.
15:02:20 <zodbot> This meeting is logged and archived in a public location.
15:02:20 <zodbot> The chair is davdunc. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:20 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:02:20 <zodbot> The meeting name has been set to 'fedora_cloud_meeting'
15:02:31 <cyberpear> .hi
15:02:32 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
15:02:33 <dcavalca> .hi
15:02:34 <zodbot> dcavalca: dcavalca 'Davide Cavalca' <dcavalca@fb.com>
15:02:37 <michel> .hello salimma
15:02:39 <Eighth_Doctor> .hello ngompa
15:02:42 <zodbot> michel: salimma 'Michel Alexandre Salim' <michel@michel-slm.name>
15:02:45 <zodbot> Eighth_Doctor: ngompa 'Neal Gompa' <ngompa13@gmail.com>
15:02:48 <cmurf[m]> .hi
15:02:48 <zodbot> cmurf[m]: Sorry, but you don't exist
15:02:53 <cmurf[m]> haha
15:02:56 <cmurf[m]> awesome
15:03:02 <Eighth_Doctor> cmurf: you're `cmurf[m]` on the IRC side
15:03:08 <Eighth_Doctor> so it doesn't recognize you
15:03:11 <davdunc> hello everyone. even cmurf!
15:03:15 <cmurf[m]> it never does
15:03:19 <cmurf[m]> .hello cmurf
15:03:19 <zodbot> cmurf[m]: Sorry, but you don't exist
15:03:34 <cmurf[m]> i have an alias in fas and noggin, it's ignored
15:03:41 <cmurf[m]> .hello chrismurphy
15:03:42 <zodbot> cmurf[m]: chrismurphy 'Chris Murphy' <bugzilla@colorremedies.com>
15:03:58 <davdunc> we know you are here. :)
15:04:04 <michel> since FAS went multi-alias, I wonder if zodbot understood it
15:04:22 <davdunc> Alright. not a lot on the top of the queue today.
15:05:05 <davdunc> #topic QCOW image are not sparse
15:05:09 <davdunc> #link https://pagure.io/cloud-sig/issue/340
15:05:51 <davdunc> cmurf[m]: did  you want to share more here?
15:06:27 <cmurf[m]> issue has the details but probably the biggest gain would be to not make vagrant images 40G
15:07:34 <davdunc> so we have a fix proposed in lorax?
15:08:06 <davdunc> do we want to consider a workaround until that is completed?
15:08:31 <cmurf[m]> if sparse files are OK, and it seems they are or we'd have issues (right?) then then issuing `fstrim` before umount would also fix it and they'd be smaller than the above fix
15:08:49 <Eighth_Doctor> so do we want to just add fstrim to the kickstart?
15:09:01 <cmurf[m]> why is it a 40G image? is that kickstart?
15:09:12 <Eighth_Doctor> no, that's what ImageFactory does
15:09:19 <Eighth_Doctor> it creates a disk about that big and then runs Anaconda on it
15:09:26 <Eighth_Doctor> it essentially does a kickstart install
15:09:29 <Eighth_Doctor> it's not even using Lorax
15:10:12 <cmurf[m]> using `fstrim -av` in %post would work as a temp solution
15:10:52 <davdunc> #action add fstrim -av to the %post until BZ#1971186 is resolved
15:11:31 <davdunc> is everyone in agreement then?
15:11:41 <cyberpear> be sure to link to the bug in the kickstart so future us can find out "why" it's there :P
15:12:03 <davdunc> haha. cyberpear good callout
15:12:23 <Eighth_Doctor> yeah, we can do that
15:12:24 <davdunc> #link https://bugzilla.redhat.com/show_bug.cgi?id=1971186
15:12:29 <Eighth_Doctor> who's going to do it?
15:12:51 <davdunc> I am up for just adding it to the ks.
15:13:01 <davdunc> can do that after the meeting.
15:13:06 <davdunc> I can.
15:13:15 <Eighth_Doctor> sounds good
15:13:23 <Eighth_Doctor> you should action yourself for the record
15:13:50 <davdunc> #action davdunc to add fstrim -av to the cloud base kickstart
15:14:28 <davdunc> and Ill consider that agreed. I should add that in.
15:15:00 <davdunc> #agreed temporarily adding in fstrim command to the cloud kickstart to decrease the root volume size.
15:15:46 <davdunc> # topic add btrfs to containers overlay
15:15:57 <davdunc> #link https://pagure.io/cloud-sig/issue/332
15:16:08 <davdunc> I did that wrong. :)
15:16:21 <davdunc> #topic add btrfs to containers overlay
15:17:10 <davdunc> this was also proposed by cmurf[m]  and we discussed this at the last meeting.
15:17:26 <cmurf[m]> so i guess upstream containers/storage folks are against it
15:18:03 <cmurf[m]> meanwhile there are folks who use the btrfs driver automatically if the fs is btrfs
15:18:25 <davdunc> do we have a path forward without upstream support?
15:18:31 <cmurf[m]> e.g. Docker (but I haven't tested this so I'm not sure how common it is)
15:19:17 <cmurf[m]> Fedora can have its own default
15:20:18 <davdunc> So do we need that as an entry in storage.conf like Dan said and is that a system-wide change?
15:21:02 <cmurf[m]> I think it's a good idea to use the change process to explore the issues.
15:21:19 <cmurf[m]> Seems self contained but maybe not.
15:21:25 <Eighth_Doctor> we should probably do some testing before that, but the upcoming version supports drop-in files to configure things
15:21:25 <davdunc> I think that's the right course of action.
15:23:14 <cmurf[m]> Or would it be a system-wide change?
15:23:20 <davdunc> #action we need to some testing for the btrfs driver for container storage driver set to btrfs
15:23:57 <cmurf[m]> Is there a test suite to run?
15:24:11 <davdunc> I don't know.
15:24:13 <cmurf[m]> I've done rudimentary tests with the btrfs driver in podman and it works fine
15:25:50 <davdunc> is this a benefit for the new changes Eighth_Doctor has so generously committed supporting btrfs on the cloud images? We might want to explore it as a contained change for cloud images only.
15:26:05 <davdunc> though it would be helpful for workstation as well, would it not?
15:26:15 <Eighth_Doctor> it'll be useful for all flavors using btrfs
15:26:25 <cmurf[m]> I think if it'd be just Cloud edition, and affects only podman, it'd be self contained.
15:26:48 <Eighth_Doctor> but the containers team believes that it's not usable or tested (by what definition, I dunno)
15:26:48 <cmurf[m]> so question is if Cloud Edition wants to lead on it before attempting a system-wide change for f36?
15:27:07 <dcavalca> do we have some good testcases for this?
15:27:19 * Eighth_Doctor shrugs
15:27:21 <davdunc> dcavalca: I think the answer to that is no.
15:27:38 <Eighth_Doctor> the two main cases we need to test is the podman case and the kubernetes+cri-o case
15:28:03 <Eighth_Doctor> both need to be validated as fully functional for even cloud edition to change
15:28:12 <Eighth_Doctor> much less moving all btrfs-based Fedora variants
15:28:30 <Eighth_Doctor> we'd also need to make sure both rootless and rootful cases work as expected
15:28:54 <cmurf[m]> yeah i've done the latter
15:28:58 <davdunc> that makes sense. So we need tickets to identify that both of those conditions are met?
15:29:08 <Eighth_Doctor> yes
15:29:24 <Eighth_Doctor> we need to come up with a way to test this reliably too, ideally in a way we can run them in OpenQA
15:29:40 <Eighth_Doctor> we may also, as the Cloud WG, want to take ownership of the kubernetes package in Fedora
15:29:51 <Eighth_Doctor> and make sure we validate it for Cloud Edition
15:30:02 <cmurf[m]> for rootless, if `user_subvol_rm_allowed` mount option is used, it all works the same as rootful; since that is not a default mount option, what we get is a fallback to unlink() (basically it's rm -rf)
15:30:31 <davdunc> so are we taking ownership of the packages or just validating them? it seems like ownership would follow along with FCOS more than cloud to me.
15:30:32 <Eighth_Doctor> but one thing I expect we're going to want to do is make sure btrfs driver works fully in rootless mode
15:30:39 <michel> Fedora toolbox uses podman and container images, right?
15:30:50 <Eighth_Doctor> davdunc: validating the usability for now
15:30:52 <michel> so any change to the containers should benefit Silverblue power users too
15:31:06 <Eighth_Doctor> technically the kubernetes package right now is semi-orphaned
15:31:18 <Eighth_Doctor> it's nominally cared for, but isn't used or tested by anybody
15:31:42 <Eighth_Doctor> cri-o is still being reviewed for unretirement
15:32:08 <davdunc> Eighth_Doctor: that's an interesting consideration. Is there a high level of activity there?
15:32:31 <Eighth_Doctor> the kubernetes package mostly just gets version bumps and not much else
15:32:41 <Eighth_Doctor> there are folks sending pull requests to fix things, so I assume it works
15:32:56 <Eighth_Doctor> but it'd be a good idea to produce some basic tests on Cloud Edition for it
15:33:12 <Eighth_Doctor> none of that is something I expect for F35, this is for the future
15:33:13 <davdunc> okay. that's sounding like something we should pick up separately if we need to, but we do need to validate it.
15:33:21 <Eighth_Doctor> yup
15:33:47 <Eighth_Doctor> de minimis, we need to validate podman rootless and rootful with btrfs, fix bugs in containers/storage for btrfs in those workflows
15:34:08 <Eighth_Doctor> but we should also test k8s+cri-o with btrfs to ensure that works too
15:34:44 <davdunc> #action davdunc to file tickets on the test cases, one for podman and one for k8s+cri-o with btrfs
15:35:11 <cmurf[m]> i guess we need to ask around if there's a testsuite? because if running "hello world" counts as validation, all of those use cases except k8s, is validated (by me) :P
15:35:12 <davdunc> do we need to confirm root vs. rootless in poth of those cases or just for podman
15:35:31 <davdunc> cmurf[m]: I would call that a smoke test.
15:35:43 <Eighth_Doctor> davdunc: I don't know, we need to find out more for k8s
15:35:51 <Eighth_Doctor> that whole world is weird to me
15:36:35 <davdunc> Eighth_Doctor: I am personally good with that as a point of completion and then we can add more as we run into them.
15:36:47 <Eighth_Doctor> yup
15:37:14 <Eighth_Doctor> the reason k8s is important is that the flag setting will affect any user of containers/storage
15:37:21 <Eighth_Doctor> that's podman and cri-o
15:37:29 <davdunc> okay. So we will have the two tickets for test conditions.
15:37:46 <cmurf[m]> oh... and upgrades
15:38:04 <cmurf[m]> we'd have to prevent it from silently flipping to btrfs driver when doing upgrades
15:38:26 <davdunc> yes.
15:38:59 <davdunc> more fodder for Eighth_Doctor 's consideration of owning the k8s packages
15:39:23 <davdunc> to ensure that we have the conditions right for managing the config.
15:40:38 <davdunc> anything else we want to consider here? We are just testing it right now, correct?
15:41:15 <Eighth_Doctor> yes
15:41:20 <Eighth_Doctor> that's all
15:41:30 <Eighth_Doctor> we need the data for being able to prepare the change proposal anyway
15:41:31 <dcavalca> speaking of k8s, if anybody cares, I got helm packaged a while ago
15:41:39 <Eighth_Doctor> oh nice!
15:41:40 <dcavalca> mostly for shit and giggles, as I use it at home
15:41:44 <davdunc> :D
15:41:45 <Eighth_Doctor> haha
15:41:48 <dcavalca> but if anybody wants to co-maintain it, happy to add you
15:42:20 <dcavalca> https://src.fedoraproject.org/rpms/golang-helm-3
15:42:47 <davdunc> very cool.
15:42:57 <michel> helm will always be an Emacs thing for me
15:42:59 <michel> name collisions
15:43:03 <Eighth_Doctor> For reference, here's the kubernetes package: https://src.fedoraproject.org/rpms/kubernetes
15:43:07 <davdunc> should we move over to the bigger changes?
15:43:37 <davdunc> #topic F35 Change: Build Fedora Cloud Images with Hybrid BIOS+UEFI Boot Support
15:44:09 <Eighth_Doctor> yup
15:44:11 <davdunc> Eighth_Doctor: you have completed the modifications for this, right? we are all buttoned up on this.
15:44:15 <Eighth_Doctor> yes
15:44:20 <Eighth_Doctor> we are, however, missing tests in OpenQA
15:44:24 <cmurf[m]> It's also working
15:44:41 <davdunc> we are missing too many tests in OpenQA.
15:44:42 <Eighth_Doctor> #link https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/238
15:44:57 <Eighth_Doctor> well, actually I don't think I can do the link thing
15:45:01 <Eighth_Doctor> but there is the issue
15:45:25 <davdunc> #link https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/238
15:45:34 <davdunc> I think it worked.
15:45:38 <Eighth_Doctor> on the upside, we now basically will never have the problem of images not booting because the environment was randomly UEFI instead of BIOS at image creation time
15:45:38 <cmurf[m]> Fedora-Cloud-Base-Rawhide-20210706.n.0.x86_64.raw boots both BIOS and UEFI
15:45:57 <davdunc> fantastic.
15:46:10 <cmurf[m]> but for some reason i don't understand, we're missing images since 20210706
15:46:11 <cmurf[m]> https://koji.fedoraproject.org/koji/packageinfo?packageID=21547
15:46:19 <cmurf[m]> both rawhide and f34 and f33
15:46:27 <davdunc> I noticed that!
15:46:29 <cmurf[m]> * rawhide and f34 and f33
15:46:42 <davdunc> yesterday I was looking for them and came up empty handed.
15:47:16 <cmurf[m]> since neither f33 or f34 were modified (by us, as far as i know) it's curious they too are affected
15:48:12 <davdunc> wait.
15:48:22 <davdunc> are you saying the change isn't in 33 and 34?
15:48:47 <davdunc> b/c it should only be in rawhide.
15:48:55 <davdunc> or did I misunderstand?
15:49:09 * davdunc had a different problem I think.
15:49:23 <cmurf[m]> correct
15:50:03 <davdunc> yea, we should see ext4 with <f35
15:50:18 <davdunc> and no UEFI support for x86_64
15:50:24 <cmurf[m]> "are affected" => are failing like rawhide
15:50:34 <cmurf[m]> sorry
15:50:41 <davdunc> hmm.
15:50:45 <davdunc> that's no good.
15:51:11 <davdunc> #action davdunc to continue working on the boot issue, but extend to 33 and 34
15:51:13 <cmurf[m]> but i'm not sure where the logs are
15:51:22 <cmurf[m]> i can't find cloud image creation tasks in koji
15:52:10 <cmurf[m]> mboddu would know the answer to this if Conan Kudo doesn't
15:52:21 <Eighth_Doctor> I have no idea
15:52:21 <cmurf[m]> where to find the logs
15:52:42 <davdunc> They point back to the execution of the image-factory build don't they.
15:52:46 <davdunc> https://kojipkgs.fedoraproject.org//packages/Fedora-Cloud-Base/34/20210706.0/data/logs/image/oz-x86_64.log
15:52:56 <davdunc> isn't this what you are looking for?
15:52:56 <cmurf[m]> easiest to ask in #fedora-releng
15:53:23 <cmurf[m]> yes but for 20210706 and 20210707
15:53:25 <cmurf[m]> the missing ones
15:53:40 <davdunc> okay.
15:53:45 <cmurf[m]> logs for the attempted (but presumably failed) creation of the images
15:54:14 * mboddu reading back
15:54:45 <cmurf[m]> oh wait
15:54:59 <cmurf[m]> Fedora-Workstation-Live-Rawhide-20210706.n.0
15:55:09 <cmurf[m]> that's the most recent one there too
15:55:15 <davdunc> yea.
15:55:18 <cmurf[m]> something else is going on
15:55:32 <mboddu> davdunc: From a brief reading back, all the compose logs are available at https://kojipkgs.fedoraproject.org/compose/ depending on which compose you are looking for
15:55:32 <davdunc> so 0707 just hasn't been published?
15:55:53 <mboddu> For cloud, they will be under https://kojipkgs.fedoraproject.org/compose/cloud/
15:56:05 <davdunc> mboddu: thanks
15:57:21 <mboddu> davdunc: The compose logs contain koji task numbers, you can go to koji and look at the task for those task specific logs
15:57:44 <davdunc> mboddu: super! yes. thanks for that.
15:57:56 <davdunc> let's table this for now.
15:58:04 <davdunc> we just have a couple of minutes left.
15:58:43 <davdunc> #topic Open Floor
15:58:55 <davdunc> Anything else we need to discuss?
15:59:04 <cmurf[m]> nope
15:59:44 <davdunc> I want to thank Eighth_Doctor for getting in the commits that we needed for GPT and for btrfs on the images
15:59:55 <Eighth_Doctor> I think there's someone here who wants to talk to us
15:59:56 <Eighth_Doctor> :)
15:59:57 <michel> Conan Kudo++
16:01:05 <Eighth_Doctor> doowder, I believe, you wanted to talk to us?
16:01:26 <doowder> Thanks
16:01:38 <doowder> I think I will put it in as an issue to make sure everyone has context
16:01:44 <doowder> and we can go from there
16:02:02 <doowder> That will probably be the most effective way so everyone has background on it
16:02:05 <davdunc> if you want to hit us with the high-level pitch I am listening.
16:02:14 <doowder> Cool--
16:02:20 <davdunc> we're all friends here.
16:02:25 <doowder> haha thanks
16:02:54 <doowder> So I am Alex from Shells by the way, and we have been talking with mattdm and neal about getting Fedora up on Shells
16:03:04 <davdunc> oh awesome!
16:03:36 <doowder> Yeah!  So we obviously want to do it right, but need some help with some of the integration to make sure it fits Fedora's liking as well as ours!
16:03:56 <doowder> So I know its extra work, but was hoping to get some help on getting it up and running and maintaining
16:04:11 <doowder> I would definitely be happy to provide accounts, etc. to everyone getting involved to help out
16:04:48 <doowder> So anyway, thats my high level pitch haha
16:04:50 <davdunc> #topic Add Fedora to Shells.
16:05:03 <davdunc> I think the cloud sig would be happy to pitch in there.
16:05:13 <davdunc> what do others think?
16:05:37 <doowder> That would be great!
16:05:43 <dcavalca> yeah, this seems fine
16:05:53 <cmurf[m]> sounds good to me
16:05:55 <carlwgeorge> not my meeting (i'm here waiting for the fpc meeting to start) but personally i'm concerned about associating fedora with shells.com due to the freenode fiasco
16:06:21 <davdunc> thanks carlwgeorge
16:06:23 <doowder> carlwgeorge understood, and i would love to help address any concerns from my end if possible
16:06:47 <carlwgeorge> the concern is exactly what i just laid out, it's pretty simple
16:06:57 <davdunc> we'll do that in the ticket and discuss next meeting .
16:07:08 <doowder> That sounds good
16:07:19 <doowder> I am available to chat about anything related to that though
16:07:30 <davdunc> I think that would be great to work out and we'll see where we can get to.
16:07:42 <doowder> Yeah sounds good
16:07:45 <davdunc> I think we are way over time now so anything before we end?
16:07:52 <doowder> Thanks so much davdunc
16:08:06 <davdunc> thanks for coming in and chatting doowder .
16:08:10 <Eighth_Doctor> thanks all!
16:08:14 <davdunc> #endmeeting