17:00:06 <zbyszek> #startmeeting FESCO (2021-06-08)
17:00:06 <zodbot> Meeting started Tue Jun  8 17:00:06 2021 UTC.
17:00:06 <zodbot> This meeting is logged and archived in a public location.
17:00:06 <zodbot> The chair is zbyszek. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:06 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:00:06 <zodbot> The meeting name has been set to 'fesco_(2021-06-08)'
17:00:06 <zbyszek> #meetingname fesco
17:00:06 <zodbot> The meeting name has been set to 'fesco'
17:00:06 <zbyszek> #chair nirik, decathorpe, zbyszek, sgallagh, mhroncok, dcantrell, defolos, mboddu, Conan_Kudo, Pharaoh_Atem, Son_Goku, King_InuYasha, Sir_Gallantmon, Eighth_Doctor
17:00:06 <zodbot> Current chairs: Conan_Kudo Eighth_Doctor King_InuYasha Pharaoh_Atem Sir_Gallantmon Son_Goku dcantrell decathorpe defolos mboddu mhroncok nirik sgallagh zbyszek
17:00:07 <zbyszek> #topic init process
17:00:15 <mboddu> .hello mohanboddu
17:00:15 <zodbot> mboddu: mohanboddu 'Mohan Boddu' <mboddu@bhujji.com>
17:00:19 <zbyszek> .hello2
17:00:20 <zodbot> zbyszek: zbyszek 'Zbigniew Jędrzejewski-Szmek' <zbyszek@in.waw.pl>
17:00:20 <Eighth_Doctor> .hello ngompa
17:00:21 <dcantrell> .hello2
17:00:23 <zodbot> Eighth_Doctor: ngompa 'Neal Gompa' <ngompa13@gmail.com>
17:00:26 <zodbot> dcantrell: dcantrell 'David Cantrell' <dcantrell@redhat.com>
17:00:47 <defolos[m]> .hello2
17:00:48 <zodbot> defolos[m]: Sorry, but you don't exist
17:00:57 <defolos[m]> .hello defolos
17:00:58 <zodbot> defolos[m]: defolos 'Dan Čermák' <dan.cermak@cgc-instruments.com>
17:01:07 <bcotton> .hello2
17:01:08 <zodbot> bcotton: bcotton 'Ben Cotton' <bcotton@redhat.com>
17:01:15 <sgallagh> .hello2
17:01:16 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
17:01:43 <zbyszek> It seems we have quorum, but let's wait a bit.
17:01:43 <nirik> morning
17:01:52 <zbyszek> mhroncok said he might not make it today.
17:02:01 <nirik> sorry i'm late, was trapped by a affectionate cat. ;)
17:02:10 <zbyszek> defolos, mboddu: welcome!
17:02:18 <decathorpe> .hello2
17:02:18 <zodbot> decathorpe: decathorpe 'Fabio Valentini' <decathorpe@gmail.com>
17:02:26 <mboddu> zbyszek: Glad to be here :)
17:02:32 <Eighth_Doctor> I'm double-booked and in three meetings atm, so I'm going to be a bit off atm
17:03:00 <zbyszek> #topic New meeting time?
17:03:08 <sgallagh> Eighth_Doctor: At least you have an excuse. I'm just "a bit off" in general.
17:03:31 <Eighth_Doctor> please for the love of god a new meeting time
17:03:31 <sgallagh> Is this time inconvenient for anyone besides Eighth_Doctor ?
17:03:37 <decathorpe> we could run another whenisgood?
17:03:55 <decathorpe> my schedule will change in two weeks anyway
17:03:56 <nirik> or frameadate
17:03:58 <zbyszek> #action zbyszek will make a whenisgood poll
17:04:07 <sgallagh> Eighth_Doctor: If god loved us he wouldn't have put us on FESCo :)
17:04:36 <defolos[m]> +1 on the whenisgood
17:04:36 <Eighth_Doctor> sgallagh: oh dear
17:04:37 <bcotton> sgallagh++
17:04:37 <zodbot> bcotton: Karma for sgallagh changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:04:42 <zbyszek> I also would like to move the meeting earlier. At 7 pm I'm wobbly and can only think about supper.
17:04:44 <dcantrell> sgallagh: fesco is secular
17:04:53 <sgallagh> dcantrell: More like "circular"
17:04:56 <sgallagh> aaanyway
17:05:04 <zbyszek> Yep, let's move on.
17:05:04 <sgallagh> zbyszek: Timeline on the whenisgood?
17:05:09 <michel> .hello salimma
17:05:11 * nirik notes frameadate is open source, whenisgood isn't
17:05:12 <zodbot> michel: salimma 'Michel Alexandre Salim' <michel@michel-slm.name>
17:05:15 <zbyszek> Tomorrow probably.
17:05:20 * mboddu prefers framadate
17:05:20 <michel> (cloud meeting running late)
17:05:21 <sgallagh> Responses by Friday to select next week's schedule?
17:05:41 <defolos[m]> Yes
17:05:41 <zbyszek> sgallagh: sounds good. nirik: I'll try frameadate.
17:05:46 <mboddu> Just because framadate is open source
17:05:58 <zbyszek> whenisgood is also somewhat confusing.
17:06:03 <mboddu> nirik: FYI, its framadate (may be pronounced as frame a date)
17:06:20 <nirik> yeah, that
17:06:29 <zbyszek> Took inspiration from UNIX: creat
17:06:34 <dcantrell> nice
17:06:36 <zbyszek> #topic #2617 F35 Change: Make btrfs the default file system for Fedora Cloud
17:06:36 <zbyszek> .fesco 2617
17:06:37 <zodbot> zbyszek: Issue #2617: F35 Change: Make btrfs the default file system for Fedora Cloud - fesco - Pagure.io - https://pagure.io/fesco/issue/2617
17:07:25 <zbyszek> FWIW, I found the discussion on fedora-devel not very enlightening. It was all about the meeting process and what happened in RHEL-8-timeframe.
17:07:41 <nirik> I used to be weakly in favor, now i am i think weakly not... but I should have raised my concerns on the list before. ;(
17:07:59 <zbyszek> I'm ready to vote +1, but I was wondering if there are technical issues that somebody would like to raise here.
17:08:19 <dcantrell> my concerns are still the same as when we moved to btrfs by default on desktop
17:08:23 <decathorpe> yeah. I also don't think what is supported on RHEL should hold this decision back
17:08:24 <defolos[m]> I'm weakly in favor as btrfs is pretty useful
17:08:33 <sgallagh> I'm really concerned about btrfs images not being bootable on RHEL
17:08:39 <dcantrell> I do not feel btrfs has really been proven reliable enough to be a default (note that's different than not offering it at all)
17:08:44 <nirik> I'm still unsure about openstack. I know in the past it used to have features to resize images before booting them (ie, not using cloud-init). It also had ability to do snapshots OUTSIDE the image...
17:09:20 <dcavalca> sgallagh: btrfs images should be bootable on RHEL hosts with no issues
17:09:21 <nirik> I think most of the big cloud providers just use cloud-init anymore.
17:09:25 <dcantrell> sgallagh: that's also a concern I have.  even though we're not RHEL, we exist in a world where RHEL exists and likelihood of RHEL being involved where Fedora Cloud images are is real
17:09:31 <defolos[m]> <dcantrell "I do not feel btrfs has really b"> Facebook would disagree
17:09:43 <sgallagh> dcavalca: VMs sure, container images...?
17:09:46 <michel> sgallagh: it's not that it's not bootable on RHEL though?
17:09:50 <defolos[m]> <dcantrell "sgallagh: that's also a concern "> Agreed there
17:09:52 <michel> it's that you can't mount the filesystem
17:10:17 <mboddu> I also have the concerns as sgallagh mentioned
17:10:35 <decathorpe> don't VMs use the kernel from inside the image to mount stuff?
17:10:44 <dcantrell> defolos[m]: I am aware that facebook uses btrfs, but their anecdotal evidence of it working fine is still not convincing to me.  their entire production environment is a mystery.  they could lose systems every 15 minutes to btrfs and the "fix" is to just spin up a new one.  we'll never know
17:10:52 <dcavalca> sgallagh: I'm not sure how container images are involved here?
17:10:59 <Eighth_Doctor> dcantrell: what about SUSE?
17:11:00 <cmurf> cloud WG meeting is happening now and we're discussing this now
17:11:03 <Eighth_Doctor> they've been using it since 2014
17:11:04 <michel> dcantrell: we definitely do *not* lose systems every 15 mins
17:11:06 <defolos[m]> Yes
17:11:18 <sgallagh> dcavalca: Maybe I misread, but I thought that was part of this: to make the Cloud container image use btrfs? Maybe I'm misremembering
17:11:18 <zbyszek> dcantrell: but we switched to btrfs in F34, and it's been very quiet.
17:11:23 <sgallagh> It's been a weird couple weeks
17:11:27 <dcantrell> Eighth_Doctor: suse has a bad track record of jumping to new filesystems before they are really reliable.  they jumped to reiserfs when it was still eating the ends of files
17:11:33 <defolos[m]> Btrfs has been in use a suse for ages
17:11:44 <defolos[m]> And proven reliable
17:11:52 <Eighth_Doctor> dcantrell: a track record of 1 isn't exactly a track record
17:11:55 <dcantrell> zbyszek: I am aware.  I'm just saying in the grand scheme of things, I still think it is too early to make btrfs a default.
17:12:01 <dcavalca> sgallagh: this change is about cloud images (i.e. the stuff you'd deploy on EC2)
17:12:05 <Eighth_Doctor> and suse did xfs before reiserfs too
17:12:14 <michel> defolos: the issues that Suse had are not relevant to us though (small root with auto snapshotting and no cleanup)
17:12:17 <sgallagh> dcavalca: OK, I got confused.
17:12:29 <dcantrell> Eighth_Doctor: ok fine, then I'm comparing to their decision to use reiserfs at all then.
17:12:32 <Eighth_Doctor> (or has the world forgotten that too? I know rh used to give suse crap about xfs before they hired the SGI team)
17:12:49 <sgallagh> For the record, I'm not really concerned about the reliability of btrfs; I'm willing to trust Facebook on this one
17:12:50 <cmurf> dcantrell FB does lose systems due to hardware problems, what josef has told me over and over is that Btrfs systems fall over at the same rate as XFS systems
17:12:50 <dcavalca> dcantrell: fwiw, I'm on the team that would get phone calls if we were losing machines due to btrfs every 15m, and I'm sleeping pretty well at night :)
17:13:06 <cmurf> and that these have been tracked down to failing hardware in every case in recent years
17:13:40 <Eighth_Doctor> heck that was even the case with the btrfs desktop change in F33
17:13:46 <cmurf> i've been triaging the desktop reports from users and they are very few, the #1 reported case are bitflips that are tracked back to bad RAM
17:13:48 <michel> IIRC we actually had the opposite issue with XFS (silent corruption that's only discovered later). not dissing on XFS, other filesystems would also not catch it
17:13:53 <Eighth_Doctor> me and cmurf did that triage and followed through deeply there
17:13:58 <dcantrell> dcavalca: glad you're sleeping, but still it's "take our word for it".  I want to see a much wider range of btrfs users across all distributions who have deliberately opted in to btrfs
17:14:03 <cmurf> i have the user test the memory and they find a problem, replace the ram, problem fixed
17:14:29 <sgallagh> I do worry that (as noted above) there is likely to be a strong overlap between RHEL VM hosts and Fedora VM guests.
17:14:30 <Eighth_Doctor> dcantrell: there's a range of minor distributions moving toward it, too, but frankly I don't consider that relevant
17:14:39 <dcantrell> sgallagh: regarding the RHEL mountability problem, do you think we could get someone from RH storage to comment on this change?
17:14:42 <sgallagh> Inability to use libguestfs and other tools make me wary
17:14:46 <zbyszek> dcantrell: that "testing across distributions" is what we're doing in Fedora: first desktop, now maybe cloud
17:14:59 * nirik would feel better about this with more info on if it would work ok with openstack and the like, but i suppose we could always approve it now and make sure in announcements/notes we ask people to test on every cloud they can.
17:15:04 <dcantrell> to test it in fedora doesn't require making it default, is all I'm saying
17:15:07 <cmurf> in a slightly smaller subset of problems, we see drives that are dying (early death transient corruptions) or just flat out don't properly honor flush/fua which is a problem for any file system
17:15:18 <defolos[m]> My only concern really is the rhel issue
17:15:26 <defolos[m]> As well
17:15:34 <sgallagh> dcantrell: We don't ship any VM images with non-default filesystems, so far as I'm aware
17:15:52 <dcantrell> Eighth_Doctor: does suse count in that group?  (kidding)
17:15:55 <nirik> perhaps the kmod sig in centos stream could have a btrfs kmod, not sure how well that would help rhel tho.
17:16:04 <sgallagh> Might be time for a straw poll to see if we have +5 before continuing to argue.
17:16:16 <dcavalca> nirik: that's actually something that's being discussed (both within kmods and within hyperscale)
17:16:38 * nirik nods
17:16:41 <Eighth_Doctor> there are at least three different strategies for btrfs on the RHEL family that I'm aware of
17:16:49 <dcavalca> sgallagh: libguestfs uses a dedicated kernel for its appliance iirc
17:16:54 <dcantrell> fwiw to everyone, I am using btrfs, I just feel it's still too early to call it a default.  maybe I'm being overly concerned
17:17:01 <dcavalca> it should be possible to have that support btrfs even if the system kernel doesn't
17:17:14 <sgallagh> Does it today, though?
17:17:30 <sgallagh> If yes, that would be sufficient to move me to a +1
17:17:42 <Eighth_Doctor> probably not, but that's the same kind of concern that happened with rpmzstd
17:17:50 <Eighth_Doctor> they wouldn't turn it on at all unless we did it first
17:18:02 <Eighth_Doctor> catch-22 sucks
17:18:19 <zbyszek> Eighth_Doctor: yeah, I think if we approve this in Fedora, this will make support in RHEL much more likely.
17:18:24 <cmurf> i'm not sure in what universe RHEL, without a single btrfs developer, can take the lead on btrfs if fedora doesn't aggressively take the lead on it. *shrug*
17:19:03 <dcantrell> zbyszek: I don't know how true that is
17:19:43 <Eighth_Doctor> dcantrell: if it's not, we can ship something in a third party repo or something
17:19:50 <michel> dcantrell: not sure about the 'much' but the opposite is definitely true, if it's not tested in Fedora first it won't get to RHEL
17:19:51 <dcantrell> Eighth_Doctor: true
17:20:00 <dcantrell> michel: yes, true
17:20:08 <defolos[m]> <michel "dcantrell: not sure about the 'm"> Right
17:20:14 <Eighth_Doctor> it basically becomes impossible to move things forward unless we do something first
17:20:18 <dcantrell> I think RH has made it clear that the only thing that would move them is a big customer asking for btrfs
17:20:23 <dcantrell> maybe Facebook could be that customer
17:20:28 <decathorpe> well, at least btrfs should be easier to support than stratis ;)
17:20:32 <Eighth_Doctor> :)
17:20:32 <cmurf> Cloud EPEL ship a libguestfs appliance that has a fedora kernel? thereby giving RHEL guestfish the ability to inspect btrfs images?
17:20:40 <zbyszek> dcantrell: internal customer, all fedora developers from rh :)
17:20:42 <dcantrell> does anyone understand stratis?
17:20:53 <Eighth_Doctor> I barely do
17:21:00 <Eighth_Doctor> I've spent more time than I care to admit digging into it
17:21:00 <dcantrell> zbyszek: I wish that had more pull than it does now, but that ends up being a hard sell
17:21:21 <dcantrell> Eighth_Doctor: I cannot wrap my head around it
17:21:22 <cmurf> s/Cloud/Could/
17:21:23 <cmurf> sorry
17:21:25 <nirik> "support" means different things to rhel than it does to fedora. ;)
17:21:29 <dcantrell> cmurf: whoa
17:21:47 <zbyszek> cmurf: don't cloud the issue
17:21:47 <Eighth_Doctor> dcantrell: we can have a conversation offline about it
17:21:48 <Eighth_Doctor> it's... ugly
17:21:49 <dcantrell> nirik: sure, but for most things support==compiles
17:21:57 <jforbes> I don't think changing this in Fedora will have any impact on RHEL btrfs support at all
17:21:58 <dcantrell> Eighth_Doctor: sure
17:22:23 <Eighth_Doctor> frankly, it doesn't matter, because pure userspace solutions are possible
17:22:33 <Eighth_Doctor> they're not great, but they exist
17:22:38 <dcantrell> so that was my next question
17:22:47 <dcantrell> is the user experience terrible for the pure userspace solution?
17:22:49 * decathorpe needs to watch the road for a bit, if it comes to a vote, count me as +1
17:23:08 <michel> dcantrell: Facebook is sadly not a big RHEL customer :)
17:23:14 <dcavalca> dcantrell: depends on what the userspace solution is
17:23:30 <dcavalca> for libguestfs, if we ship an appliance with the right kernel, it'll work out of the box with no difference
17:23:34 * Eighth_Doctor knows of a couple of big RHEL customers that might be interested, but he'd have to see if his contacts are still there...
17:23:41 <sgallagh> Is there a FUSE filesystem for it?
17:23:42 <dcantrell> michel: if they were, it might make these conversations simpler
17:23:47 <nirik> we could punt a week and ask more about libguestfs options and support in various cloud products...
17:23:50 * mboddu is still not decided on a +1 or a -1 :(
17:24:05 <mboddu> I agree with nirik
17:24:05 <dcavalca> sgallagh: there isn't, but it's technically possible to build one on top of btrfsprogs
17:24:11 <Eighth_Doctor> sgallagh: so.... not officially
17:24:17 <dcantrell> agreed with nirik
17:24:17 <michel> sgallagh: we were discussing FUSE as an option the other day
17:24:43 <zbyszek> nirik: +1.
17:24:53 <mhroncok> I ma here
17:24:55 <mhroncok> *am
17:24:56 <defolos[m]> Same here
17:24:58 <mhroncok> sorry for being late
17:25:01 <zbyszek> Eighth_Doctor, cmurf, dcavalca: can you add a description of userspace options to the wiki?
17:25:10 <dcavalca> zbyszek: sure
17:25:11 <defolos[m]> Although I lean towards +1
17:25:11 <zbyszek> michel also ^
17:25:43 <sgallagh> I still think it would be worthwhile to see if we have +5 before we defer another week
17:25:44 <Eighth_Doctor> zbyszek: I guess so...
17:25:57 <sgallagh> We can always approve with conditions
17:26:00 <michel> yeah, we can clarify the scope about containers and document userspace options
17:26:23 <defolos[m]> Michel Alexandre Salim: ++
17:26:24 <zbyszek> OK, let's make a quick poll: approve now or punt until next week?
17:26:32 <dcantrell> punt until next week
17:26:38 <sgallagh> I think I've come around to approval
17:26:48 <nirik> I'd prefer to wait a week
17:26:55 <mhroncok> I might as well be +1, but I have not read the discussion here yet
17:27:00 <defolos[m]> Punt
17:27:01 <mboddu> punt
17:27:14 <Eighth_Doctor> approve
17:27:14 <cmurf> there's no harm in waiting a week and we can restart the discussion on devel@ which got a bit side tracked
17:27:35 <zbyszek> #agree We'll wait a week for more discussion and information about userspace access options.
17:27:41 <cmurf> cloud wg pretty much decided to go with the change in their meeting just a bit ago
17:27:43 <mhroncok> however, I see no need to rush
17:28:12 <Eighth_Doctor> well, cloud wg decided it last month too :)
17:28:19 <zbyszek> #action Eighth_Doctor, cmurf, dcavalca, michel will provide some more information about userspace access options
17:28:25 <Eighth_Doctor> but we reaffirmed it after this week
17:28:40 <mboddu> And also if someone can confirm/deny the rhel problem as well, it would be great
17:28:45 <zbyszek> Eighth_Doctor: yeah, but if people want more discussion, it's better to do that.
17:28:52 <zbyszek> #topic Next week's chair
17:29:02 <zbyszek> vlntrs?
17:29:36 <zbyszek> If nobody wants, I can take it again.
17:29:45 <sgallagh> I'll do it
17:29:57 <zbyszek> #action sghallagh will chair next meeting
17:29:58 <zbyszek> #topic Open Floor
17:30:04 <zbyszek> sgallagh: thanks
17:30:13 <sgallagh> I won't be available the week after next though
17:30:29 <sgallagh> (In training that week)
17:30:51 * defolos[m] would be afk and finish supper
17:31:10 <zbyszek> If nothing, let's wrap this up. I'll wait 30s more.
17:31:23 <sgallagh> Did we congratulate defolos[m] and mboddu on their appointment to FESCo?
17:31:32 * mhroncok did, in the ticket
17:31:35 <mhroncok> :)
17:31:42 <zbyszek> sgallagh: I said "welcome" during init process
17:31:47 <sgallagh> Or possibly offer condolences? :_D
17:31:53 <dcantrell> defolos[m], mboddu: congratulations and welcome!
17:31:59 <mboddu> sgallagh: Haha :)
17:32:03 <defolos[m]> Lol
17:32:10 <mboddu> Thanks everyone :)
17:32:10 <defolos[m]> <dcantrell "defolos, mboddu: congratulations"> Thanks!
17:32:25 <zbyszek> In case anyone was wondering, ignatenkobrain is alive, just busy.
17:32:35 <zbyszek> I know cverna is OK too.
17:32:40 <dcantrell> good to hear
17:32:53 <sgallagh> Thanks for letting us know
17:32:54 <mboddu> Glad to hear, I haven't seen igor in a while
17:33:08 <zbyszek> See you'all next week.
17:33:09 <defolos[m]> Good to hear that
17:33:11 <zbyszek> #endmeeting