16:05:52 <dustymabe> #startmeeting fedora_cloud_meeting
16:05:52 <zodbot> Meeting started Tue Jun 23 16:05:52 2020 UTC.
16:05:52 <zodbot> This meeting is logged and archived in a public location.
16:05:52 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:05:52 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:05:52 <zodbot> The meeting name has been set to 'fedora_cloud_meeting'
16:05:55 <jdoss> .hello2
16:05:57 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:05:58 <dustymabe> #topic roll call
16:05:59 <King_InuYasha> .hello ngompa
16:06:00 <zodbot> King_InuYasha: ngompa 'Neal Gompa' <ngompa13@gmail.com>
16:06:01 <dustymabe> .hello2
16:06:03 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:06:53 <dustymabe> #chair michaelgugino jdoss King_InuYasha
16:06:53 <zodbot> Current chairs: King_InuYasha dustymabe jdoss michaelgugino
16:07:33 <michaelgugino> Do I need to say .hello2?
16:07:55 <cyberpear> o/
16:07:58 <cyberpear> .hello2
16:07:59 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:07:59 <King_InuYasha> ideally, it'd be nice to know who you are :)
16:08:13 <dustymabe> #chair cyberpear
16:08:13 <zodbot> Current chairs: King_InuYasha cyberpear dustymabe jdoss michaelgugino
16:08:30 <cyberpear> bigger turnout than typical!
16:08:41 <michaelgugino> .hello2
16:08:42 <zodbot> michaelgugino: michaelgugino 'Michael Gugino' <gugino.michael@yahoo.com>
16:08:53 <michaelgugino> It's not hard to figure out who I am ;)
16:09:09 <King_InuYasha> whoa, yahoo
16:09:09 <dustymabe> :)
16:09:23 <dustymabe> #topic action items from last meeting
16:09:55 <dustymabe> I think our last meeting time was when zodbot was down and we don't have logs from it, but I don't think there was much to report
16:10:07 <King_InuYasha> yeah
16:10:09 <dustymabe> from the meeting before that I think michaelgugino was going to work on ticket 301
16:10:24 <dustymabe> #topic  regular releases: uploading to clouds
16:10:30 <dustymabe> #link https://pagure.io/cloud-sig/issue/301
16:10:50 <dustymabe> The last update there in the ticket is from the ad-hoc session we did.
16:11:10 <dustymabe> michaelgugino have you had a chance to try out setting up a fedora messaging listener ?
16:11:53 <michaelgugino> yes, I did look into do this for a few hours recently.  The code is straight forward, but I'm not sure how to write a proper config or do a basic hello world type of thing to iterate on.  fedmesg.com or whatever it is, is being squatted, so relevant docs are not available.  I tried to find the source repository for that site, but I was unable.
16:12:41 <dustymabe> michaelgugino: it's even more confusing because there is an old implementation (fedmsg) and a new implementation (fedora messaging)
16:12:54 <dustymabe> try starting with the docs here: https://fedora-messaging.readthedocs.io/en/latest/#user-guide
16:14:06 <michaelgugino> okay, I have not seen that one.
16:14:16 <dustymabe> I would say a good starting point is for us to get a container with a small python listener that listens for when composes finish and then print out links to where the cloud artifacts are in the compose
16:14:46 <dustymabe> this container is a POC and can run anywhere, since you can listen to fedora messages from any machine on the internet
16:14:50 <dustymabe> so you can do it from your laptop
16:14:57 <michaelgugino> those docs look actually useful.  For instance "Fedora’s message broker has a publicly accessible virtual host located at amqps://rabbitmq.fedoraproject.org/%2Fpublic_pubsub"  I already feel miles ahead of where I was.
16:15:39 <michaelgugino> Yeah, I couldn't figure out where to listen, and write a configuration file.  This I think will be more useful.  I've been meaning to ask you for some more help, but by the time allowed, it was almost today, so I figured I'd ask about it here.
16:16:01 <dustymabe> no worries
16:16:04 <King_InuYasha> fedora-messaging package ships a config file
16:16:08 <King_InuYasha> you can probably work from that
16:16:19 <dustymabe> the end goal is to be able to do something like this with your POC: https://github.com/coreos/fedora-coreos-releng-automation/tree/master/coreos-koji-tagger#rough-notes-for-running-locally
16:16:31 <dustymabe> except you won't need a keytab
16:17:10 <dustymabe> so it should be something as simple as podman build && podman run - and then watch the logs to see messages get printed to the screen
16:17:27 <michaelgugino> yeah, I'm on the same page I think
16:17:32 <dustymabe> +1
16:17:34 <dustymabe> thanks michaelgugino
16:17:43 <dustymabe> anything else for this topic?
16:17:58 <michaelgugino> nope
16:18:47 <dustymabe> #topic swaponzram for fedora cloud base image
16:18:58 <dustymabe> there was a mailing list post about this a while back
16:19:00 <dustymabe> one sec
16:19:39 <dustymabe> #link https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/message/QIO6LVG3MHVL4MRDPHS2RI6Z4MEL745M/
16:19:43 <jdoss> On most cloud providers swap is not a thing.
16:19:57 <dustymabe> jdoss: that has been my stance in the past
16:20:04 <King_InuYasha> that's going to change
16:20:06 <dustymabe> although I'm warming up to this idea
16:20:11 <King_InuYasha> cloud providers want to do hibernation
16:20:25 <King_InuYasha> and that means S4 suspend process needs to work
16:21:16 <dustymabe> my understanding (now) of swaponzram is that it actually doesn't require any swap device (on hard drive) at all. I previously misunderstood it to work in conjunction with swap backed by hard drive
16:21:41 <jdoss> Right OK so but how does that work with swaponzram?
16:21:56 <dustymabe> jdoss: who was that question for?
16:22:04 <King_InuYasha> swaponzram just makes it so you have compressed ram
16:22:05 <jdoss> more towards King_InuYasha
16:22:17 <King_InuYasha> it's not helpful at all towards S4 suspend
16:22:17 <dustymabe> right - i don't think swaponzram helps with hibernation
16:22:33 <King_InuYasha> we'd need swap partitions or swap files for that
16:23:04 <King_InuYasha> https://lwn.net/Articles/821158/
16:23:14 <dustymabe> considering swaponzram doesn't need a separate swap partition and it seems there are some benefits to using it, I don't think I'm opposed to enabling it for the cloud base image
16:23:36 <dustymabe> assuming the current CVE gets fixed
16:24:13 <King_InuYasha> sure
16:24:17 <King_InuYasha> I've got nothing against it
16:24:34 <michaelgugino> I'm trying to sort out in my head what swaponzram helps with.  Usually you don't need to swap until ram is full.  Seems like setting aside a part of ram to use when ram is full is very circular.
16:24:43 <jdoss> King_InuYasha: I figured as much so I was wondering why would care about Hibernation as a use case for swaponzram or Hibernation at all on Fedora Cloud. I will read that LWN later to see why people would care about Hibernation.
16:25:09 <jdoss> for my uses cases, I don't need swap. (Not saying we shouldn't consider it)
16:25:11 <King_InuYasha> I put that as a response to the general swap on cloud concept
16:25:18 <dustymabe> michaelgugino: right - but I think it helps prevent swap from getting full
16:25:18 <jdoss> Fair enough.
16:25:42 <dustymabe> so typically you set up swap so you don't get a bunch of OOMs
16:26:05 <cyberpear> definitely in favor of swaponzram for cloud
16:26:09 <dustymabe> in the cloud maybe you need a beefier instance and you can just start to use that
16:26:21 <dustymabe> but I think swaponzram actually gives us a nice balance
16:26:25 <cyberpear> it takes basically zero memory until you need it
16:26:41 <cyberpear> then if you do need it, it's there to save the day
16:26:46 <michaelgugino> I need to see an articulated usecase, I guess.
16:27:31 <dustymabe> michaelgugino: i.e. a case where swaponzram helps you as opposed to 1) no swap 2) disk based swap ?
16:27:40 <michaelgugino> right
16:27:46 <cyberpear> (MacOS has something similar, which I'd once found out by accident when trying to evaluate how it handled swap... I'd filled a bunch of memory programmatically, and it wasn't taking any memory! -- so had ot fill it w/ random data to actually make it fill)
16:28:15 <cyberpear> swaponzram is much higher performance than disk-based swap
16:28:31 <cyberpear> and doesn't eat up valuable disk space on cloud instances
16:28:52 <michaelgugino> I would say that memory is much more valuable than storage on cloud instances at this point.
16:29:03 <King_InuYasha> yes
16:29:05 <jdoss> I agree michaelgugino
16:29:09 <cyberpear> nor does it eat memory unless you're already running out, in which case, it compresses the memory for you so you have extra room for your workload
16:29:59 <dustymabe> right, but currently we don't configure swap at all (by default) for Fedora Cloud base
16:30:22 <dustymabe> so what I'm mostly comparing here is swaponzram, vs no swap
16:30:23 <cyberpear> indeed. and, yes, memory is the most precious cloud resource.  swaponzram helps you use it more efficiently
16:30:38 <michaelgugino> cloud-init can configure swap, right?  If you're missing a parition/lvm to put swap on, you can create a swap file.
16:30:41 <jdoss> I haven't had a case where I have wanted any of my workloads to use swap. Elasticsearch for example wants you to disable swap because it can slow things down. swaponzram seems great for desktops. I can't think of any of my use cases in the cloud where I would want to ever use swap.
16:31:33 <dustymabe> jdoss: typically I would have agreed with you
16:31:37 <dustymabe> but... :)
16:31:45 <michaelgugino> Right, I feel like on the cloud, once you start swapping, you're delaying the inevitable in most cases.  You've got a memory leak or otherwise your application is properly tuned to the size of your memory, etc.
16:31:56 <michaelgugino> *is not
16:32:02 <King_InuYasha> except, you put linux in a bad place when you don't have swap at all
16:32:16 <dustymabe> as cyberpear said earlier, you don't actually use zramonswap unless you get to a point where you're running out of memory already
16:32:27 <King_InuYasha> swap on zram means that you're not _really_ swapping, and linux's process management works properly, since inactive processes can be compressed
16:33:05 <jdoss> I can't wait for the FCOS debate on this dustymabe ;)
16:33:11 <King_InuYasha> blech
16:33:27 <dustymabe> jdoss: it already started
16:33:28 <jdoss> King_InuYasha: for real. bleh.
16:33:53 <King_InuYasha> of course, I have my can of worms I want to open, but...
16:33:56 <dustymabe> which is where I actually realized how swaponzram works (not requiring any backing partition on disk at all)
16:34:59 <dustymabe> the way I look at zramonswap now is that - it's pretty much like a safety harness.
16:35:01 <michaelgugino> We're talking about the cloud.  Once you hit the need for swap, you have already failed.  Adding some kind of compressible swap is only delaying the server falling over IMO.
16:35:02 <cyberpear> as far as I can tell, there's no down side, unless you consider "OOM Killer not killing my processes as fast" as a downside.
16:35:28 <jdoss> If we do enable it, we need to make sure uses can disable it.
16:35:30 <cyberpear> Read the "in defense of swap" article... it makes a very good argument
16:35:44 <jdoss> cyberpear: got a link handy?
16:35:46 <dustymabe> #link https://chrisdown.name/2018/01/02/in-defence-of-swap.html
16:35:53 <jdoss> ty dustymabe
16:35:57 <cyberpear> I've got production systems w/ 32G of ram, of which 16 is "available", but still have swapped out 1G
16:36:17 <cyberpear> the rest of the RAM is disk cache, and the 1G was just inactive "anonymous pages" memory
16:37:08 <jdoss> Do we have the link to the Fedora change request for swaponzram?
16:37:17 <dustymabe> #link https://fedoraproject.org/wiki/Changes/SwapOnZRAM#zram_Basic_function
16:37:31 <King_InuYasha> I'd pair this with earlyoom to make it so our environments are more responsive, but ehh
16:37:55 <michaelgugino> That article is about disk-based swap.  Nobody is saying disk-based swap is a bad idea.
16:38:20 <dustymabe> michaelgugino: so you configure you cloud instances with disk based swap ?
16:38:38 <King_InuYasha> dustymabe: I do ;)
16:39:03 <michaelgugino> I don't, no, but for certain workloads, it could make some sense.  In a file-system read-heavy environment, you can cache more objects rather than paging to disk.
16:39:27 <michaelgugino> (even though you're paging to disk, they objects are still in memory, rather than having to be re-read and reconstructed)
16:39:39 <jdoss> King_InuYasha: Speaking of EarlyOOM, I have been using https://github.com/rfjakob/earlyoom on my Fedora workstation for a while now and it's been not terrible.
16:40:11 <dustymabe> yeah I think the important thing is that we focus on what are sane defaults
16:40:24 <michaelgugino> Anyway, I'd like to see some cold hard facts/numbers before enabling this by default.  Find a particular usecase where this actually helps.
16:40:30 <dustymabe> my understanding in the past is that disk based swap was not something we wanted to configure by default because that has implications on the base disk image
16:41:01 <jdoss> dustymabe: can we come up with a way to enable after boot to test?
16:41:22 <King_InuYasha> dustymabe: I'd rather do something with a swapfile
16:41:23 <dustymabe> jdoss: I think so
16:41:57 <dustymabe> it would be a cloud-init snippet to enable it
16:41:58 <jdoss> or at least a a guide on doing so and maybe we can sprinkle some swaponzram into some workloads and see how it looks? I feel it will just be very subjective to how people are using Fedora Cloud.
16:41:59 <cyberpear> yeah, swap on disk by default is hard because of the partition auto-sizing magic, I'd think
16:42:13 <cyberpear> swaponzram could help something like "I've got a perfectly-tuned application/RAM balance, then dnf-daemon fired up in the background and caused my app to get OOM'ed" as a contrived example
16:43:08 <dustymabe> so we'll circle back on this in a few weeks - michaelgugino feel free to ping chris murhpy on the mailing list thread to ask for some specific examples
16:43:21 <michaelgugino> okay, I will do that.
16:43:34 <dustymabe> you have a link to the email that I posted earlier?
16:44:08 <michaelgugino> I believe I am subscribed to that list
16:44:56 <dustymabe> cool - I'll create a pagure ticket where we can track this dicsussion too
16:44:59 <michaelgugino> yes, I have the link and I'm subscribed.
16:45:07 <dustymabe> #topic open floor
16:45:14 <dustymabe> anyone with anything for open floor?
16:45:21 <jdoss> Most active Fedora Cloud meeting to date?
16:45:35 <King_InuYasha> so...
16:46:03 <jdoss> https://www.youtube.com/watch?v=lKie-vgUGdI
16:46:25 <King_InuYasha> so Chris Murphy and I have been working on this change proposal for desktops: https://fedoraproject.org/wiki/Changes/BtrfsByDefault
16:46:32 <jdoss> There are dozens of us... DOZENS!
16:46:40 <King_InuYasha> essentially proposing to switch to Btrfs for desktop variants
16:47:11 <King_InuYasha> it may be worth a consideration to look into doing this for cloud as well, based on the information we got from Facebook talking to us about btrfs a couple weeks ago
16:47:11 <jdoss> Didn't desktop go from Ext4 to Btrfs to xfs?
16:47:19 <King_InuYasha> desktop never went to xfs
16:47:35 <King_InuYasha> currently, the only variant using xfs is server edition
16:47:40 <jdoss> That's right.
16:48:03 <King_InuYasha> this is essentially the breakdown of options we've considered from the Workstation case: https://docs.google.com/spreadsheets/d/11Gszb48eYGSK37vOsFvkxY_nvcm_T26UL313wg7Aes0/edit#gid=0
16:48:24 <King_InuYasha> I think most of these actually would apply to us in the server case except for the sd-homed stuff
16:48:42 <dustymabe> Fedora CoreOS uses XFS
16:49:09 <King_InuYasha> I figured
16:49:48 <jdoss> But desktop did go from Ext4 to btrfs and then back to ext4 right?
16:49:52 <King_InuYasha> nope
16:49:57 <jdoss> because of btrfs issue?
16:49:58 <King_InuYasha> never actually switched to btrfs
16:50:11 <dustymabe> i'm a bit hesitant on the btrfs from for cloud - not really any particular concern other than "let's see how it goes for workstation"
16:50:15 <King_InuYasha> they went from ext3 -> ext4 -> ext4+LVM
16:50:19 <jdoss> or it was an option to install with btrfs.
16:50:31 <cyberpear> I'm excited about btrfs, and it really helps container casse
16:50:33 <cyberpear> *cases
16:50:38 <King_InuYasha> it's been an option to install with btrfs since ~2012-ish, I think
16:50:55 <jdoss> I had an install with btfs a while back and I lost data but that was like early 2012ish yeah.
16:51:11 <cyberpear> now that XFS has reflinks now, containers are better than before
16:51:14 <King_InuYasha> yeah, I've been personally running btrfs on all my machines for the past five years
16:51:35 <King_InuYasha> it's worked out wonderfully on ~100+ Fedora systems of mine
16:51:49 <jdoss> King_InuYasha: so you want to include Fedora Cloud on this change request?
16:51:51 <King_InuYasha> (mostly VMs with ~5-ish physical ones)
16:52:06 <King_InuYasha> I don't know, I figured I'd bring it up to everyone
16:52:29 <cyberpear> (it was great they finally merged your btrfs anaconda PR!)
16:52:36 * King_InuYasha sighs
16:52:40 <King_InuYasha> yeah, that took a while
16:52:44 <dustymabe> :)
16:52:47 <cyberpear> I wouldn't oppose including Cloud in the btrfs change
16:53:05 <cyberpear> or if folks want to wait a cycle after Desktop, I wouldn't oppose that either
16:53:18 <dustymabe> what I would prefer is to not change multiple times (since i've got limited time to spend on "cloud" stuff)
16:53:34 <dustymabe> I think I'd prefer to wait and see what happens with workstation
16:53:36 <King_InuYasha> I personally think it'd make a ton of sense, but I don't want to add Cloud without a variant "owner" to add to the list
16:53:38 <jdoss> I am neutral/positive on this change. I kinda side with dustymabe here too.
16:54:10 <King_InuYasha> no worries
16:54:15 <dustymabe> michaelgugino: any thoughts?
16:54:18 <King_InuYasha> I just wanted to bring it up so we can start thinking about it
16:54:32 <King_InuYasha> dustymabe: maybe make a ticket so we can track the idea?
16:54:42 <dustymabe> King_InuYasha: sounds good to me - do you mind?
16:54:47 <michaelgugino> dustymabe: my thoughts are, I don't want to use btrfs on much of anything.  Maybe it's stable now, IDK, but if it's not a file system that is supported on RHEL, I'm not running it.
16:54:54 <jdoss> King_InuYasha: what is the info you got from FB within regards to this?
16:55:22 * cyberpear missed part of the FB meeting
16:55:25 <King_InuYasha> they talked to us in depth about how they've been using it, what the corner cases are, and what they're plans for the evolution for their use-cases are
16:55:32 <dustymabe> King_InuYasha: was that meeting recorded ?
16:55:35 <King_InuYasha> no
16:55:40 <cyberpear> there's minutes
16:55:48 <King_InuYasha> but if we wanted to have another session, I could organize one
16:55:53 <King_InuYasha> to talk about cloud cases
16:56:25 <dustymabe> if you want to hear some of the sound bites you can listen to linux unplugged
16:56:30 <dustymabe> from that week
16:56:32 <King_InuYasha> they talked about running it on "millions" of servers, stressing the crap out of it with their infrastructure, and how it stood up to everything
16:56:32 <dustymabe> https://linuxunplugged.com/358
16:56:45 <cyberpear> https://lists.fedoraproject.org/archives/list/desktop@lists.fedoraproject.org/thread/RE3ML4225NDIS7M7QQ43MJX6OVL36MID/
16:57:10 <michaelgugino> As far as the cloud case, I want the primary storage to be xfs or ext4.  I would store valuable data not on root volume anyway.
16:57:42 <King_InuYasha> michaelgugino: the main value of btrfs in the cloud is the IO control stuff
16:57:48 <dustymabe> michaelgugino: right - i started off hesitant about btrfs so I have my data partitions on my desktop on ext4
16:57:56 <King_InuYasha> err IO isolation
16:58:04 <dustymabe> but I think now I can start moving some stuff over
16:58:14 <King_InuYasha> and the reflinks and snapshots and such for container and VM stuff
16:58:28 <dustymabe> for the system partitions honestly all of that stuff is easy to automate bringing it up so it doesn't much matter
16:58:36 <jdoss> Side topic, systemd-homed is this on the horizon for Fedora?
16:58:58 <dustymabe> #info dustymabe opened https://pagure.io/cloud-sig/issue/307 to talk about swaponzram
16:59:00 <michaelgugino> I don't know what the perceived benefits are, but if I did, I would opt-in to use them.  I think always adding new stuff that I have to learn is annoying.  I'm probably not the best proponent for running on the bleeding edge, for me, desktop system filestorage was solved 10 years ago
16:59:03 <King_InuYasha> jdoss: we don't know yet
16:59:12 <cyberpear> I think it's controversial, but some folks want "encrypted home dirs without password on boot"
16:59:46 <dustymabe> #action King_InuYasha to create a ticket to discuss possible future feature of btrfs for cloud base
16:59:57 <King_InuYasha> cool
16:59:57 * dustymabe notes we're out of time
17:00:01 <dustymabe> any other items for today?
17:00:06 <King_InuYasha> nothing from me
17:00:12 * jdoss waves
17:00:16 <jdoss> Thanks folks.
17:00:24 <dustymabe> #endmeeting