16:29:41 <dustymabe> #startmeeting fedora_coreos_meeting
16:29:41 <zodbot> Meeting started Wed Jul 22 16:29:41 2020 UTC.
16:29:41 <zodbot> This meeting is logged and archived in a public location.
16:29:41 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:29:41 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:29:41 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:29:45 <dustymabe> #topic roll call
16:30:19 <dustymabe> .hello2
16:30:20 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:30:35 <bgilbert> .hello2
16:30:38 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:30:45 <slowrie> .hello2
16:30:46 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:30:52 <darkmuggle> .hello2
16:30:53 <zodbot> darkmuggle: darkmuggle 'None' <me@muggle.dev>
16:31:00 <skunkerk> .hello sohank2602
16:31:01 <jlebon> .hello2
16:31:01 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:31:04 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:32:41 <dustymabe> #chair bgilbert slowrie darkmuggle skunkerk jlebon
16:32:41 <zodbot> Current chairs: bgilbert darkmuggle dustymabe jlebon skunkerk slowrie
16:33:04 <lorbus> .hello2
16:33:05 <zodbot> lorbus: lorbus 'Christian Glombek' <cglombek@redhat.com>
16:34:20 <dustymabe> welcome all
16:34:58 <dustymabe> Reminder.. Add meeting labels to tickets you want to discuss. If you don't have the ability to add labels to tickets just comment in the ticket and we'll try to get the meeting label added.
16:35:18 <dustymabe> #topic Action items from last meeting
16:35:48 <dustymabe> * bgilbert to update the ticket #571
16:36:07 <bgilbert> #info bgilbert updated #571
16:36:19 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/571#issuecomment-658891337
16:36:24 <dustymabe> thanks bgilbert
16:36:42 <dustymabe> moving on to meeting tickets. I'll try to start with the one we didn't get to last week
16:36:52 <dustymabe> #topic remove console=ttyS0 on metal
16:36:58 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/567
16:37:23 * dustymabe didn't see walters in the roll call - anyone else want to set the stage for this one?
16:37:30 <dustymabe> #chair lorbus
16:37:30 <zodbot> Current chairs: bgilbert darkmuggle dustymabe jlebon lorbus skunkerk slowrie
16:37:50 <dustymabe> if not, I can try
16:38:15 <dustymabe> ok here goes
16:39:00 <dustymabe> There seems to be some cases where having a default console=ttyS0 value on our kernel command line can cause long bootup times for some hardware.
16:39:18 <dustymabe> The proposal is to remove the console=ttyS0 argument
16:39:31 <dustymabe> Though there are things we need to consider
16:40:05 <dustymabe> like how much does changing the default help vs hurt (i.e. the people that are depending on that default today vs the number of people the no default would help)
16:40:08 <walters> .hello2
16:40:09 <zodbot> walters: walters 'Colin Walters' <walters@redhat.com>
16:40:16 <dustymabe> welcome walters
16:40:19 <dustymabe> #chair walters
16:40:19 <zodbot> Current chairs: bgilbert darkmuggle dustymabe jlebon lorbus skunkerk slowrie walters
16:40:35 <dustymabe> Some questions I posed in the ticket:
16:40:39 <dustymabe> How much does the 'slow serial' problem affect actual users today?
16:40:41 <dustymabe> What do other OS platforms do when configuring the serial console?
16:40:43 <dustymabe> If we were to change our current settings would that be worse than what the current defaults are?
16:40:45 <dustymabe> This gets into your question of the default of having a serial console attached or not.
16:41:25 <jlebon> yeah, that's an important related issue (whether you have a serial console attached or not)
16:41:28 <dustymabe> OR maybe we could smartly detect during install (for bare metal platforms) and update the serial console argument accordingly
16:42:10 <jlebon> because it means otherwise that you can't use the emergency console if you're using tty0, which is surprising
16:42:45 * dustymabe slightly remembers a conversation somewhere about us trying to make emergency console shells go to every configured `console=`
16:42:55 <dustymabe> bgilbert: do you remember that?
16:43:15 <bgilbert> IIRC there are two issues
16:43:24 <jlebon> if that's possible, that'd be great. it's a big UX pain
16:43:44 <bgilbert> we could make emergency console shells go to all consoles by hacking up emergency.target to start multiple shells
16:44:01 <bgilbert> but there's also the issue that userspace writes to /dev/console only go to the primary
16:44:18 <bgilbert> there was talk of a kernel patch to allow enabling /dev/console writes to specified secondary consoles
16:44:24 <bgilbert> but nothing was done AFAIK
16:44:29 <dustymabe> bgilbert: correct
16:45:10 <dustymabe> though even if they just get a shell they can run `systemctl --failed` and that gets you most of the wa there
16:45:43 <jlebon> would be cool if the the kernel/systemd implemented the same thing grub did, which is to "auto-select" on the device in which you typed something
16:46:25 <dustymabe> yeah, i would feel more comfortable about "changing behavior" if we could do that
16:46:30 <bgilbert> there are code improvements that can be made, but realistically, are we going to do them in the short term?
16:46:49 <bgilbert> this has been on the radar for a while already
16:47:21 <dustymabe> bgilbert: right, the multiple console and which one to send output to has been on the radar for a while, but we never proposed yanking one of them
16:47:25 <dustymabe> so that is the new part
16:48:24 <bgilbert> IMO serial as primary console doesn't make a lot of sense.  that's not the typical configuration and is fairly surprising.
16:48:54 <bgilbert> we've had a nontrivial number of users asking for help getting Ignition output because it went to serial instead of VGA
16:49:08 <bgilbert> and presumably there are more who didn't ask
16:49:09 <jlebon> bgilbert: +1
16:50:03 <bgilbert> so I'd be +1 to dropping ttyS0 from new releases (not upgrades of course)
16:50:09 <cyberpear> it's not a FCOS-specific problem; just more common here since we purposely fail boot
16:50:25 <bgilbert> coreos-installer now has good UI for adding it back
16:50:31 <bgilbert> cyberpear: +1
16:50:50 <walters> dropping ttyS0 is basically "remove all console=" arguments right?
16:51:10 <cyberpear> (same issue comes up on generic cloud images when folks try to console via try v serial)
16:51:25 <dustymabe> bgilbert: let me disect your statement slightly
16:51:27 <bgilbert> walters: yeah
16:51:35 <darkmuggle> What about having the installer detect the console its running from and use that as the default?
16:51:47 <dustymabe> "serial as primary console doesn't make a lot of sense"
16:51:56 <dustymabe> so would you support it as a secondary console (flip the ordering) ?
16:52:24 <dustymabe> darkmuggle: i thought about that too, would be nice
16:52:35 <dustymabe> darkmuggle: that really only applies to interactive installs though
16:52:36 <jlebon> yup that's https://github.com/coreos/fedora-coreos-tracker/issues/567#issuecomment-658200836
16:53:02 <darkmuggle> Serial as the primary might make sense depending on the configuration. For RHCOS it might more common, but less common on FCOS. And then for the cloud platforms make it a per-platform option.
16:53:05 <walters> i'm OK with that
16:53:29 <dustymabe> walters: can you be more specific with "that" :)
16:53:36 <walters> that = propagating iso console
16:53:41 <dustymabe> +1
16:54:28 <bgilbert> darkmuggle: I'm concerned that that'd be too much magic.  the installer might be running from an xterm or over SSH
16:54:32 <walters> but "isoconsole" notably builds on "use kernel default console", since the iso default is basically this
16:55:01 <bgilbert> darkmuggle: and environment-specific defaults are unpleasant to debug.  we had some of that in CL's coreos-install
16:55:24 <walters> oh hmm right, i guess implicitly here was making this specific to the *iso* maybe?
16:55:43 <bgilbert> dustymabe: I'm okay with it as secondary.  @spamcop noted in the bug that ttyS0,115200 is not necessarily correct
16:55:57 <bgilbert> I actually want to question the broader premise
16:56:31 <bgilbert> LOM systems will know how to scrape the graphics card, and many can dump VGA text mode output into a network connection as though it were a serial port
16:56:53 <bgilbert> so it's not super-clear to me that serial output is useful at all in most cases
16:57:20 <walters> heh, wow; I didn't know that
16:57:28 <bgilbert> AFAIK the whole ecosystem defaults to VGA, so the ecosystem is already adapted to that
16:57:36 <dustymabe> hmm, you must be using some fancy LOMs :) - the ones I've dealt with usually have like one sreen of text and almost no scrollback
16:58:00 <dustymabe> and forget about copy and paste.. welcome 1000 screenshots in a bug report
16:58:38 <jlebon> was gonna say, is anyone noawadays actually physically connecting to their machine's COM port? if you're using e.g. IPMI to boot the ISO, then presumably you must have some way to deal with other ISOs which don't output to serial (which IIUC is more common)
16:58:51 * nasirhm waves
16:59:05 <dustymabe> hi nasirhm
16:59:06 <jlebon> even `virt-install` BTW gives a warning about this when you boot an ISO
16:59:10 <dustymabe> #chair nasirhm
16:59:11 <zodbot> Current chairs: bgilbert darkmuggle dustymabe jlebon lorbus nasirhm skunkerk slowrie walters
16:59:38 <dustymabe> jlebon: i'm honestly not thinking of interactive cases, but rather automated installs and the default of the installed system
16:59:46 <nasirhm> dustymabe: Hey o/ , Sorry for being late, I'm attending GUADEC in parallel as well :)
16:59:54 <dustymabe> IPMI boards typically support SoL (serial over lan)
17:00:18 <dustymabe> so yeah, people aren't connecting to physical COM ports, they're using new technology, but same idea
17:00:49 <bgilbert> principle of least surprise, though.  people using LOM hardware know they're using it
17:01:01 <jlebon> dustymabe: right, but then they know to change the kargs after installing, no?
17:01:01 <bgilbert> we can document how to enable serial, and they'll do it if they want it
17:01:14 <dustymabe> yeah, i figure that's fair
17:01:22 <dustymabe> any concrete proposals?
17:02:03 <bgilbert> #proposed We'll drop all console= kargs on new installs.  Upgrades of existing machines won't be affected.  We'll document how to configure alternate consoles.
17:02:21 <nasirhm> bgilbert++
17:02:21 <zodbot> nasirhm: Karma for bgilbert changed to 6 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:02:53 <dustymabe> i think we need some nuance here
17:02:55 <jlebon> ack.  note also dropping all the kargs makes it easier for people to add the right kargs :)
17:03:13 <dustymabe> bgilbert: sorry, tangent coming
17:03:14 <bgilbert> if the change proves unpopular it's relatively safe to roll back
17:03:15 <jlebon> (because of https://github.com/coreos/fedora-coreos-tracker/issues/533#issuecomment-644319795)
17:03:40 <dustymabe> I'm just not sure.. So let me ask a question
17:03:53 <dustymabe> does this only apply to bare metal? what do we do for clouds ?
17:04:14 <bgilbert> right, so that's an open problem
17:04:25 <bgilbert> we never implemented a way to have cloud-specific grub configs
17:04:30 <bgilbert> which we need for, at least, Packet
17:04:46 <bgilbert> which wants console=ttyS1,115200 on amd64, and something else on ARM
17:05:49 <bgilbert> (probably we should have had a grub variable for the Ignition platform ID, but now that everyone's hand-hacking the kargs, it might be too late)
17:05:54 <dustymabe> the difference I see is that for bare metal we have a step where the user can change the setting, for clouds, their instance is launched
17:06:48 <jlebon> for clouds we still need the most appropriate karg. because otherwise the kernel will likely default to VGA even on clouds where you can connect/capture serial
17:06:54 <bgilbert> sure, but that's on us.  clouds just need a cloud-specific default
17:07:11 <bgilbert> once that exists, cloud users shouldn't need to change it
17:07:13 <dustymabe> bgilbert: what about clouds that allow both VGA and serial console access ?
17:07:25 <dustymabe> now we're worrying about which one the user is expecting
17:07:41 <bgilbert> ugh
17:07:44 <cyberpear> I'd be fine dropping all console= kargs
17:07:52 <bgilbert> I think we pick the "best" one and document it for that cloud
17:07:53 <dustymabe> sorry I'm just trying to think this through
17:08:05 <bgilbert> key difference between that and bare metal is that on cloud, we know the feature sets and what will work
17:08:19 <bgilbert> and we can say "to get Ignition errors, do X"
17:08:23 <cyberpear> (OpenStack can access either serial or VGA, depending...)
17:08:37 <jlebon> for those, i'd say we should just keep the status quo (which is probably serial)
17:11:32 <dustymabe> ok i've tangented, it's just something we need to work out
17:11:39 <bgilbert> in blscfg-land, do we have any option besides modifying the platform ID stamp script to inject the console arg?
17:12:13 <bgilbert> coreos-installer would have to mirror that code too
17:13:25 <dustymabe> ok so the existing #proposed stands, with some details to work out
17:13:27 <bgilbert> #proposed We'll drop all console= kargs on new bare-metal installs, and implement a platform-specific default on other platforms.  Upgrades of existing machines won't be affected.  We'll document how to configure alternate consoles.
17:13:53 <dustymabe> ack/nacks ?
17:14:03 <jlebon> ack
17:14:08 <miabbott> ack
17:15:00 <bgilbert> ack
17:15:01 <dustymabe> walters: cyberpear: darkmuggle ?
17:15:20 <jlebon> bgilbert: i see you ack'ing your own proposal :)
17:15:29 <dustymabe> :)
17:15:32 <bgilbert> it's not obvious that I support it :-P
17:15:35 <cyberpear> ack
17:15:35 <x3mboy> .hello2
17:15:36 <zodbot> x3mboy: x3mboy 'Eduard Lucena' <eduardlucena@gmail.com>
17:16:24 <dustymabe> ok in general I support making this change, some people have come up with some compelling arguements. I'd just like to say that changing the default here might cause more issues than it solves. I think the users that this helps is not significant enough to change the default. It's a bit like: "do we enable DHCP by default or require everyone to configure networking because some people
17:16:26 <dustymabe> need static networking" is the same as "do we enable a serial console device by default or require everyone to configure it because some people need a different one or not one at all".
17:17:19 <dustymabe> no need for further discussion, just my personal opinion and I think disagreements are healthy
17:17:25 <nasirhm> dustymabe: I agree with changing the default here might cause more issues than it solves.
17:17:43 <dustymabe> #agreed We'll drop all console= kargs on new bare-metal installs, and implement a platform-specific default on other platforms.  Upgrades of existing machines won't be affected.  We'll document how to configure alternate consoles.
17:18:13 <dustymabe> next topic
17:18:17 <dustymabe> sorry that one went long
17:18:36 <dustymabe> #topic 2020-06-29: gather status update for Fedora Council
17:18:41 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/576
17:19:00 <dustymabe> What's changed in the last 6 weeks or so in FCOS land?
17:19:18 <dustymabe> jlebon: something about networking on ignition boot
17:19:41 <jlebon> yeah that's a good one
17:19:45 <dustymabe> nice work by abai on the download page and releases page
17:19:54 <dustymabe> jlebon: if you could help with better wording that would be great
17:20:15 <bgilbert> various useful features in the coreos-installer 0.3.0 release
17:20:18 <dustymabe> we could have a bullet about the switch we pulled off for persistent NIC naming
17:20:23 <bgilbert> --ignition-url, kargs
17:20:56 <dustymabe> https://hackmd.io/xnAKBX1nSKq0ZEJm4EFfTQ
17:22:12 <dustymabe> Anything to mention about LUKs / complex root device stuff ?
17:22:24 <jlebon> i'll add something
17:22:35 <dustymabe> we also did a lot of work in Ignition
17:23:38 <bgilbert> Ignition gs:// URLs
17:23:59 <cyberpear> definitely valuable to have some proactive communication about the new features/ how to use them
17:24:11 <nasirhm> The hackmd note looks awesome, alot of new features :D
17:24:34 <dustymabe> yeah I need to look at some of my changelogs to see what else happened. So much work going on
17:24:45 <dustymabe> I'll try to get to the next topic real quick before the top of the hour
17:25:26 <dustymabe> #topic Turn off (mute or disable) system beep on Fedora CoreOS Live by default
17:25:31 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/569
17:26:14 <dustymabe> anybody with any thoughts on disabling the PC beep by default on the Live ISO?
17:27:33 <dustymabe> part of me feels like this would be going a bit too far in customizing whatever the Fedora defaults are and part of me feels like "whatever it's small"
17:27:46 <dustymabe> and part of me wonders why this is the default in Fedora
17:28:10 <jlebon> heh i feel about the same way
17:28:22 <bgilbert> +1
17:28:51 <dustymabe> i like how you agree with my nebulous statements that don't give us any actual direction
17:29:04 <dustymabe> 🤗
17:29:29 <dustymabe> we could dig in to why it's the default in fedora and possibly change that. Or maybe there is some fedora configuration that happens for other editions that we are missing
17:29:39 <nasirhm> I'm having a similar feeling, It seems lika a teeny tiny change but can be a modification of Fedora Defaults.
17:29:45 <dustymabe> or we could just apply the change (apply it on the live iso/interactive boot)
17:30:02 <dustymabe> i'm open to any path forward
17:30:41 <cyberpear> seems like a change to drive in Fedora generally
17:30:53 <nasirhm> How about getting a heads up on Fedora defaults side and see what change needs to be made.
17:31:05 <dustymabe> ok that seems like a path forward
17:31:36 <dustymabe> #info we'll look into what the defaults are for Fedora with regards to PC beep and try to determine what thye correct path forward is.
17:31:44 <dustymabe> #topic open floor
17:31:55 <dustymabe> sorry all for the meeting running long
17:31:58 <dustymabe> any topics for open floor ?
17:32:31 <dustymabe> FYI: I'm meeting with releng/infra later this week to see if we can make progress/find solutions on the extensions proposal and also the multi-arch POC
17:32:37 <nasirhm> dustymabe++ thanks for running the meeting and the ping on -coreos channel
17:33:50 <dustymabe> thanks nasirhm - thanks for helping contribute to our documentation
17:33:52 <dustymabe> nasirhm++
17:33:52 <zodbot> dustymabe: Karma for nasirhm changed to 12 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:34:03 * jlebon has to drop for mtgs
17:34:10 <jlebon> thanks dustymabe! thanks all!
17:34:44 <dustymabe> #endmeeting