16:29:41 #startmeeting fedora_coreos_meeting 16:29:41 Meeting started Wed Jul 22 16:29:41 2020 UTC. 16:29:41 This meeting is logged and archived in a public location. 16:29:41 The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:29:41 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:29:41 The meeting name has been set to 'fedora_coreos_meeting' 16:29:45 #topic roll call 16:30:19 .hello2 16:30:20 dustymabe: dustymabe 'Dusty Mabe' 16:30:35 .hello2 16:30:38 bgilbert: bgilbert 'Benjamin Gilbert' 16:30:45 .hello2 16:30:46 slowrie: slowrie 'Stephen Lowrie' 16:30:52 .hello2 16:30:53 darkmuggle: darkmuggle 'None' 16:31:00 .hello sohank2602 16:31:01 .hello2 16:31:01 skunkerk: sohank2602 'Sohan Kunkerkar' 16:31:04 jlebon: jlebon 'None' 16:32:41 #chair bgilbert slowrie darkmuggle skunkerk jlebon 16:32:41 Current chairs: bgilbert darkmuggle dustymabe jlebon skunkerk slowrie 16:33:04 .hello2 16:33:05 lorbus: lorbus 'Christian Glombek' 16:34:20 welcome all 16:34:58 Reminder.. Add meeting labels to tickets you want to discuss. If you don't have the ability to add labels to tickets just comment in the ticket and we'll try to get the meeting label added. 16:35:18 #topic Action items from last meeting 16:35:48 * bgilbert to update the ticket #571 16:36:07 #info bgilbert updated #571 16:36:19 #link https://github.com/coreos/fedora-coreos-tracker/issues/571#issuecomment-658891337 16:36:24 thanks bgilbert 16:36:42 moving on to meeting tickets. I'll try to start with the one we didn't get to last week 16:36:52 #topic remove console=ttyS0 on metal 16:36:58 #link https://github.com/coreos/fedora-coreos-tracker/issues/567 16:37:23 * dustymabe didn't see walters in the roll call - anyone else want to set the stage for this one? 16:37:30 #chair lorbus 16:37:30 Current chairs: bgilbert darkmuggle dustymabe jlebon lorbus skunkerk slowrie 16:37:50 if not, I can try 16:38:15 ok here goes 16:39:00 There seems to be some cases where having a default console=ttyS0 value on our kernel command line can cause long bootup times for some hardware. 16:39:18 The proposal is to remove the console=ttyS0 argument 16:39:31 Though there are things we need to consider 16:40:05 like how much does changing the default help vs hurt (i.e. the people that are depending on that default today vs the number of people the no default would help) 16:40:08 .hello2 16:40:09 walters: walters 'Colin Walters' 16:40:16 welcome walters 16:40:19 #chair walters 16:40:19 Current chairs: bgilbert darkmuggle dustymabe jlebon lorbus skunkerk slowrie walters 16:40:35 Some questions I posed in the ticket: 16:40:39 How much does the 'slow serial' problem affect actual users today? 16:40:41 What do other OS platforms do when configuring the serial console? 16:40:43 If we were to change our current settings would that be worse than what the current defaults are? 16:40:45 This gets into your question of the default of having a serial console attached or not. 16:41:25 yeah, that's an important related issue (whether you have a serial console attached or not) 16:41:28 OR maybe we could smartly detect during install (for bare metal platforms) and update the serial console argument accordingly 16:42:10 because it means otherwise that you can't use the emergency console if you're using tty0, which is surprising 16:42:45 * dustymabe slightly remembers a conversation somewhere about us trying to make emergency console shells go to every configured `console=` 16:42:55 bgilbert: do you remember that? 16:43:15 IIRC there are two issues 16:43:24 if that's possible, that'd be great. it's a big UX pain 16:43:44 we could make emergency console shells go to all consoles by hacking up emergency.target to start multiple shells 16:44:01 but there's also the issue that userspace writes to /dev/console only go to the primary 16:44:18 there was talk of a kernel patch to allow enabling /dev/console writes to specified secondary consoles 16:44:24 but nothing was done AFAIK 16:44:29 bgilbert: correct 16:45:10 though even if they just get a shell they can run `systemctl --failed` and that gets you most of the wa there 16:45:43 would be cool if the the kernel/systemd implemented the same thing grub did, which is to "auto-select" on the device in which you typed something 16:46:25 yeah, i would feel more comfortable about "changing behavior" if we could do that 16:46:30 there are code improvements that can be made, but realistically, are we going to do them in the short term? 16:46:49 this has been on the radar for a while already 16:47:21 bgilbert: right, the multiple console and which one to send output to has been on the radar for a while, but we never proposed yanking one of them 16:47:25 so that is the new part 16:48:24 IMO serial as primary console doesn't make a lot of sense. that's not the typical configuration and is fairly surprising. 16:48:54 we've had a nontrivial number of users asking for help getting Ignition output because it went to serial instead of VGA 16:49:08 and presumably there are more who didn't ask 16:49:09 bgilbert: +1 16:50:03 so I'd be +1 to dropping ttyS0 from new releases (not upgrades of course) 16:50:09 it's not a FCOS-specific problem; just more common here since we purposely fail boot 16:50:25 coreos-installer now has good UI for adding it back 16:50:31 cyberpear: +1 16:50:50 dropping ttyS0 is basically "remove all console=" arguments right? 16:51:10 (same issue comes up on generic cloud images when folks try to console via try v serial) 16:51:25 bgilbert: let me disect your statement slightly 16:51:27 walters: yeah 16:51:35 What about having the installer detect the console its running from and use that as the default? 16:51:47 "serial as primary console doesn't make a lot of sense" 16:51:56 so would you support it as a secondary console (flip the ordering) ? 16:52:24 darkmuggle: i thought about that too, would be nice 16:52:35 darkmuggle: that really only applies to interactive installs though 16:52:36 yup that's https://github.com/coreos/fedora-coreos-tracker/issues/567#issuecomment-658200836 16:53:02 Serial as the primary might make sense depending on the configuration. For RHCOS it might more common, but less common on FCOS. And then for the cloud platforms make it a per-platform option. 16:53:05 i'm OK with that 16:53:29 walters: can you be more specific with "that" :) 16:53:36 that = propagating iso console 16:53:41 +1 16:54:28 darkmuggle: I'm concerned that that'd be too much magic. the installer might be running from an xterm or over SSH 16:54:32 but "isoconsole" notably builds on "use kernel default console", since the iso default is basically this 16:55:01 darkmuggle: and environment-specific defaults are unpleasant to debug. we had some of that in CL's coreos-install 16:55:24 oh hmm right, i guess implicitly here was making this specific to the *iso* maybe? 16:55:43 dustymabe: I'm okay with it as secondary. @spamcop noted in the bug that ttyS0,115200 is not necessarily correct 16:55:57 I actually want to question the broader premise 16:56:31 LOM systems will know how to scrape the graphics card, and many can dump VGA text mode output into a network connection as though it were a serial port 16:56:53 so it's not super-clear to me that serial output is useful at all in most cases 16:57:20 heh, wow; I didn't know that 16:57:28 AFAIK the whole ecosystem defaults to VGA, so the ecosystem is already adapted to that 16:57:36 hmm, you must be using some fancy LOMs :) - the ones I've dealt with usually have like one sreen of text and almost no scrollback 16:58:00 and forget about copy and paste.. welcome 1000 screenshots in a bug report 16:58:38 was gonna say, is anyone noawadays actually physically connecting to their machine's COM port? if you're using e.g. IPMI to boot the ISO, then presumably you must have some way to deal with other ISOs which don't output to serial (which IIUC is more common) 16:58:51 * nasirhm waves 16:59:05 hi nasirhm 16:59:06 even `virt-install` BTW gives a warning about this when you boot an ISO 16:59:10 #chair nasirhm 16:59:11 Current chairs: bgilbert darkmuggle dustymabe jlebon lorbus nasirhm skunkerk slowrie walters 16:59:38 jlebon: i'm honestly not thinking of interactive cases, but rather automated installs and the default of the installed system 16:59:46 dustymabe: Hey o/ , Sorry for being late, I'm attending GUADEC in parallel as well :) 16:59:54 IPMI boards typically support SoL (serial over lan) 17:00:18 so yeah, people aren't connecting to physical COM ports, they're using new technology, but same idea 17:00:49 principle of least surprise, though. people using LOM hardware know they're using it 17:01:01 dustymabe: right, but then they know to change the kargs after installing, no? 17:01:01 we can document how to enable serial, and they'll do it if they want it 17:01:14 yeah, i figure that's fair 17:01:22 any concrete proposals? 17:02:03 #proposed We'll drop all console= kargs on new installs. Upgrades of existing machines won't be affected. We'll document how to configure alternate consoles. 17:02:21 bgilbert++ 17:02:21 nasirhm: Karma for bgilbert changed to 6 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:02:53 i think we need some nuance here 17:02:55 ack. note also dropping all the kargs makes it easier for people to add the right kargs :) 17:03:13 bgilbert: sorry, tangent coming 17:03:14 if the change proves unpopular it's relatively safe to roll back 17:03:15 (because of https://github.com/coreos/fedora-coreos-tracker/issues/533#issuecomment-644319795) 17:03:40 I'm just not sure.. So let me ask a question 17:03:53 does this only apply to bare metal? what do we do for clouds ? 17:04:14 right, so that's an open problem 17:04:25 we never implemented a way to have cloud-specific grub configs 17:04:30 which we need for, at least, Packet 17:04:46 which wants console=ttyS1,115200 on amd64, and something else on ARM 17:05:49 (probably we should have had a grub variable for the Ignition platform ID, but now that everyone's hand-hacking the kargs, it might be too late) 17:05:54 the difference I see is that for bare metal we have a step where the user can change the setting, for clouds, their instance is launched 17:06:48 for clouds we still need the most appropriate karg. because otherwise the kernel will likely default to VGA even on clouds where you can connect/capture serial 17:06:54 sure, but that's on us. clouds just need a cloud-specific default 17:07:11 once that exists, cloud users shouldn't need to change it 17:07:13 bgilbert: what about clouds that allow both VGA and serial console access ? 17:07:25 now we're worrying about which one the user is expecting 17:07:41 ugh 17:07:44 I'd be fine dropping all console= kargs 17:07:52 I think we pick the "best" one and document it for that cloud 17:07:53 sorry I'm just trying to think this through 17:08:05 key difference between that and bare metal is that on cloud, we know the feature sets and what will work 17:08:19 and we can say "to get Ignition errors, do X" 17:08:23 (OpenStack can access either serial or VGA, depending...) 17:08:37 for those, i'd say we should just keep the status quo (which is probably serial) 17:11:32 ok i've tangented, it's just something we need to work out 17:11:39 in blscfg-land, do we have any option besides modifying the platform ID stamp script to inject the console arg? 17:12:13 coreos-installer would have to mirror that code too 17:13:25 ok so the existing #proposed stands, with some details to work out 17:13:27 #proposed We'll drop all console= kargs on new bare-metal installs, and implement a platform-specific default on other platforms. Upgrades of existing machines won't be affected. We'll document how to configure alternate consoles. 17:13:53 ack/nacks ? 17:14:03 ack 17:14:08 ack 17:15:00 ack 17:15:01 walters: cyberpear: darkmuggle ? 17:15:20 bgilbert: i see you ack'ing your own proposal :) 17:15:29 :) 17:15:32 it's not obvious that I support it :-P 17:15:35 ack 17:15:35 .hello2 17:15:36 x3mboy: x3mboy 'Eduard Lucena' 17:16:24 ok in general I support making this change, some people have come up with some compelling arguements. I'd just like to say that changing the default here might cause more issues than it solves. I think the users that this helps is not significant enough to change the default. It's a bit like: "do we enable DHCP by default or require everyone to configure networking because some people 17:16:26 need static networking" is the same as "do we enable a serial console device by default or require everyone to configure it because some people need a different one or not one at all". 17:17:19 no need for further discussion, just my personal opinion and I think disagreements are healthy 17:17:25 dustymabe: I agree with changing the default here might cause more issues than it solves. 17:17:43 #agreed We'll drop all console= kargs on new bare-metal installs, and implement a platform-specific default on other platforms. Upgrades of existing machines won't be affected. We'll document how to configure alternate consoles. 17:18:13 next topic 17:18:17 sorry that one went long 17:18:36 #topic 2020-06-29: gather status update for Fedora Council 17:18:41 #link https://github.com/coreos/fedora-coreos-tracker/issues/576 17:19:00 What's changed in the last 6 weeks or so in FCOS land? 17:19:18 jlebon: something about networking on ignition boot 17:19:41 yeah that's a good one 17:19:45 nice work by abai on the download page and releases page 17:19:54 jlebon: if you could help with better wording that would be great 17:20:15 various useful features in the coreos-installer 0.3.0 release 17:20:18 we could have a bullet about the switch we pulled off for persistent NIC naming 17:20:23 --ignition-url, kargs 17:20:56 https://hackmd.io/xnAKBX1nSKq0ZEJm4EFfTQ 17:22:12 Anything to mention about LUKs / complex root device stuff ? 17:22:24 i'll add something 17:22:35 we also did a lot of work in Ignition 17:23:38 Ignition gs:// URLs 17:23:59 definitely valuable to have some proactive communication about the new features/ how to use them 17:24:11 The hackmd note looks awesome, alot of new features :D 17:24:34 yeah I need to look at some of my changelogs to see what else happened. So much work going on 17:24:45 I'll try to get to the next topic real quick before the top of the hour 17:25:26 #topic Turn off (mute or disable) system beep on Fedora CoreOS Live by default 17:25:31 #link https://github.com/coreos/fedora-coreos-tracker/issues/569 17:26:14 anybody with any thoughts on disabling the PC beep by default on the Live ISO? 17:27:33 part of me feels like this would be going a bit too far in customizing whatever the Fedora defaults are and part of me feels like "whatever it's small" 17:27:46 and part of me wonders why this is the default in Fedora 17:28:10 heh i feel about the same way 17:28:22 +1 17:28:51 i like how you agree with my nebulous statements that don't give us any actual direction 17:29:04 🤗 17:29:29 we could dig in to why it's the default in fedora and possibly change that. Or maybe there is some fedora configuration that happens for other editions that we are missing 17:29:39 I'm having a similar feeling, It seems lika a teeny tiny change but can be a modification of Fedora Defaults. 17:29:45 or we could just apply the change (apply it on the live iso/interactive boot) 17:30:02 i'm open to any path forward 17:30:41 seems like a change to drive in Fedora generally 17:30:53 How about getting a heads up on Fedora defaults side and see what change needs to be made. 17:31:05 ok that seems like a path forward 17:31:36 #info we'll look into what the defaults are for Fedora with regards to PC beep and try to determine what thye correct path forward is. 17:31:44 #topic open floor 17:31:55 sorry all for the meeting running long 17:31:58 any topics for open floor ? 17:32:31 FYI: I'm meeting with releng/infra later this week to see if we can make progress/find solutions on the extensions proposal and also the multi-arch POC 17:32:37 dustymabe++ thanks for running the meeting and the ping on -coreos channel 17:33:50 thanks nasirhm - thanks for helping contribute to our documentation 17:33:52 nasirhm++ 17:33:52 dustymabe: Karma for nasirhm changed to 12 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:34:03 * jlebon has to drop for mtgs 17:34:10 thanks dustymabe! thanks all! 17:34:44 #endmeeting