16:29:34 #startmeeting fedora_coreos_meeting 16:29:34 Meeting started Wed Oct 21 16:29:34 2020 UTC. 16:29:34 This meeting is logged and archived in a public location. 16:29:34 The chair is nasirhm. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:29:34 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:29:34 The meeting name has been set to 'fedora_coreos_meeting' 16:29:54 #topic roll call 16:30:07 .hello2 16:30:08 nasirhm: nasirhm 'Nasir Hussain' 16:30:32 .hello2 16:30:33 dustymabe: dustymabe 'Dusty Mabe' 16:30:35 .hello2 16:30:37 slowrie: slowrie 'Stephen Lowrie' 16:30:45 .hello sohank2602 16:30:48 skunkerk: sohank2602 'Sohan Kunkerkar' 16:30:59 .hello2 16:31:00 jdoss: jdoss 'Joe Doss' 16:31:02 there we go 16:31:14 .hello2 16:31:15 jlebon: jlebon 'None' 16:31:22 .hello redbeard 16:31:23 red_beard: redbeard 'Brian 'redbeard' Harrington' 16:31:39 .hello2 16:31:40 lucab: lucab 'Luca Bruno' 16:32:26 .hello2 16:32:27 cyberpear: cyberpear 'James Cassell' 16:32:37 #chair lucab red_beard jlebon jdoss skunkerk slowrie cyberpear dustymabe 16:32:37 Current chairs: cyberpear dustymabe jdoss jlebon lucab nasirhm red_beard skunkerk slowrie 16:32:53 #topic Action items from last meeting 16:33:14 #info dustymabe to send an email to the coreos list (not status) with a PSA about the meeting time sticking with UTC 16:34:11 #undo 16:34:11 Removing item from minutes: INFO by nasirhm at 16:33:14 : dustymabe to send an email to the coreos list (not status) with a PSA about the meeting time sticking with UTC 16:34:27 #info dustymabe sent email to the list about meeting sticking to UTC time https://github.com/coreos/fedora-coreos-tracker/issues/650 16:34:30 #undo 16:34:30 Removing item from minutes: INFO by dustymabe at 16:34:27 : dustymabe sent email to the list about meeting sticking to UTC time https://github.com/coreos/fedora-coreos-tracker/issues/650 16:34:36 #info dustymabe sent email to the list about meeting sticking to UTC time https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/message/KTEAQSBAZD6LVCKYJKZXB3LEHK723UB2/ 16:34:51 done! 16:34:59 Thank you dustymabe for working on it. 16:35:31 #topic How to enable kdump feature on Fedora CoreOS 16:35:43 #link https://github.com/coreos/fedora-coreos-tracker/issues/622 16:36:14 jlebon: you added the meeting label, do you want to give a summary to start discussion? 16:36:25 yup sure 16:37:06 there have been a number of requests both internally and externally to have kdump enabled in FCOS/RHCOS 16:37:36 this isn't something that we would enable by default, but we want to make it easy for users to enable it if they want 16:38:12 tech-wise, there isn't much that needs to be done to have kdump work. it mostly Just Works already. 16:38:45 so what's the open question? missing bits? 16:38:56 so we could in theory leave it as an extension users overlay 16:39:26 however, given that (1) kdump is tiny and (2) we want to reduce friction (possibly adding fcct sugar), there's general consensus that we should just bake it in 16:40:05 this ties into a few things, notably that by baking it in, it's easier to have it from day-1 16:40:26 i think it makes sense, but i'm always the easy one to convince to include something 16:40:27 we do eventually want to support "day-1 extensions" but there's still a lot of work that needs to happen 16:40:45 it might be worth at least answering the questions that we came up with recently for requests like this 16:40:55 the other piece of "day-1" is karg support via Ignition which is likely much easier 16:41:14 dustymabe: yeah, that's a good idea 16:41:20 the list: https://github.com/coreos/fedora-coreos-tracker/issues/641#issue-712286532 16:41:27 Can the package be run from a container? 16:41:29 Is it possible to layer the package onto the base OS as a day 2 operation? 16:41:32 Does the package have additional dependencies? (i.e. does it drag in Python, Perl, etc) 16:41:34 What is the size of the package? 16:41:35 Can the packaging be adjusted to just deliver binaries? 16:41:37 What is the intended use case of the package? 16:41:40 How can this package be abused as a Turing complete interpreter? 16:41:49 probably should've numbered those :) 16:41:59 one sec 16:42:25 1 Can the package be run from a container? 16:42:27 2 Is it possible to layer the package onto the base OS as a day 2 operation? 16:42:29 3 Does the package have additional dependencies? (i.e. does it drag in Python, Perl, etc) 16:42:31 4 What is the size of the package? 16:42:33 5 Can the packaging be adjusted to just deliver binaries? 16:42:35 6 What is the intended use case of the package? 16:42:38 7 How can this package be abused as a Turing complete interpreter? 16:42:53 .hello2 16:42:53 1. probably could be shoehorned into a container... though it's pretty "host-level". e.g. it needs to write to /boot 16:42:53 bgilbert: bgilbert 'Benjamin Gilbert' 16:43:12 #chair bgilbert 16:43:12 Current chairs: bgilbert cyberpear dustymabe jdoss jlebon lucab nasirhm red_beard skunkerk slowrie 16:43:32 2. it is, we're discussing here reducing friction further 16:44:06 3. not AFAIK, need to sanity-check that 16:44:11 jlebon: to be fair, it's just "writing to part of the host filesystem", that it's /boot is incidental 16:44:48 citing that as a reason exacerbates the meta-problem of folks thinking content _needs_ to be in the host mount namespace 16:45:41 i do think the dracut part should be containerized, not necessarily via podman; that's what rpm-ostree does with bwrap when you do `rpm-ostree initramfs --enable` 16:45:53 i'm not disagreeing with the sentiment that kdump is useful or even disagreeing with it's inclusion, just suggesting a bit more rigor with the justification 16:46:03 4. RPM pkg comes in at 486k, including docs 16:46:20 here's what I see when I layer it 16:46:22 Added: 16:46:24 dracut-squash-050-61.git20200529.fc32.x86_64 16:46:26 ethtool-2:5.9-1.fc32.x86_64 16:46:28 kexec-tools-2.0.20-17.fc32.x86_64 16:46:30 snappy-1.1.8-2.fc32.x86_64 16:46:32 squashfs-tools-4.3-25.fc32.x86_64 16:47:02 just a datapoint ^^ 16:47:23 red_beard: true, but /boot is pretty special though 16:47:43 it's dropping things directly into the ostree /boot fs layout 16:48:08 so technically it'd be cleaner to have it bound to ostree lifecycle wise 16:48:46 i probably should've added those details earlier :) 16:49:41 dustymabe: thanks. hmm yeah we should probably dig a bit into those 16:49:44 so what about question 6 and 7 16:50:04 6. to support kdump.service ? 16:50:16 6. intended use case is kdump enablement 16:51:37 7. i don't know enough about kdump, but unlikely :) 16:52:20 i'm sure some fun could be had with kexec, but if a bad actor can run kexec (privileged), you're owned already 16:52:33 we should think of this more as "growing the host API" rather than "baking in an extension" 16:52:58 sounds somewhat related to the bootupd project? 16:53:01 i.e. do we want FCOS/RHCOS to support kdump OOTB? 16:54:06 i think it makes sense, especially for something so low level, to not run it in a container. 16:54:21 it also would be nice to be able to support it OOTB 16:54:34 though I don't know specifically how many people would enable it 16:54:39 cyberpear: hmm, can you expand? they're mostly orthogonal i think (except that both do things in /boot) 16:54:49 cyberpear: that *is* a good point 16:55:14 Can some please provide some information on OOTB. 16:55:23 out of the box 16:55:24 out of the box 16:55:27 jlebon: i figured they might be similar since both muck w/ /boot 16:55:28 slowrie: youfast 16:55:46 Ah, Thanks :) 16:55:57 dustymabe: I stopped coughing long enough to type a response :) 16:56:02 "Linux systems handle updates for bootloader data in an inconsistent and ad-hoc way. .... The goal of [Bootupd] project is to be a cross-distribution, OS update system agnostic tool to manage updates for things like /boot/efi...." 16:56:34 cyberpear: right they're similar in that respect :) 16:56:45 okay, so maybe it's a difference of /boot vs /boot/efi, therefore unrelated to bootupd? 16:58:20 bootupd only owns the bootloaders, `/boot` in fact doesn't contain any of that although bootupd stores its states there 16:58:37 actually... `The scope is otherwise limited; for example, bootupd will not manage anything related to the kernel such as kernel arguments; that's for tools like grubby and ostree.` 16:58:45 IOW yes, not related to bootupd 16:58:51 so is anyone opposed to us trying to support this mechanism (used for crash investigation) more natively by including the kexec-tools package in the base layer? 16:59:15 /boot isn't a single thing. ostree manages /boot/loader and /boot/ostree, bootupd manages the MBR and /boot/efi 16:59:16 * cverna waves 16:59:33 👋 16:59:35 #chair cverna 16:59:35 Current chairs: bgilbert cverna cyberpear dustymabe jdoss jlebon lucab nasirhm red_beard skunkerk slowrie 16:59:41 i'd like to hear more about how this gets configured _consistently_ so that folks aren't YOLOing their way through the setup 17:00:14 red_beard: good point. ideally the FCCT sugar would help a lot with this 17:00:24 otherwise, we're just getting back to creating another general purpose distro 17:00:52 red_beard: it'd be pretty much what you'd expect: docs with config fragments to copypaste, and/or FCC sugar 17:01:12 I think in an ideal world: Ignition sets the kargs (and reboots during first boot initrd), enables kdump service. On every boot the kdump daemon checks if a crashkernel exists for the running kernel (generates if not present). FCC sugar to generate that Ignition snippet automatically. Did I miss anything? 17:02:05 so, related to the inclusion, we should have tasks to cover the generation of the docs & config fragments and issues filed for the sugar for FCCT, right? 17:02:56 to mark this as "done", that is 17:03:01 if it gets included 17:03:59 slowrie: maybe. i think it depends on how kdump.conf is configured in the wild. we may want sugar for that too, or it might be too broad 17:04:22 red_beard: that makes sense to me 17:05:15 jlebon: yeah; for the FCCT sugar ideally we scope it down to the littlest common set as possible with docs pointing to writing your own kdump.conf if you want more configuration 17:06:02 hmm, looks like kdump.conf doesn't support drop-ins. something to look into 17:06:32 the sugar could allow you to write arbitrary kdump.conf contents 17:06:54 when enabled, this also locks a bunch of memory in kernel-space, so we are not yet sure if people usually turn it on at install time, or just in time when needed 17:07:00 brb 17:08:05 anyone want to take a stab at closing off this discussion topic? 17:08:15 sorry, back now 17:08:20 dustymabe: i can try a proposa 17:08:21 l 17:10:06 #proposed we will include kexec-tools in FCOS to make crash collection built in. we will add documentation and possibly FCCT sugar to standardize how it's configured. we will investigate dependencies that it pulls in. 17:10:32 ack 17:10:35 +1 17:10:46 +1 17:10:57 +1 17:10:59 jlebon: if you don't mind can you try to answer 1-7 questions in the ticket, just for the record ? 17:11:06 +1 17:11:23 #agreed we will include kexec-tools in FCOS to make crash collection built in. we will add documentation and possibly FCCT sugar to standardize how it's configured. we will investigate dependencies that it pulls in. 17:11:27 dustymabe: ack, sure 17:11:53 red_beard: cyberpear: did that proposal look good? 17:11:57 sorry, maybe i didn't wait long enough. red_beard, lucab any reservations? 17:12:08 +1 17:12:10 yeah, seems fine 17:12:40 cool 17:12:45 cool, thanks! 17:13:15 Hmm, So let's move to our next ticket for today 17:13:42 #topic Fedora 33 rebase work 17:13:50 #link https://github.com/coreos/fedora-coreos-tracker/issues/609 17:14:03 #info there is a new `next` release coming out today/tomorrow 17:14:07 dustymabe: would you like to provide some context 17:14:21 Over the past two weeks we have: 17:14:40 added the migration to systemd-resolved: https://github.com/coreos/fedora-coreos-tracker/issues/646 17:14:50 fixed the chrony dhcp propagation test: https://github.com/coreos/fedora-coreos-tracker/issues/643 17:15:08 fixed the ext.config.root-reprovision* tests: https://github.com/coreos/fedora-coreos-tracker/issues/644 17:16:00 stopped downgrading the crypto policy for SSH RSA-1 keyex: https://github.com/coreos/fedora-coreos-config/pull/702 17:16:27 we have a new potential issue that did pop up 17:16:52 which is that in f33 it seems like systems get the hostname of `fedora` instead of `localhost` if there is no other hostname given: https://github.com/coreos/fedora-coreos-tracker/issues/649 17:17:03 which seems to be affecting OKD (I still need to look into it) 17:17:26 Thanks for the info about the highlights dustymabe , ++ to everyone who made it possible :) 17:17:33 oh also it's worth noting that dnsmasq is in the base layer now https://github.com/coreos/fedora-coreos-tracker/issues/519 17:18:07 and ICYMI: systemd-networkd can be package layered if you really want to: https://github.com/coreos/fedora-coreos-tracker/issues/610#issuecomment-707809280 17:18:26 * dustymabe done 17:18:34 oh actually 17:18:47 we're trying to schedule a test day for the F33 rebase for FCOS 17:18:59 Nov 6th was the tentative date 17:19:38 dustymabe: should i assign an action item for you to work on the hostname issue 17:20:05 and we made a badge for participation in the test day: https://pagure.io/fedora-badges/issue/782 17:20:16 woah nice! 17:20:18 nasirhm: I think I've got it 17:20:21 no need for action item 17:20:23 Do we have our test cases documented for people to follow for the test day ? 17:20:37 dustymabe: Cool :) 17:21:32 nasirhm: not yet, there will be a test day page created in the coming week 17:22:34 great, I would love to work on documenting them for the test case :) 17:24:14 Time for open floor 17:24:26 #topic Open Floor 17:25:50 I've got nothing for open floor I don't think 17:25:54 Anybody got anything to dicuss for the open floor ? 17:25:59 same here 17:26:35 docs 17:27:08 cyberpear: docs ? 17:27:17 we don't ship docs "because we don't ship info or man" -- would it make sense to ship a "tools" container built-in so that we ship docs (in the container) and a way of viewing them (also in the container)? 17:28:11 ship == build and publish to a registry? 17:28:16 cyberpear: "built-in" as in part of the OS image? 17:28:32 I was thinking as part of the image, but ship to a registry would be a good start, too 17:28:48 cyberpear: That sounds straight froward, but I'm wondering what benefit would it provide to add all those docs in the OS (when we're currently not shipping man pages too.) 17:29:10 cyberpear: one thing we had discussed at CoreOS early on was having man pages hosted on a site and then an alternate `man` command which sent the version of the OS in a header and streamed the man page to the user 17:29:14 the default fedora container doesn't include docs either, and even if it did, it wouldn't have rpm-ostree docs since that's not shipped in the Fedora Base contianer 17:29:27 red_beard: i've had similar ideas in the past 17:29:39 then, it's also easy to package that _into_ a container for users to ship 17:30:05 specifically, i'd say that idea goes back to March 2014 - https://github.com/girlpages/ :D 17:30:08 the streamed-man-pages sounds cool, but probably doesn't exist yet? 17:30:15 definitely doesn't exist 17:30:55 not as of yet, basically it needs CI to take the man pages from the builds and push them into a http location (rendered from nroff/troff) 17:31:38 what's the biggest reason we don't ship `man`? is it size of the docs? 17:31:45 and translations 17:32:13 it also pulls in perl 17:32:19 we have tools container (https://src.fedoraproject.org/container/tools/blob/master/f/Dockerfile) that is not really maintained but if someone is interested, it should be straight forward to rebuild and add the docs 17:32:25 man doesn't pull in perl, does it? 17:32:44 cverna: I can take care of it, as I'm currently working on rebuilding some of the containers. 17:33:30 cyberpear: also, you're not really supposed to be running things from the FCOS command line :-) 17:33:31 cyberpear: yes, it does: https://koji.fedoraproject.org/koji/rpminfo?rpmID=23454403 17:34:21 bgilbert: +1 :) 17:34:25 bgilbert: though.. if you're developing against a target environment, I can see running things on the command line in order to figure out what you need to put in an Ignition config as valid 17:34:36 red_beard: it doesn't get pulled when I install it on F33 17:34:45 maybe it's a build requirement? 17:34:45 dustymabe: yeah, I'm +1 to providing a way to get corresponding docs 17:35:03 cyberpear: maybe you have perl already installed 17:35:12 i dunno, that "requires: perl-interpreter" seems pretty explicit 17:35:32 that's on the SRPM though. the binary rpm indeed doesn't: https://koji.fedoraproject.org/koji/rpminfo?rpmID=23228846 17:35:46 not sure how to read the koji page, but rpm -qp --requires shows no perl 17:36:01 jlebon: +1 17:36:02 though doing `dnf remove man-db` in my pet container does want to remove some perl pkgs, so i suspect it gets pulled in by one of its deps 17:36:20 one sec 17:36:53 Added: 17:36:55 gdbm-libs-1:1.18.1-3.fc32.x86_64 17:36:57 groff-base-1.22.3-22.fc32.x86_64 17:36:58 libpipeline-1.5.2-2.fc32.x86_64 17:37:00 man-db-2.9.0-3.fc32.x86_64 17:37:52 not sure we'd want to include, but that's the depchain 17:37:54 i'm very pleased to be incorrect then. 17:37:54 oh neat. either this changed recently, or i was plain wrong :) 17:38:14 we're a bit over, maybe we can continue discussion later? 17:38:38 +1 to discuss later in the #fedora-coreos channel or in the next meeting. 17:38:48 +1 FWIW, i too would really love easily accessible man pages versioned to the OS 17:38:56 those 4 packages add 4.1M to the install AFAICT 17:39:03 happy to end it here, though 17:39:26 are we good to end ? 17:39:31 nasirhm: +1 17:39:42 +1 17:39:46 #endmeeting