16:31:27 <dustymabe> #startmeeting fedora_coreos_meeting
16:31:27 <zodbot> Meeting started Wed Mar 11 16:31:27 2020 UTC.
16:31:27 <zodbot> This meeting is logged and archived in a public location.
16:31:27 <zodbot> The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:31:27 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:31:27 <zodbot> The meeting name has been set to 'fedora_coreos_meeting'
16:31:32 <dustymabe> #topic roll call
16:31:33 <jlebon> .hello2
16:31:34 <zodbot> jlebon: jlebon 'None' <jonathan@jlebon.com>
16:31:36 <jdoss> .hello2
16:31:37 <zodbot> jdoss: jdoss 'Joe Doss' <joe@solidadmin.com>
16:31:54 <slowrie> .hello2
16:31:55 <zodbot> slowrie: slowrie 'Stephen Lowrie' <slowrie@redhat.com>
16:32:42 <miabbott> .hello2
16:32:43 <zodbot> miabbott: miabbott 'Micah Abbott' <miabbott@redhat.com>
16:32:45 <cyberpear> .hello2
16:32:46 <zodbot> cyberpear: cyberpear 'James Cassell' <fedoraproject@cyberpear.com>
16:32:51 <bcotton> .hello2
16:32:52 <zodbot> bcotton: bcotton 'Ben Cotton' <bcotton@redhat.com>
16:32:56 <bgilbert> .hello2
16:32:57 <zodbot> bgilbert: bgilbert 'Benjamin Gilbert' <bgilbert@backtick.net>
16:33:09 <skunkerk> .hello2
16:33:10 <zodbot> skunkerk: Sorry, but you don't exist
16:33:40 <jdoss> Don't let Zodbot get you down skunkerk. We know you exist bud.
16:33:44 <dustymabe> #chair jlebon jdoss slowrie miabbott cyberpear bcotton bgilbert skunkerk
16:33:44 <zodbot> Current chairs: bcotton bgilbert cyberpear dustymabe jdoss jlebon miabbott skunkerk slowrie
16:33:49 <dustymabe> .hello2
16:33:50 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:34:09 <dustymabe> skunkerk: what was your FAS username again ?
16:34:41 <skunkerk> dustymabe, sohank2602
16:34:52 <dustymabe> skunkerk: try `.hello sohank2602`
16:35:08 <skunkerk> .hello sohank2602
16:35:09 <zodbot> skunkerk: sohank2602 'Sohan Kunkerkar' <skunkerk@redhat.com>
16:35:17 <darkmuggle> .hello darkmuggle
16:35:18 <zodbot> darkmuggle: darkmuggle 'None' <darkarts@utlemming.org>
16:35:50 <dustymabe> there you go.. if your IRC nick and FAS username match then `.hello2` is a convenience function so you don't have to type your FAS username
16:36:05 <dustymabe> so you'll have to use `.hello FASUSERNAME`
16:36:22 * jdoss waves at bcotton
16:36:33 <bcotton> hi jdoss!
16:36:48 <dustymabe> #chair darkmuggle
16:36:48 <zodbot> Current chairs: bcotton bgilbert cyberpear darkmuggle dustymabe jdoss jlebon miabbott skunkerk slowrie
16:37:02 <skunkerk> dustymabe, ah, I see. I was blindly following that command. Thanks for the info.
16:37:06 <dustymabe> #topic Action items from last meeting
16:37:21 <dustymabe> looking at the minutes from last meeting looks like we didn't have any action items
16:37:24 <dustymabe> \o/
16:37:31 <dustymabe> we'll dive right into tickets
16:37:36 <dustymabe> looks like there are quite a few this week
16:37:58 <dustymabe> #topic Garbage collection policy for OS releases
16:38:05 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/99
16:38:44 <dustymabe> jlebon: I think this was tagged in for `meeting` by you
16:39:15 <jlebon> ahhh yes
16:39:35 <jlebon> yeah, i think we should start thinking about adding some basic GC
16:39:45 <jlebon> at least for the non-production streams
16:40:01 <dustymabe> +1
16:40:18 <jlebon> the suggestion i had there was pruning those to 60 days
16:40:51 <jlebon> the pool untagging bit will be tricky, but we can do that in a follow-up
16:41:30 <dustymabe> so would we have some process/tool that periodically runs to clean out our s3 buckets ?
16:41:48 <dustymabe> we might already have something implemented somewhere that I don't know about
16:42:17 <jlebon> not sure. either that, or just have the pipeline fire up a job at the end
16:42:21 <cyberpear> For production, I'd say the most aggressive would be to keep the GA release and the final update before EOL of a major version, to align with Fedora. Not saying we should be that aggressive, though.
16:42:48 * dustymabe notes that there is work for pruning the OSTree repo side in https://github.com/coreos/fedora-coreos-releng-automation/pull/79
16:43:11 <jlebon> there's also barriers; we could keep those for longer than others
16:43:27 <cyberpear> How much space is saved by pruning the repo?
16:43:43 <dustymabe> cyberpear: define repo :)
16:43:49 <jlebon> yeah, i think most of the code exists for pruning the various things. we just need some glue code to wire it all up
16:44:17 <dustymabe> there are a lot of things to "prune" - cloud uploaded images - object storage buckets - hosted OSTree repos - koji tags
16:44:58 <jlebon> right
16:45:02 <cyberpear> The ostree repo
16:45:06 <bgilbert> cyberpear: I'd be in favor of strict time-based pruning, if we do prune the production repos
16:45:12 <bgilbert> cyberpear: Fedora releases aren't supposed to matter here
16:45:53 <dustymabe> yeah. for right now I think we can limit the discussion to non-prod because that's where the low hanging fruit is
16:46:00 <jlebon> honestly the amount of data produced by the non-production streams eclipses production ones :)
16:46:12 <cyberpear> bgilbert: maybe not, but I'd still advocate archiving the referenced versions.
16:46:34 <bgilbert> dustymabe: jlebon: yup
16:46:59 <cyberpear> I'm fine with pruning non prod at will
16:47:01 <dustymabe> so jlebon what do you propose for us for "next steps"
16:48:21 <jlebon> dustymabe: (1) investigate what needs to be pruned and (2) write a script to do this for the non-production streams, and (3) decide on lifetime before turning it on
16:48:52 <dustymabe> jlebon: for 1 we can start with the 4 things I mentioned above
16:49:06 <dustymabe> for 3 I think your proposal of 60 days is reasonable
16:49:21 <jlebon> right, just sanity checking that that's all there is (i'm pretty sure it is also :) )
16:49:45 <dustymabe> anyone have any other inputs on the above steps ^^?
16:49:59 <jlebon> i guess with the ostree importer some of this is a bit in flux too
16:50:18 <dustymabe> jlebon: for that piece the fedora-ostree-pruner should handle it
16:50:38 <jlebon> right, yeah
16:50:53 <dustymabe> it uses a "depth based" approach for non-prod but that's roughly equivalent to days
16:51:32 <dustymabe> although I could convert it to time based too
16:51:41 <dustymabe> either way the approximation isn't too far off
16:52:19 <jlebon> i think we could consider not pruning some things too. e.g. we could prune the images of old builds, but not meta.json
16:52:32 <dustymabe> yeah, that would be nice to keep around
16:52:37 <jlebon> but yeah, we can discuss that in the ticket
16:53:00 <dustymabe> the point of pruning is to recoop storage/cost and those should be close to nothing
16:53:04 <dustymabe> ok next topic
16:53:08 <jlebon> (brings up the question of whether a "pruned" build should be reflected somehow in the json)
16:53:44 <dustymabe> probably wouldn't hurt
16:54:31 <dustymabe> #info we agree that we should prune things. we have limited the scope for now to non-production streams of FCOS. jlebon will add more info to #99 about the next steps
16:54:42 <dustymabe> #topic Consider adding kernel command-line argument for automatically logging in on console
16:54:48 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/112
16:55:01 <dustymabe> ok I think I tagged this one in for discussion
16:55:06 <dustymabe> i'
16:56:07 <dustymabe> i've been helping out users quite a bit over the past week and I've found on more than one occasion that it would help our debugging if I could tell the user to add coreos.autologin or similar to the kernel command line so we could debug
16:56:49 <dustymabe> has anyone else had a similar experience?
16:56:55 <cyberpear> Right vs hack the root password...
16:57:33 <dustymabe> cyberpear: right.. it can be easy for one of us to crack open an image and modify it so that it will come up (despite errors), but explaining that to a user can be hard and time consuming
16:58:21 <dustymabe> so the question is: should we re-instate some mechanism for this
16:59:08 <dustymabe> and if yes, are there ideal characteristics of an implementation that would mitigate possible concerns?
16:59:24 <jlebon> there's no denying it's easier than using init=/bin/sh or whatever to set a passwd
17:00:03 * jdoss has to dip into another meeting
17:00:10 <cyberpear> Do the security folks consider the friction as a feature?
17:00:33 <dustymabe> the problem with init=/bin/sh is that doing that doesn't test the actual bringup of the instance
17:00:45 <dustymabe> which is usually what we're trying to debug
17:00:58 <dustymabe> I guess you'd have to do init=/bin/sh change the passwd and then reboot
17:01:00 <jlebon> dustymabe: that's just to set a passwd, right? and then you'd reboot
17:01:04 <jlebon> https://docs.fedoraproject.org/en-US/fedora-coreos/access-recovery/
17:01:07 <dustymabe> right
17:01:21 <bgilbert> we apparently discussed this in January 2019; notes at https://github.com/coreos/fedora-coreos-tracker/issues/112#issuecomment-456683191
17:01:28 <bgilbert> one suggestion from there is to fix `single`
17:01:39 <bgilbert> to not require the root password we don't have
17:02:26 <bgilbert> that seems useful for debugging, while not encouraging people to leave a shell open on the console
17:03:00 <dustymabe> so `single` set passwd, reboot ?
17:03:43 <bgilbert> I was thinking `single` would be sufficient to debug the system
17:03:51 <bgilbert> depends what you're trying to debug, I guess
17:03:57 <dustymabe> but that only brings the system up in single user mode, right?
17:04:41 <bgilbert> if the goal is really to debug full multiuser, I think it probably makes sense to just add the autologin parameter
17:04:57 <jlebon> dustymabe: i take it most of the debugging issues is around passing an ignition config?
17:05:16 <dustymabe> jlebon: typically
17:05:43 <dustymabe> users staring at console login prompt and not being able to get in and they don't know why
17:06:01 <dustymabe> they don't know how to get log output
17:06:19 <dustymabe> all they know is blank screen asking them for a password and ssh core@host doesn't work
17:06:46 <dustymabe> so I'm thinking our method would be:
17:07:01 <dustymabe> - can you provide logs for the bootup? answer is "no, I don't know how"
17:07:22 <dustymabe> - can you reboot the system and provide coreos.autologin=ttyX and then we can debug
17:08:04 <dustymabe> the autologin IMHO should only apply to the one boot where they provided that argument (i.e. not permanent)
17:08:53 <jlebon> i don't think anyone wants it to be permanent :)
17:09:05 <bgilbert> we could have the MOTD, and maybe /etc/issue, complain that autologin is enabled on ttyX
17:09:16 <dustymabe> bgilbert: WFM
17:09:32 <dustymabe> we should be able to rig something up with console-login-helper-messages to display that
17:10:41 <dustymabe> so what's the general sentiment? bad idea, OK idea, good idea ?
17:10:52 * dustymabe randomly calls on slowrie for input
17:10:54 <jlebon> so is the single / init=/bin/sh  to set a pw and reboot is too much overhead?
17:11:27 <jlebon> i mean, i agree it's *more*. but is it workable enough for new users to use?
17:11:38 <dustymabe> i think it's a bit much, just my personal opinion
17:12:14 <dustymabe> it is something you have to reference each time (mainly because of the selinux step)
17:12:25 <dustymabe> which means I have to find the link to the docs and share it each time I want to help someone
17:12:27 <slowrie> dustymabe: don't really have a particular opinion on this
17:12:37 <bgilbert> dustymabe: single solves the SELinux part, no?
17:13:14 <dustymabe> bgilbert: I assume it would
17:14:01 <walters> see also https://github.com/systemd/systemd/issues/7115
17:14:21 <dustymabe> one more pitch for coreos.autologin: it can be quite useful for tutorials/learning materials.. though bad for security
17:14:37 <walters> oh yeah and https://github.com/systemd/systemd/issues/11596#issuecomment-540195956
17:15:13 <bgilbert> walters: we should add that dropin regardless
17:15:27 <dustymabe> I think I already did somewhere
17:15:48 * dustymabe has memory of doing it anyway
17:15:55 <dustymabe> maybe it was part of the installer
17:16:19 <bgilbert> dustymabe: `single` doesn't work as of 31.20200223.2.0
17:16:39 <bgilbert> root account is locked
17:16:41 <jlebon> heh just tried it here too :)
17:16:51 <jlebon> but yeah, we could easily fixed that as you mentioned
17:16:59 <jlebon> i like that it's a known karg
17:17:09 <dustymabe> https://github.com/coreos/coreos-installer/commit/15a79263d0bd5d72056a6080f6687dc10cba2dda
17:17:30 <dustymabe> so yeah, we do it for the installer
17:17:30 <bgilbert> ahhh
17:17:35 <bgilbert> yeah, we should do that generally
17:17:50 <bgilbert> we could fix that first, and then see how the UX is
17:17:58 <bgilbert> before going all the way to the autologin argument
17:18:00 <dustymabe> SGTM
17:18:03 <jlebon> +1
17:18:55 <dustymabe> #info we'll investigate allowing the use of `single` with a locked root account to see if that gives us the ergonomics we want for debugging user's issues
17:19:28 <dustymabe> single doesn't really help us much with the "I'm learning" case I mentioned above, but that is legitimately less important
17:19:43 <cyberpear> Can we push that change to Fedora instead of carrying it?
17:19:57 <cyberpear> (The force change)
17:20:09 <dustymabe> cyberpear: i love applying things to Fedora base instead of carrying it
17:20:23 <dustymabe> cyberpear: would you care to socialize it and get it approved upstream ?
17:20:44 <dustymabe> I unfortunately have too many other fires burning right this moment
17:21:26 * dustymabe apologizes for this topic taking a bunch of time
17:21:36 <jlebon> dustymabe: one could say learning to troubleshoot using single is important :)  teach a man to fish and all that
17:22:18 <dustymabe> ack
17:22:30 <dustymabe> ⏲️ - next topic
17:22:58 <dustymabe> #topic Kubernetes requires conntrack binary
17:23:05 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/404
17:23:17 * dustymabe laughs at the fact that this issue is number 404
17:23:30 <dustymabe> we can push this to next week if we want
17:23:51 <dustymabe> but I figured it's worth bringing up in the context of our recent discussions about usbguard wireguard teaming etc..
17:23:56 <cyberpear> Lol
17:24:05 <jlebon> heh nice
17:24:24 <dustymabe> i.e. is this something we think is important to the host or should we possible punt to the "reliable extensions" solution
17:25:24 <bgilbert> so the reporter is rebuilding the FCOS AMI via Packer
17:25:35 <bgilbert> and injecting the kubelet binary
17:25:43 <dustymabe> bgilbert: yep.. which is another "thing" we should discuss
17:25:46 <bgilbert> it's really tempting to say "then inject conntrack while you're there"
17:25:51 <jlebon> i don't know much about conntrack, how tightly coupled is it to kube versioning-wise?
17:26:08 * dustymabe is in the same boat as jlebon ^^
17:26:08 <bgilbert> i.e. I agree with https://github.com/coreos/fedora-coreos-tracker/issues/404#issuecomment-593840141
17:26:22 <cyberpear> Yes, in that case, inject both using same method
17:27:25 <dustymabe> i.e. we should reply to `In my case I am building an AMI via Packer and copy the kubelet binary to the OS (following the pattern laid out here - https://github.com/awslabs/amazon-eks-ami).` with: "can you inject conntrack in that same step" ?
17:28:30 <bgilbert> dustymabe: +1
17:28:40 <dustymabe> #info punting the conversation on this to a later meeting. will ask user for more clarification for now
17:28:49 <jlebon> hmm ok, reading up more on it, it seems more like a base OS thing that kube just expects to be there
17:29:16 <dustymabe> jlebon: yeah?
17:29:19 <jlebon> so making it an extension could also work, but yeah, that's part of a higher-level discussion about kube
17:29:35 <dustymabe> kk
17:29:41 <dustymabe> we'll punt and revisit this
17:29:45 <jlebon> +1
17:30:08 <dustymabe> #topic open floor
17:30:22 <dustymabe> I'll punt the remaining tickets to next week, but will do a PSA for them here
17:30:34 <bcotton> o/
17:30:47 <dustymabe> We have Red Hat Summit (now a virtual event), devconf.us, and Flock upcoming later this year
17:31:03 <dustymabe> please consider what we (the FCOS community) can do to contribute to these events
17:31:06 <dustymabe> see details in
17:31:13 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/416
17:31:21 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/420
17:31:32 <dustymabe> #link https://github.com/coreos/fedora-coreos-tracker/issues/421
17:31:36 <dustymabe> EOM
17:31:40 * dustymabe waves at bcotton
17:31:56 <jlebon> i will likely go to one or both of those things and probably present something
17:32:06 <bcotton> an update on https://pagure.io/fedora-qa/issue/613 for those who didn't see: i'm working on getting the template created
17:32:29 <bcotton> but i need a better default assignee for bugs that *do* end up in BZ (unless we're agreed that we should bury dusty under tickets)
17:32:39 <dustymabe> #info We have Red Hat Summit (now a virtual event), devconf.us, and Flock upcoming later this year please consider what we (the FCOS community) can do to contribute to these events see details in https://github.com/coreos/fedora-coreos-tracker/issues/416 https://github.com/coreos/fedora-coreos-tracker/issues/420 https://github.com/coreos/fedora-coreos-tracker/issues/421
17:33:16 <dustymabe> bcotton: we should probably have some group email we can send things too, but I don't have any cycles to figure that out right now
17:33:32 <jlebon> bcotton: oh sweet didn't even know we were doing this
17:33:40 <dustymabe> ideally it would be fedora-coreos-bugzilla@fedoraproject.org or something like that and we have a FAS group associated with that email address
17:33:44 <jlebon> can the assignee be an FAS group?
17:33:58 <bcotton> jlebon: the assignee can be any email address with a Bugzilla account
17:34:09 <bcotton> so not a FAS group directly
17:34:20 <jlebon> ack, then what dustymabe said makes sense
17:34:27 <bcotton> (and maybe not indirectly either. i'm not sure if FAS supports a group email, but that would be nice)
17:34:54 <dustymabe> bcotton: I think there are some ways to do it. For example in infra when I get added to a group I all of a sudden start getting a bunch of alert emails
17:35:19 <dustymabe> bcotton: so that would be the ideal scenario, I just don't have any time to try to figure out how to wire that up
17:35:45 <dustymabe> so the options are 1. just assign me 2. help me figure out the group thing and do it that way and put me in the group initially
17:36:17 <bcotton> i can do #2, but in reality it will probably look a lot like #1 for a while :-)
17:36:36 <dustymabe> that's fine. if it's a group I can add some other people to it eventually
17:37:29 <dustymabe> for anyone else who doesn't know: we created a component in bugzilla that tries to route people to github, but also leaves open the possibility that someone might not have a github account and would still like to report a bug there
17:37:47 <dustymabe> so hopefully the traffic in this component is very low volume'
17:37:59 <dustymabe> but we'd still need someone or group to watch it
17:38:21 <dustymabe> ok ⌛
17:38:27 <dustymabe> any more topics for open floor?
17:38:32 <dustymabe> and... bcotton++
17:38:32 <zodbot> dustymabe: Karma for bcotton changed to 14 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:38:37 <dustymabe> thanks so much for sticking with this
17:39:26 <jlebon> support for 4k drives coming in https://github.com/coreos/coreos-assembler/pull/1130  :)
17:39:35 <dustymabe> tangent: let's see if we can give skunkerk his first cookie!
17:39:38 <dustymabe> sohank2602++
17:39:57 <dustymabe> darn you zodbot
17:40:09 <dustymabe> #info support for 4k drives coming in https://github.com/coreos/coreos-assembler/pull/1130
17:40:18 <dustymabe> #endmeeting