16:31:27 #startmeeting fedora_coreos_meeting 16:31:27 Meeting started Wed Mar 11 16:31:27 2020 UTC. 16:31:27 This meeting is logged and archived in a public location. 16:31:27 The chair is dustymabe. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:31:27 Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:31:27 The meeting name has been set to 'fedora_coreos_meeting' 16:31:32 #topic roll call 16:31:33 .hello2 16:31:34 jlebon: jlebon 'None' 16:31:36 .hello2 16:31:37 jdoss: jdoss 'Joe Doss' 16:31:54 .hello2 16:31:55 slowrie: slowrie 'Stephen Lowrie' 16:32:42 .hello2 16:32:43 miabbott: miabbott 'Micah Abbott' 16:32:45 .hello2 16:32:46 cyberpear: cyberpear 'James Cassell' 16:32:51 .hello2 16:32:52 bcotton: bcotton 'Ben Cotton' 16:32:56 .hello2 16:32:57 bgilbert: bgilbert 'Benjamin Gilbert' 16:33:09 .hello2 16:33:10 skunkerk: Sorry, but you don't exist 16:33:40 Don't let Zodbot get you down skunkerk. We know you exist bud. 16:33:44 #chair jlebon jdoss slowrie miabbott cyberpear bcotton bgilbert skunkerk 16:33:44 Current chairs: bcotton bgilbert cyberpear dustymabe jdoss jlebon miabbott skunkerk slowrie 16:33:49 .hello2 16:33:50 dustymabe: dustymabe 'Dusty Mabe' 16:34:09 skunkerk: what was your FAS username again ? 16:34:41 dustymabe, sohank2602 16:34:52 skunkerk: try `.hello sohank2602` 16:35:08 .hello sohank2602 16:35:09 skunkerk: sohank2602 'Sohan Kunkerkar' 16:35:17 .hello darkmuggle 16:35:18 darkmuggle: darkmuggle 'None' 16:35:50 there you go.. if your IRC nick and FAS username match then `.hello2` is a convenience function so you don't have to type your FAS username 16:36:05 so you'll have to use `.hello FASUSERNAME` 16:36:22 * jdoss waves at bcotton 16:36:33 hi jdoss! 16:36:48 #chair darkmuggle 16:36:48 Current chairs: bcotton bgilbert cyberpear darkmuggle dustymabe jdoss jlebon miabbott skunkerk slowrie 16:37:02 dustymabe, ah, I see. I was blindly following that command. Thanks for the info. 16:37:06 #topic Action items from last meeting 16:37:21 looking at the minutes from last meeting looks like we didn't have any action items 16:37:24 \o/ 16:37:31 we'll dive right into tickets 16:37:36 looks like there are quite a few this week 16:37:58 #topic Garbage collection policy for OS releases 16:38:05 #link https://github.com/coreos/fedora-coreos-tracker/issues/99 16:38:44 jlebon: I think this was tagged in for `meeting` by you 16:39:15 ahhh yes 16:39:35 yeah, i think we should start thinking about adding some basic GC 16:39:45 at least for the non-production streams 16:40:01 +1 16:40:18 the suggestion i had there was pruning those to 60 days 16:40:51 the pool untagging bit will be tricky, but we can do that in a follow-up 16:41:30 so would we have some process/tool that periodically runs to clean out our s3 buckets ? 16:41:48 we might already have something implemented somewhere that I don't know about 16:42:17 not sure. either that, or just have the pipeline fire up a job at the end 16:42:21 For production, I'd say the most aggressive would be to keep the GA release and the final update before EOL of a major version, to align with Fedora. Not saying we should be that aggressive, though. 16:42:48 * dustymabe notes that there is work for pruning the OSTree repo side in https://github.com/coreos/fedora-coreos-releng-automation/pull/79 16:43:11 there's also barriers; we could keep those for longer than others 16:43:27 How much space is saved by pruning the repo? 16:43:43 cyberpear: define repo :) 16:43:49 yeah, i think most of the code exists for pruning the various things. we just need some glue code to wire it all up 16:44:17 there are a lot of things to "prune" - cloud uploaded images - object storage buckets - hosted OSTree repos - koji tags 16:44:58 right 16:45:02 The ostree repo 16:45:06 cyberpear: I'd be in favor of strict time-based pruning, if we do prune the production repos 16:45:12 cyberpear: Fedora releases aren't supposed to matter here 16:45:53 yeah. for right now I think we can limit the discussion to non-prod because that's where the low hanging fruit is 16:46:00 honestly the amount of data produced by the non-production streams eclipses production ones :) 16:46:12 bgilbert: maybe not, but I'd still advocate archiving the referenced versions. 16:46:34 dustymabe: jlebon: yup 16:46:59 I'm fine with pruning non prod at will 16:47:01 so jlebon what do you propose for us for "next steps" 16:48:21 dustymabe: (1) investigate what needs to be pruned and (2) write a script to do this for the non-production streams, and (3) decide on lifetime before turning it on 16:48:52 jlebon: for 1 we can start with the 4 things I mentioned above 16:49:06 for 3 I think your proposal of 60 days is reasonable 16:49:21 right, just sanity checking that that's all there is (i'm pretty sure it is also :) ) 16:49:45 anyone have any other inputs on the above steps ^^? 16:49:59 i guess with the ostree importer some of this is a bit in flux too 16:50:18 jlebon: for that piece the fedora-ostree-pruner should handle it 16:50:38 right, yeah 16:50:53 it uses a "depth based" approach for non-prod but that's roughly equivalent to days 16:51:32 although I could convert it to time based too 16:51:41 either way the approximation isn't too far off 16:52:19 i think we could consider not pruning some things too. e.g. we could prune the images of old builds, but not meta.json 16:52:32 yeah, that would be nice to keep around 16:52:37 but yeah, we can discuss that in the ticket 16:53:00 the point of pruning is to recoop storage/cost and those should be close to nothing 16:53:04 ok next topic 16:53:08 (brings up the question of whether a "pruned" build should be reflected somehow in the json) 16:53:44 probably wouldn't hurt 16:54:31 #info we agree that we should prune things. we have limited the scope for now to non-production streams of FCOS. jlebon will add more info to #99 about the next steps 16:54:42 #topic Consider adding kernel command-line argument for automatically logging in on console 16:54:48 #link https://github.com/coreos/fedora-coreos-tracker/issues/112 16:55:01 ok I think I tagged this one in for discussion 16:55:06 i' 16:56:07 i've been helping out users quite a bit over the past week and I've found on more than one occasion that it would help our debugging if I could tell the user to add coreos.autologin or similar to the kernel command line so we could debug 16:56:49 has anyone else had a similar experience? 16:56:55 Right vs hack the root password... 16:57:33 cyberpear: right.. it can be easy for one of us to crack open an image and modify it so that it will come up (despite errors), but explaining that to a user can be hard and time consuming 16:58:21 so the question is: should we re-instate some mechanism for this 16:59:08 and if yes, are there ideal characteristics of an implementation that would mitigate possible concerns? 16:59:24 there's no denying it's easier than using init=/bin/sh or whatever to set a passwd 17:00:03 * jdoss has to dip into another meeting 17:00:10 Do the security folks consider the friction as a feature? 17:00:33 the problem with init=/bin/sh is that doing that doesn't test the actual bringup of the instance 17:00:45 which is usually what we're trying to debug 17:00:58 I guess you'd have to do init=/bin/sh change the passwd and then reboot 17:01:00 dustymabe: that's just to set a passwd, right? and then you'd reboot 17:01:04 https://docs.fedoraproject.org/en-US/fedora-coreos/access-recovery/ 17:01:07 right 17:01:21 we apparently discussed this in January 2019; notes at https://github.com/coreos/fedora-coreos-tracker/issues/112#issuecomment-456683191 17:01:28 one suggestion from there is to fix `single` 17:01:39 to not require the root password we don't have 17:02:26 that seems useful for debugging, while not encouraging people to leave a shell open on the console 17:03:00 so `single` set passwd, reboot ? 17:03:43 I was thinking `single` would be sufficient to debug the system 17:03:51 depends what you're trying to debug, I guess 17:03:57 but that only brings the system up in single user mode, right? 17:04:41 if the goal is really to debug full multiuser, I think it probably makes sense to just add the autologin parameter 17:04:57 dustymabe: i take it most of the debugging issues is around passing an ignition config? 17:05:16 jlebon: typically 17:05:43 users staring at console login prompt and not being able to get in and they don't know why 17:06:01 they don't know how to get log output 17:06:19 all they know is blank screen asking them for a password and ssh core@host doesn't work 17:06:46 so I'm thinking our method would be: 17:07:01 - can you provide logs for the bootup? answer is "no, I don't know how" 17:07:22 - can you reboot the system and provide coreos.autologin=ttyX and then we can debug 17:08:04 the autologin IMHO should only apply to the one boot where they provided that argument (i.e. not permanent) 17:08:53 i don't think anyone wants it to be permanent :) 17:09:05 we could have the MOTD, and maybe /etc/issue, complain that autologin is enabled on ttyX 17:09:16 bgilbert: WFM 17:09:32 we should be able to rig something up with console-login-helper-messages to display that 17:10:41 so what's the general sentiment? bad idea, OK idea, good idea ? 17:10:52 * dustymabe randomly calls on slowrie for input 17:10:54 so is the single / init=/bin/sh to set a pw and reboot is too much overhead? 17:11:27 i mean, i agree it's *more*. but is it workable enough for new users to use? 17:11:38 i think it's a bit much, just my personal opinion 17:12:14 it is something you have to reference each time (mainly because of the selinux step) 17:12:25 which means I have to find the link to the docs and share it each time I want to help someone 17:12:27 dustymabe: don't really have a particular opinion on this 17:12:37 dustymabe: single solves the SELinux part, no? 17:13:14 bgilbert: I assume it would 17:14:01 see also https://github.com/systemd/systemd/issues/7115 17:14:21 one more pitch for coreos.autologin: it can be quite useful for tutorials/learning materials.. though bad for security 17:14:37 oh yeah and https://github.com/systemd/systemd/issues/11596#issuecomment-540195956 17:15:13 walters: we should add that dropin regardless 17:15:27 I think I already did somewhere 17:15:48 * dustymabe has memory of doing it anyway 17:15:55 maybe it was part of the installer 17:16:19 dustymabe: `single` doesn't work as of 31.20200223.2.0 17:16:39 root account is locked 17:16:41 heh just tried it here too :) 17:16:51 but yeah, we could easily fixed that as you mentioned 17:16:59 i like that it's a known karg 17:17:09 https://github.com/coreos/coreos-installer/commit/15a79263d0bd5d72056a6080f6687dc10cba2dda 17:17:30 so yeah, we do it for the installer 17:17:30 ahhh 17:17:35 yeah, we should do that generally 17:17:50 we could fix that first, and then see how the UX is 17:17:58 before going all the way to the autologin argument 17:18:00 SGTM 17:18:03 +1 17:18:55 #info we'll investigate allowing the use of `single` with a locked root account to see if that gives us the ergonomics we want for debugging user's issues 17:19:28 single doesn't really help us much with the "I'm learning" case I mentioned above, but that is legitimately less important 17:19:43 Can we push that change to Fedora instead of carrying it? 17:19:57 (The force change) 17:20:09 cyberpear: i love applying things to Fedora base instead of carrying it 17:20:23 cyberpear: would you care to socialize it and get it approved upstream ? 17:20:44 I unfortunately have too many other fires burning right this moment 17:21:26 * dustymabe apologizes for this topic taking a bunch of time 17:21:36 dustymabe: one could say learning to troubleshoot using single is important :) teach a man to fish and all that 17:22:18 ack 17:22:30 ⏲️ - next topic 17:22:58 #topic Kubernetes requires conntrack binary 17:23:05 #link https://github.com/coreos/fedora-coreos-tracker/issues/404 17:23:17 * dustymabe laughs at the fact that this issue is number 404 17:23:30 we can push this to next week if we want 17:23:51 but I figured it's worth bringing up in the context of our recent discussions about usbguard wireguard teaming etc.. 17:23:56 Lol 17:24:05 heh nice 17:24:24 i.e. is this something we think is important to the host or should we possible punt to the "reliable extensions" solution 17:25:24 so the reporter is rebuilding the FCOS AMI via Packer 17:25:35 and injecting the kubelet binary 17:25:43 bgilbert: yep.. which is another "thing" we should discuss 17:25:46 it's really tempting to say "then inject conntrack while you're there" 17:25:51 i don't know much about conntrack, how tightly coupled is it to kube versioning-wise? 17:26:08 * dustymabe is in the same boat as jlebon ^^ 17:26:08 i.e. I agree with https://github.com/coreos/fedora-coreos-tracker/issues/404#issuecomment-593840141 17:26:22 Yes, in that case, inject both using same method 17:27:25 i.e. we should reply to `In my case I am building an AMI via Packer and copy the kubelet binary to the OS (following the pattern laid out here - https://github.com/awslabs/amazon-eks-ami).` with: "can you inject conntrack in that same step" ? 17:28:30 dustymabe: +1 17:28:40 #info punting the conversation on this to a later meeting. will ask user for more clarification for now 17:28:49 hmm ok, reading up more on it, it seems more like a base OS thing that kube just expects to be there 17:29:16 jlebon: yeah? 17:29:19 so making it an extension could also work, but yeah, that's part of a higher-level discussion about kube 17:29:35 kk 17:29:41 we'll punt and revisit this 17:29:45 +1 17:30:08 #topic open floor 17:30:22 I'll punt the remaining tickets to next week, but will do a PSA for them here 17:30:34 o/ 17:30:47 We have Red Hat Summit (now a virtual event), devconf.us, and Flock upcoming later this year 17:31:03 please consider what we (the FCOS community) can do to contribute to these events 17:31:06 see details in 17:31:13 #link https://github.com/coreos/fedora-coreos-tracker/issues/416 17:31:21 #link https://github.com/coreos/fedora-coreos-tracker/issues/420 17:31:32 #link https://github.com/coreos/fedora-coreos-tracker/issues/421 17:31:36 EOM 17:31:40 * dustymabe waves at bcotton 17:31:56 i will likely go to one or both of those things and probably present something 17:32:06 an update on https://pagure.io/fedora-qa/issue/613 for those who didn't see: i'm working on getting the template created 17:32:29 but i need a better default assignee for bugs that *do* end up in BZ (unless we're agreed that we should bury dusty under tickets) 17:32:39 #info We have Red Hat Summit (now a virtual event), devconf.us, and Flock upcoming later this year please consider what we (the FCOS community) can do to contribute to these events see details in https://github.com/coreos/fedora-coreos-tracker/issues/416 https://github.com/coreos/fedora-coreos-tracker/issues/420 https://github.com/coreos/fedora-coreos-tracker/issues/421 17:33:16 bcotton: we should probably have some group email we can send things too, but I don't have any cycles to figure that out right now 17:33:32 bcotton: oh sweet didn't even know we were doing this 17:33:40 ideally it would be fedora-coreos-bugzilla@fedoraproject.org or something like that and we have a FAS group associated with that email address 17:33:44 can the assignee be an FAS group? 17:33:58 jlebon: the assignee can be any email address with a Bugzilla account 17:34:09 so not a FAS group directly 17:34:20 ack, then what dustymabe said makes sense 17:34:27 (and maybe not indirectly either. i'm not sure if FAS supports a group email, but that would be nice) 17:34:54 bcotton: I think there are some ways to do it. For example in infra when I get added to a group I all of a sudden start getting a bunch of alert emails 17:35:19 bcotton: so that would be the ideal scenario, I just don't have any time to try to figure out how to wire that up 17:35:45 so the options are 1. just assign me 2. help me figure out the group thing and do it that way and put me in the group initially 17:36:17 i can do #2, but in reality it will probably look a lot like #1 for a while :-) 17:36:36 that's fine. if it's a group I can add some other people to it eventually 17:37:29 for anyone else who doesn't know: we created a component in bugzilla that tries to route people to github, but also leaves open the possibility that someone might not have a github account and would still like to report a bug there 17:37:47 so hopefully the traffic in this component is very low volume' 17:37:59 but we'd still need someone or group to watch it 17:38:21 ok ⌛ 17:38:27 any more topics for open floor? 17:38:32 and... bcotton++ 17:38:32 dustymabe: Karma for bcotton changed to 14 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 17:38:37 thanks so much for sticking with this 17:39:26 support for 4k drives coming in https://github.com/coreos/coreos-assembler/pull/1130 :) 17:39:35 tangent: let's see if we can give skunkerk his first cookie! 17:39:38 sohank2602++ 17:39:57 darn you zodbot 17:40:09 #info support for 4k drives coming in https://github.com/coreos/coreos-assembler/pull/1130 17:40:18 #endmeeting