00:00:39 #startmeeting F38 Final Go/No-Go meeting continued 00:00:39 Meeting started Fri Apr 14 00:00:39 2023 UTC. 00:00:39 This meeting is logged and archived in a public location. 00:00:39 The chair is adamw. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 00:00:39 Useful Commands: #action #agreed #halp #info #idea #link #topic. 00:00:39 The meeting name has been set to 'f38_final_go/no-go_meeting_continued' 00:00:49 #topic Roll call 00:00:54 who's still awake 00:01:06 hi 00:01:14 .hi 00:01:15 pboy: pboy 'Peter Boy' 00:01:16 hi pboy 00:01:31 .hi 00:01:32 farribeiro: farribeiro 'Fábio Ribeiro' 00:01:49 morning! 00:02:00 amoloney: you made it! 00:02:02 #chair amoloney 00:02:02 Current chairs: adamw amoloney 00:02:06 I sure did :) 00:02:06 handing over chair duties 00:02:07 morning 00:02:23 .hello geraldosimiao 00:02:24 geraldosimiao: geraldosimiao 'Geraldo S. Simião Kutz' 00:02:31 'light' chair duties please, Im still learning! 00:02:37 here is 21h still 00:02:41 =) 00:02:48 evening then :) 00:03:08 so is there a need to go back to test matrices? 00:03:20 or where best to begin for reconvened meetings? 00:03:44 well first I'll info I suppose 00:03:45 drinking? sorry... 00:03:53 hi! 00:04:10 i guess we can take a quick trip through the candidate and test matrix topics to update status 00:04:18 this will give us more time to test stuff :P 00:04:54 #info the F28 Go/No Go meeting was paused temporarily while a new RC was staged. We will start with the testing matrices from this point as blockers remain unchanged 00:05:07 F38 00:05:11 not 28 00:05:49 ha whoops 00:05:51 you can #undo and try again 00:05:52 F8 00:06:14 dang it Im having a mare 00:06:52 Im just copy pasting that info again with an edit if thats an ok shortcut 00:07:57 #info the previous info had a typo. The correct version of Fedora is F38 and the F38 Go/No Go meeting was paused temporarily while a new RC was staged. We will start with the testing matrices from this point as blockers remain unchanged 00:08:58 #topic Current Status - Test Matrices 00:09:16 #link https://fedoraproject.org/wiki/Category:Fedora_38_Test_Results 00:09:24 so, obviously, we're not exactly full for RC-1.6 yet 00:09:28 let me put things on the record... 00:10:03 #info RC-1.6 is essentially unchanged from RC-1.5: the contents are the same, we just re-ran the compose to avoid the network issues that caused several images to fail in RC-1.5 00:10:28 #info for this reason we will consider only 'smoke testing' of RC-1.6 necessary - that's testing the release-blocking images to be sure they boot and deploy normally 00:11:13 we are currently in progress on that testing 00:11:27 * coremodule is here 00:11:35 geraldo and I are working on workstation and kde, tflink is working on server, coremodule is working on arm images...where's everyone at? 00:12:02 workstation live is good on bios and uefi 00:12:10 i'm just finishing up the kde bios test 00:12:24 and cheksumm is ok too, for workstation and kde 00:12:35 and sway 00:12:38 Server x86_64 is good on previous KVM bios, hardware uefi 00:12:54 working with server, but not nvme, biosboot 00:13:18 worth noting that rc-1.6 only had silverblue aarch64 install image missing. This has been broken a while and we are not sure why. All other artifacts are included. 00:13:24 will do everything iso now 00:13:37 tflink including nvme issue? 00:14:36 i'm grabbing the everything boot iso now 00:16:10 in case it's not obvious, we are now spinning out the meeting while we do all the testing. :D 00:16:10 pboy: yep, did the same thing on the same machine when I first hit the nvme issue and didn't hit it 00:17:22 thanks! I'm happy. :-) 00:21:16 running bios install of everything netinst now 00:21:21 okay, all blocking aarch64 deliverables boot 00:21:26 woohoo 00:21:32 coremodule: wanna go onto cloud images? 00:22:01 did you test the uefi boot of isos as well as disk images? 00:22:05 let me see what I can do, I haven't booted a cloud image in a while... 00:22:09 not yet, trying uefi now 00:22:21 ok, you do the uefi thing then, i'll get onto cloud 00:22:25 rog 00:22:29 I'll test basic video driver at bios, kde 00:22:40 coremodule: I'm around if you have difficulty. 00:23:00 thanks davdunc[m 00:23:10 ah. adamw ditto for you. 00:23:39 davdunc (@davdunc:fedora.im): okay, since you're here: why can i never ssh into newly-created instances until i do some random fiddling with the security policies? :D 00:23:48 whichever policy i choose to apply to a newly-created instance, i can't ssh into it 00:23:59 i have to create it, then apply a *second* policy to it, before i can get in 00:24:12 hmm. like a change in the security group? 00:24:15 yeah 00:24:24 it doesn't matter if i use an existing one or let the wizard create one for me 00:24:37 i have a script that I use in ansible to generate the security group at the same time that I build the instances. 00:24:47 i just use the webui 00:25:17 ah. yea the only way to make that work from everywhere is to make it available from 0.0.0.0/0 00:25:24 and that's really not advised. 00:25:27 that's what one of the policies i have set does, i think 00:25:39 but if i use that one, i still can't ssh till i add another (more restrictive) one 00:25:54 let me whip up a security group modifier from the command line. 00:26:07 i mean, i know how to workaround it now, it's just...weird. :D 00:26:51 yea. 00:27:02 XFCE checksum is ok 00:27:05 unreachable by default. :D 00:27:15 so yeah, i have a security group called 'default' which is set to allow 'all traffic', 'all' protocols, 'all' ports, inbound and outbound 00:27:24 i created a new instance and applied that group to it 00:27:26 and i can't ssh into it 00:27:48 https://paste.centos.org/view/38675a68 00:28:23 well, you may be running into the same cloud-init bug I just hit. 00:28:48 now i add a group to it called launch-wizard-1, which only allows ssh 00:28:50 and now i can ssh in 00:29:12 but if i do it the other way round, same result (create the instance with 'launch-wizard-1', can't log in; add 'default', can log in) 00:29:13 aha. 00:29:59 there's something wrong with that default group. Either the protocol is defined incorrectly or there's something weird there. 00:30:13 it might be associated with the local network. 00:30:32 but then why does adding that group to the launch-wizard-1 group make it work when i do things that way around? 00:30:35 oh well, it's just weird. 00:30:45 adamw: have you done the EC2 tests? 00:30:50 tflink (@tflink:fedora.im): i'm working on them 00:30:51 I'll have to look at it. 00:31:06 davdunc (@davdunc:fedora.im): yeah, i can bug you agbout it later 00:31:11 k, I won't start them then 00:31:14 just thought it might be a Known Issue or something 00:31:23 tflink (@tflink:fedora.im): well, you can do the aarch64 ones? 00:31:24 i'm on x86_64 00:31:51 do we upload aarch64 amis? 00:32:14 yeah 00:32:17 yep 00:32:20 they're on the cloud page, but collapsed by default 00:32:31 expand the header "arm64 hvm standard AMIs" 00:32:52 ah, for some reason, I read that as empty rather than collapsed 00:32:54 (if we expand all the lists by default the page gets huge) 00:32:58 starting ec2 aarch64 00:36:51 basic video driver at KDE / bios is OK 00:39:00 everything netinst is good both bios and uefi 00:39:03 doing server netinst now 00:39:35 man, this is taking me back to the days when i actually tested stuff 00:40:00 cloud tests on x86_64 are all good 00:40:03 😁 00:40:58 will do workstation bios basic video driver now 00:41:02 I test serever x84 dvd biosboot hardware and netinstall on uefi hardware, both are fine 00:41:25 pboy: can you fill in the server dvd usb bios block on the matrix then? that one's missing 00:41:40 is there a default username/password for the cloud AMI? something that would work over serial console? 00:41:48 tflink (@tflink:fedora.im): i just ssh in 00:42:00 register an ssh keypair with ec2 and assign it to the instance then you can ssh in 00:42:22 yeah, I know how ec2 works. this instance isn't responding to ssh and I wanted to poke at it via serial console 00:42:26 oh hah 00:42:31 sorry, dunno, that'd be for davdunc 00:42:39 i don't think there's a sekrit backdoor though 00:42:42 adamw I'm still struggeling with that matrix, unfortunately. ashes on my head 00:42:53 no. The serial console doesn't work. 00:42:53 pboy: `dnf -y install relval`, `relval report-results` 00:43:09 tflink: you can get the system log though. 00:43:20 starting with server aarch64 SBC image, or did someone already test it? 00:43:20 davdunc: tried that, it's a blank 00:43:35 * nirik has used the serial console just fine... but the default image has no root pw set of course. 00:43:48 yeah, the console is working, I just can't log in 00:43:54 nirik: on aarch64? 00:44:08 I know it can work on the x86_64 00:44:08 oh, perhaps not. 00:44:21 yea. I don't think it will work on arm. 00:44:33 let me look into that. 00:45:05 we can set the password in cloud-init 00:45:30 pboy: coremodule already did i think, confirmation is always great of course 00:45:54 i think all we have left to knock out is server x86_64 netinst (i'm working on that) and aarch64 cloud 00:46:01 aarch64 server dvd boots and installs with uefi 00:46:12 yay 00:46:13 working on aarch64 server netinst uefi now... 00:46:27 davdunc: I have a responding serial console using t4g.medium, not sure if that's what you meant by working serial console, though 00:47:13 tflink: that's good enough for me. 00:47:22 * nirik waits for the testing to complete 🤞 00:47:53 i think we should be good in another ~15-20 mins 00:48:44 the openqa bot seems to be doing pretty well... knocking out lots of things there. 00:48:54 does anyone know how to set a root password @ creation time for an ec2 instance? 00:49:03 media writer is fine, for F38 and F37 00:49:10 or would I have to create a custom AMI 00:49:18 FYI, content is now fully synced out to alt/stage/ 00:49:26 tflink: using cloud-config.txt, you can set it. 00:50:42 here's an issue off the web that describes it https://github.com/vmware/photon/issues/659#issuecomment-769450194 00:51:23 doing server dvd checksum 00:53:56 ok, i'm onto server netinst uefi 00:54:24 I'll do server DVD bios 00:54:29 usb bios 00:55:32 geraldosimiao server dvd on biosboot x86 is working here 00:55:54 oh, goo. then I'll test other thing 00:55:58 aborting 00:56:41 *good 00:57:17 Just notice, it's 2 am here in Europe. cool action :) 00:57:37 pboy++ thanks for testing things! 00:57:39 nirik: Karma for pboy changed to 2 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 00:57:56 oh thanks 00:58:20 * mattdm will check back later :) 00:58:28 pboy++ 00:58:28 geraldosimiao: Karma for pboy changed to 3 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 01:00:30 * nirik will be back in a few min. 01:00:57 server netinst uefi install running 01:01:08 the aarch64 server netinst will take a long time, fyi 01:01:16 like an hour give or take 01:01:50 i3 doing fine too 01:02:24 server aarch raw.xz works fine, too 01:02:36 coremodule: you know, if it made it to the install phase, we can probably say it's good 01:02:52 pboy can I fill the test matrix with your username, at server dvd biosboot success? 01:03:12 adamw, works for me. I'm not there yet (image still writing), but I'll let oyu know 01:03:13 yes please ! I have to learn it 01:03:24 ok, got the ec2 ami communicating. finishing up the other basic tests 01:03:26 pboy, which aarch64 hardware are you using? 01:03:27 ec2 aarch64 ami 01:03:48 rockpro64 an d rockpi 4 01:03:55 pboy, cool! 01:04:03 ok, server netinst x86 is all good 01:04:29 I lied when I said aarch64 server dvd uefi was done. 01:04:41 It was *almost* done, and I thought it would just be a little longer. 01:04:55 It's still chugging away. No problems, just takes for fricken ever 01:05:35 heh 01:07:42 I'll do checksum cloud images 01:11:45 * nirik is back 01:13:20 alrighty, where we at 01:14:03 we're waiting on the aarch64 cloud tests and aarch64 uefi tests i believe 01:14:11 that should be it 01:14:44 openqa seems to have finished or softfailed all the x86_64 tests... still churning on the aarch64 ones 01:15:19 almost done with the aarch64 ami test 01:16:06 yeah, openqa aarch64 takes a while, i wouldn't wait on it 01:18:17 fair enough 01:18:55 checksums are good 01:19:38 installing budgie now, just in case 01:20:24 ec2 aarch64 passed 01:21:21 yay 01:22:08 so just aarch64 uefi? 01:23:12 yup 01:23:28 isn't coremodule working on that? 01:24:08 yeah, it takes forever 01:24:29 I'm almost(?) done with aarch64 server dvd uefi 01:25:02 disks are burned for netinst, so I'll get it to the installer and then report back, we can get the meeting going and I'll just let it install while we do. 01:25:31 yeah, i think if server dvd completes and netinst reaches install phase, we're good 01:26:19 🎉 01:26:38 * tflink[m] is poking at a form of win11 dual-boot and mac fmw but neither of those are blocking 01:27:09 coremodule: I'll fill an "inprogress" with your name at the test matrix 01:27:32 for aarch64 usb uefi server dvd 01:28:45 thanks geraldosimiao 01:28:59 oh, good, robatino double checked the checksums too. 01:29:43 ho did aarch64 usb uefi server netinst ? 01:30:09 geraldosimiao, me, not there yet 01:30:11 geraldosimiao: i think that coremodule is about to start that 01:30:24 ok, no problem 01:31:09 #info at this point smoke testing of rc-1.6 is almost complete with no problems appearing 01:32:09 Adam, the vms you build, the host was F38 or F37? 01:32:35 we dont have "current KVM" filled at virtualization matrix 01:34:13 geraldosimiao I installed libvirt and our Server KVM without issue, if that helps 01:35:29 geraldosimiao: openqa effectively crosses that off, i guess, since it's running f37. 01:35:42 or wait, does 'current' mean 'the same release' in this context 01:35:43 ok 01:35:43 anyway, meh 01:35:52 current means f38 01:35:53 i'm running f38 on my laptop and i can start vms, so it's fine. :P 01:35:57 hold on a sec 01:35:59 so did we do it? 01:36:20 .hello ngompa 01:36:24 Eighth_Doctor: ngompa 'Neal Gompa' 01:36:34 hey Conan Kudo 01:36:43 i'll run an install in a vm while we wait for coremodule. :P 01:36:48 oof 01:36:51 almost there 01:36:51 Conan Kudo: we're just finishing up testing, but it looks good 01:36:57 yay 01:37:01 dvd works, plugging in the netinst drives now 01:37:20 I threw LXQt up and it worked okay 01:37:28 or at least not worse than the previous compose 01:38:14 here: Prepare a machine with the current development release (Fedora 38) installed 01:38:14 Install Package-x-generic-16.pngvirt-manager (or other tool to control KVM virtual machines) 01:38:14 If you want to test your guest system using UEFI, you need to set it up first.( Note: IoT testing requires UEFI.) 01:38:38 adamw: I filled with your name there 01:38:51 thanks 01:40:22 * nirik waits for the USS 🛳️it so he can go have some dinner. ;) 01:40:57 seems Weston is installed in the LXQt image still, which makes the installed environment default to it 01:40:59 that's fun 01:42:01 * Eighth_Doctor is checking to see if that was the case in F37 now 01:42:14 Fedora-Budgie-Live-x86_64-38-1.6.iso installed and booted fine 01:42:59 weston, wayland...all towns in the same state at least. :) 01:43:09 hehe 01:43:24 see you're already using the "emoji version" at words nirik lol 01:43:33 okay, aarch64 uefi server netinst makes it to the installer, no issues. 01:43:55 Yeah, as I get more 😫 I start using more. ;) 01:44:16 Fedora-Server-dvd-x86_64-38-🥱.6.iso 01:45:57 current kvm is just fine 01:46:22 alright 01:46:25 i think we can say we're good 01:46:35 #info Sufficient smoke testing of RC-1.6 has been completed without any problems 01:46:43 hurray! 01:47:20 #topic Current Status - RC 01:47:37 #info RC-1.6 has all images except aarch64 silverblue, and no known blockers 01:48:05 is it time to poll? 01:48:18 oh lol, I think sddm is running in weston on lxqt 38 01:48:28 well, next topic 01:48:54 exciting! Thats the decision one :) 01:49:05 we do need to hit on IoT and CoreoS 01:49:06 #topic Go/No Go decision 01:49:07 which i forgot earlier 01:49:11 so, before we vote 01:49:17 pbrobinson: not around, are you? 01:49:24 I can speak for IoT 01:49:39 for pbrobinson, pwhalen 01:49:53 i think technically we should wait for the next IoT compose and ship that, right?\ 01:49:59 if lorax speaks for the trees, who speaks for the ostrees? :) 01:49:59 since we only just pushed the last packages stable today 01:50:08 We wanted 20230413.0 01:50:12 We are on 20230413.4 01:50:20 and given https://pagure.io/releng/issue/11099 we might actually need to do *two* composes 01:50:26 👋 01:50:28 ...or four! four is also good! 01:50:28 O.o 01:50:31 that's... weird 01:51:08 I've been messing with 20230413.4 since it composed (~1.5hrs), think we're good to go with that particular image. 01:51:28 coremodule: does it have all the stuff we pushed stable earlier? 01:51:42 https://pagure.io/releng/issue/11306#comment-851442 ? 01:52:14 Good question, let me see... 01:52:27 i can speak for FCOS 01:53:25 for fcos i assume we're fine, since i asked and nobody said there were any problems :D 01:53:53 the `next` we shipped yesterday to end users was pretty close to the final package set 01:54:02 adamw, I don't think any of those changes apply to IoT anyway... 01:54:11 f38 backgrounds, definitely not 01:54:33 anaconda would be in there 01:54:35 not sure about fuse3 01:54:39 yeah, good point 01:54:54 the most recent `next-devel` compose includes the updates from https://pagure.io/releng/issue/11306#comment-851164 but not the ones from the most recent push https://pagure.io/releng/issue/11306#comment-851442 01:55:09 im booting an image now. I know the most release compose was made like an hour and a half ago. Don't know for certain if those packages were explicitly pulled in. trying to check now 01:55:09 so I think at this point we're just missing the fuse3 update 01:55:28 it has the older anaconda, according to https://kojipkgs.fedoraproject.org/compose/iot/Fedora-IoT-38-20230413.4/logs/x86_64/IoT/ostree_installer-2/pylorax.log 01:55:47 but I don't anticipate any problems there? 01:56:06 dustymabe: well, the fuse3 update is for a container issue, so it's probably relevant 01:56:13 adamw: indeed 01:56:20 it'd be good to make sure that when we cut fcos over to f38 'officially', it has the fuse3 update 01:56:44 there be a new `next-devel` in the morning and we'll probably cut a new `next` tomorrow and ship it out to our users 01:57:00 that same content will get promoted to `testing` on Tuesday 01:57:13 the anaconda change is for fixing crash reporting when packaging.log is empty, which actually probably does apply to both coreos and iot 01:57:16 so our users will have 3 days or so to report any major issues :) 01:57:24 rgr 01:57:27 sounds like fcos is fine 01:57:31 * dustymabe notes coreos doesn't use anaconda 01:57:41 oh, right, i always forget we don't have an anaconda installer image for it. 01:57:54 so, fcos has a plan 01:58:04 adamw: we always have a plan! 01:58:12 for iot we probably want to try and make sure we get a compose with new fuse3 in the ostree and new anaconda on the installer image 01:58:15 heh 01:58:58 coremodule: do you see any problems making sure we have an iot compose ready by, say, tomorrow? 01:59:11 who does that have to be synced up with? releng? 01:59:34 yeah, good question. I think pwhalen is handling the image builds these days... 02:00:17 nirik (@nirik:libera.chat): can you kick off iot composes? 02:01:20 I've not ever done so, dunno how off hand... 02:01:30 oh, fun 02:01:36 it's using osbuild 02:01:50 huh, there's one running now? 02:01:51 i assumed it was just a slightly different pungi command or whatever 02:01:56 there is? well that's...good? 02:01:56 https://koji.fedoraproject.org/koji/taskinfo?taskID=99907669 02:02:17 oh, the .4 one? that's the one i was looking at 02:02:21 it still has the older anaconda in it 02:02:47 yeah, thats 38.20230413.4 apparently. 02:02:54 yeah, pwhalen kicked that off maybe 2 hours ago 02:03:01 meanwhile, we're gonna hit 100 million koji tasks soon. yikes. 02:03:37 adamw: we've been having some network issues with osbuild. I'll make sure we pull in the new packages for the official compose devel->stable 02:03:53 huh. it used the newer fuse3 but the older anaconda. why 02:04:00 okay 02:04:12 pwhalen: we just need to make sure the timeframe is workable 02:04:26 aiui we need releng to know what the Final Compose for iot is for staging tomorrow, is that right nirik? 02:05:01 Right, planning a sync with pbrobinson tomorrow to discuss. I think our is handled separately 02:05:23 yeah, I think it's sperate since it has it's own ostree dir/tree... 02:05:28 well, it all still has to be staged and synced out and ready to go on tuesday, right? 02:05:45 understood 02:05:45 fcos follows its own schedule to an extent but aiui we expect the iot bits ready to go same day as regular 02:05:54 ideally for sure. 02:06:31 okay, well...it's not ideal that we're always kinda leaving this part to the last minute, but i guess it's probably okay to trust we can pull it together 02:06:37 so, those sorted? move on to decision process? 02:07:16 this is more last minutely than we have had in a while. ;) 02:07:54 yeah, let's just sign this damn thing off, i need dinner 02:08:09 #topic Polling 02:08:10 * dustymabe notes we strive hard to ship our `testing` stream with the GA content set 02:08:16 on GA day 02:08:17 per the QA policy, there are no blockers, test coverage is sufficient, we're not looking too hard at IoT, therefore QA votes go 02:08:33 for rc-1.6, that is 02:08:38 Ack 02:08:41 FESCo? 02:08:55 go go go 02:09:00 Ack 02:09:03 RelEng? 02:09:07 * nirik swaps another hat on 02:09:11 releng is go 02:09:15 Ack 02:09:31 #agreed Fedora Linux 38 Final is GO 02:09:44 we... did it? 02:09:46 #info Fedora Linux 38 Final will release on the early target date (2022-04-17) 02:09:46 it's all niriks, all the way down 02:09:54 Eeek!! 02:09:55 Hurray! 02:09:56 hurray! 02:09:59 we did! and with absolutely strict following of all policies and procedures, no less 02:10:00 18-04 02:10:05 * adamw keeps straight face 02:10:05 oh my! 02:10:08 not 17-04 02:10:14 good grief 02:10:17 April 18 02:10:20 not 17 02:10:23 yeah 02:10:24 #undo 02:10:24 Removing item from minutes: INFO by amoloney at 02:09:46 : Fedora Linux 38 Final will release on the early target date (2022-04-17) 02:10:28 try again :) 02:10:35 thank you :) 02:10:44 this would technically be right yesterday... 02:10:57 if someone wants to LGTM this FCOS fast-track for fuse3 I can get our processes started a bit sooner: https://github.com/coreos/fedora-coreos-config/pull/2370 02:11:21 #info Fedora Linux 38 Final will release on the early target date (2022-04-18) 02:11:30 FTR - CoreOS is 👍 for GO 02:11:39 🎉 02:11:43 hurray for go 02:11:49 go!!!!! 02:11:56 #action amoloney to announce decision 02:12:03 🚀 02:12:04 #topic Open Floor 02:12:05 🎉 02:12:24