15:01:18 <adamw> #startmeeting Fedora QA meeting
15:01:18 <zodbot> Meeting started Mon Jul 12 15:01:18 2021 UTC.
15:01:18 <zodbot> This meeting is logged and archived in a public location.
15:01:18 <zodbot> The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:18 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:01:18 <zodbot> The meeting name has been set to 'fedora_qa_meeting'
15:01:27 <adamw> #meetingname fedora-qa
15:01:27 <zodbot> The meeting name has been set to 'fedora-qa'
15:01:28 <jforbes> kind of paying attention, I have a conflict at this time
15:01:32 <adamw> #topic Roll Call
15:01:44 <tablepc> .hello2
15:01:45 <zodbot> tablepc: tablepc 'Pat Kelly' <pmkellly@frontier.com>
15:01:46 <bittin> .hello2
15:01:48 <zodbot> bittin: bittin 'Luna Jernberg' <droidbittin@gmail.com>
15:02:09 <coremodule> good morning!
15:02:12 * coremodule is here.
15:02:30 <adamw> morning folks, how's everyone doing
15:02:38 <cmurf> .hi
15:02:39 <zodbot> cmurf: Sorry, but you don't exist
15:02:40 <bittin> i am doing alright
15:02:45 <adamw> hope everyone noticed I got the IRC network right this time
15:02:46 <cmurf> still?
15:03:01 <cmurf> .hi chrismurphy
15:03:01 <bittin> adamw: i did not read good enough, but thats great
15:03:01 <zodbot> cmurf: Sorry, but you don't exist
15:03:01 <tablepc> It''s evening someplaces
15:03:04 <cmurf> LOL
15:03:09 <cmurf> .hello
15:03:09 <zodbot> cmurf: (hello <an alias, 1 argument>) -- Alias for "hellomynameis $1".
15:03:16 <cmurf> .hello chrismurphy
15:03:17 <zodbot> cmurf: chrismurphy 'Chris Murphy' <bugzilla@colorremedies.com>
15:03:49 <tablepc> Yes, new improved e'mail
15:03:50 <cmurf> ok well when you try all the other options...
15:04:25 <adamw> .ahoyhoy
15:04:32 <adamw> someone should hook that one up
15:04:48 <frantisekz> .hello2
15:04:49 <zodbot> frantisekz: frantisekz 'František Zatloukal' <fzatlouk@redhat.com>
15:05:27 <lruzicka2> olah
15:05:30 <adamw> okay, let's get through this
15:05:35 <lruzicka2> .hello lruzicka
15:05:35 <adamw> #topic Previous meeting follow-up
15:05:35 <zodbot> lruzicka2: lruzicka 'Lukáš Růžička' <lruzicka@redhat.com>
15:05:50 <tablepc> Will it be difficult?
15:06:12 * cmurf saw adamw with a pliars earlier...
15:06:54 * kparal is here
15:06:58 <tablepc> That good be a problem
15:07:17 <bittin> do anyone have the log from the last meeting?
15:07:54 <adamw> bittin: you can always find the minutes and logs at https://meetbot-raw.fedoraproject.org/teams/fedora-qa/
15:08:03 <adamw> (there's a more processed front end too but i like the raw ones)
15:08:18 <adamw> #info no action items from previous meeting
15:08:28 <adamw> any other follow-up?
15:08:29 <bittin> adamw: ah thanks :)
15:08:52 <bittin> have a follow up in section #3 about the Kernel 5.13 test week
15:09:19 <bittin> but nothing from last week no
15:09:46 <adamw> rgr. we'll have that section later
15:10:36 <andi89gi> Hey
15:11:42 <adamw> #topic Fedora 35 status
15:11:56 <adamw> so up to about the middle of last week this was "pretty great", now it seems to be "completely busted" :D
15:12:28 <bittin> the Rawhide images that was released earlier today works, but 10 July, 10.5 July and 11th July etc was busted
15:12:43 <bittin> so its a bit up and down
15:12:51 <adamw> oh, today's was better? I didn't get a minute to look yet
15:13:01 <bittin> yeah all nightlies today was green
15:13:08 <adamw> it's still failing nearly everything in openQA
15:13:13 <bittin> https://openqa.fedoraproject.org/nightlies.html
15:13:24 <bittin> seems it works some days and some days not
15:14:16 <adamw> openQA tests haven't been reaching anaconda since Fedora-Rawhide-20210709.n.1
15:14:21 <adamw> 0706.n.0 was OK
15:14:43 <lruzicka2> weekend surprise
15:15:12 <cmurf> adamw is on pto surprise
15:15:42 <adamw> i was, i'm back now
15:16:03 <andi89gi> .hello andilinux
15:16:04 <zodbot> andi89gi: andilinux 'Karsten Andreas Artz' <kartz@gmx.net>
15:16:12 <adamw> 09.n.1 had new glibc and kernel, could be either of those i guess
15:16:13 <lruzicka2> PTO = something breaks
15:16:15 <tablepc> Now that you're back I'm sure OpenQA will be good again
15:16:16 <adamw> no new anaconda or dracut...
15:16:17 <adamw> heh
15:16:33 <adamw> i mean it's not openqa that's broken, it's rawhide :P
15:16:44 <lruzicka2> I guess the problem is not OpenQA, it's Fedora
15:16:57 <bittin> yeah rawhide had some broken builds for 3-4 days
15:17:01 <bittin> but todays nightlies work
15:17:01 <adamw> oh, also a new systemd
15:17:08 <tablepc> Yea, but OpenQA reports the bad news
15:17:13 <adamw> bittin: not in openqa they don't. no difference from yesterday.
15:17:22 <adamw> https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=Rawhide&build=Fedora-Rawhide-20210712.n.0&groupid=1
15:17:32 <lruzicka2> If OpenQA lived in Sparta it would have been killed already for bringing bad news
15:17:43 <bittin> adamw: hm wierd, yeah systemd 249
15:18:14 <tablepc> what's being fiddled with in systemd?
15:18:25 <bittin> tablepc: https://github.com/systemd/systemd/releases/tag/v249
15:18:53 <cmurf> looks like it booted, and we're at a login prompt, not sure what openqa is waiting for
15:19:18 <bittin> cmurf: something to autologin? :p
15:19:45 <cmurf> i'm downloading Fedora-Cloud-Base-Rawhide-20210712.n.0.x86_64.raw.xz now, need to check it for hybrid boot and fstrim anyway
15:20:21 <tablepc> levity levity that will make it better
15:20:45 <cmurf> there have been significant changes to the cloud images in the last week or so
15:20:56 <lruzicka2> cmurf, I see lots of dracut-initqueue errors ... the system never reached anything
15:21:03 <cmurf> but getting to a login prompt is a really good sign
15:21:24 <lruzicka2> cmurf, https://openqa.fedoraproject.org/tests/925881#step/_boot_to_anaconda/6
15:21:49 <lruzicka2> cmurf, this is the end, my friend: https://openqa.fedoraproject.org/tests/925881#step/_boot_to_anaconda/30
15:22:09 <adamw> cmurf: if you're looking at cloud tests they fail because login fails
15:22:20 <adamw> cmurf: installer tests don't reach anaconda, they die at the dracut emergency console
15:22:21 <cmurf> ok
15:22:35 <cmurf> ok i guess that's a different image than what i'm looking at
15:22:37 <bittin> so status right now is shaky/not so good
15:23:22 <cmurf> i'll need to look at the rdsosreport, the console doesn't have enough detail, didn't get sysroot mounted is a nice hint but there's countless ways that could blow up
15:23:36 <cmurf> systemd could be a suspect though
15:23:45 <adamw> i'm going to look at it when i'm not in a meeting. ;)
15:24:06 <cmurf> same
15:24:49 <cmurf> oh well, 4 weeks to branch, nbd
15:24:57 <adamw> #info Rawhide was working fine until the middle of last week, recent composes have big problems, most images fail to boot and Cloud images fail to log in. Problems started with 20210709.n.1 compose. we will look into what's causing the problem today
15:25:52 * nirik notes he hasn't seen any problems here...
15:26:20 <bittin> i had some problems in my VMs but its been a bit off and on between the releases
15:26:33 <nirik> I wonder if it's selinux? I am in permissive mode.
15:27:07 <nirik> bunch of dbus denials. ;)
15:28:01 <tablepc> Are me getting a Gnome drop this time?
15:28:09 <tablepc> we
15:28:30 <adamw> hey, another suspect
15:29:41 <cmurf> so there's an f34 problem related to dracut and dbus-broker, i wonder if it snuck into rawhide too?
15:29:49 <cmurf> https://bugzilla.redhat.com/show_bug.cgi?id=1976653
15:30:18 <cmurf> that could be a goosechase though because that bug looks only lvm related, whereas i'm not seeing it on btrfs with f34, and cloud images are now also btrfs
15:30:57 <cmurf> nevermind that bug only causes boot delay, not boot failure
15:31:15 <cmurf> (and strictly speaking it's startup delay, not startup failure)
15:31:18 <adamw> anyhow
15:31:30 <adamw> something's busted, we don't need to do the debugging live on stage
15:31:39 <adamw> any other notes on f35 status besides "bad"? :D
15:32:36 <andi89gi> hmm I have had issues with rawhide isos up to yesterday on VM with the boot process. No start up /boot freezing
15:33:05 <cmurf> andi89gi: which image?
15:33:31 <andi89gi> https://openqa.fedoraproject.org/nightlies.html
15:33:41 <andi89gi> from this side Workstation+KDE
15:34:02 <bittin> andi89gi: yeah those have been broken for some days off and on but the one from earlier today should work
15:34:07 <jforbes> cmurf: A lot of people filed it as failure, but it is actually a 90 second delay
15:34:19 <cmurf> yeah
15:34:24 <andi89gi> hmm really?
15:34:32 <jforbes> I believe it is in rawhide as well, but it does happen on btrfs too
15:34:37 <andi89gi> I see
15:34:40 <cmurf> well it's a fine line :) if something hangs for 5 seconds I'm reaching for the power button
15:34:48 <bittin> when they are red on the site instead of blue they are broken
15:34:54 <jforbes> And yeah, for a long time, a solid half of the filed kernel bugs were that... Anoying
15:34:58 <bittin> as where the case almost all mid last week
15:35:03 <andi89gi> cmurf: ah not 5secs - over 1min or even 2
15:35:07 <cmurf> hmm i haven't seen it
15:35:35 <andi89gi> I mean let me test the current iso of tday
15:35:37 <andi89gi> ;)
15:38:10 <adamw> so yeah, if everyone can poke current rawhide images and try to figure out what all is wrong with them, please do. :P
15:38:24 <adamw> #topic Test Day / community event status
15:38:25 <adamw> sumantro: ahoy!
15:38:37 <bittin> he is not here
15:39:04 <bittin> anyways the Kernel 5.13 test week has started, have done some testing myself also it got mentioned in Todays Linux Action News Episode #197
15:39:07 <bittin> https://linuxactionnews.com/197
15:39:17 <jforbes> kernel test ISOs are up, I haven't been following up on the results page yet, but I have seen that people are submitting test logs
15:39:18 <bittin> https://fedoraproject.org/wiki/Test_Day:2021-07-11_Kernel_5.13_Test_Week
15:39:28 <bittin> and here we have sumantro
15:39:28 <sumantro> Hey all!
15:39:50 <sumantro> Kernel 5.13 Test week is on and its show time. :)
15:39:54 <andi89gi> Hey sumantro
15:40:04 <bittin> hi sumantro
15:40:08 <jforbes> 125 tests just on x86 so far
15:40:10 <sumantro> I see a bunch of reports on https://testdays.fedoraproject.org/events/115
15:40:27 <sumantro> jforbes, thanks , its been like 1 day :D
15:40:33 <bittin> and i heard Linux Action News mentioned it on todays episode #197
15:40:38 <bittin> https://linuxactionnews.com/197
15:40:46 <bittin> just a short notice but might bring on some
15:41:03 <sumantro> bittin, yes they did https://linuxactionnews.com/197?t=804 [that time stamp is where we are mentioned]
15:41:41 <sumantro> overall, test days in general are getting a good amount of attention from the communities now :D
15:41:48 <sumantro> jforbes, any other thoughts?
15:42:03 <jforbes> Nope, pretty happy to see the number of tests so far
15:42:13 <andi89gi> yea I am participating on it ;)
15:42:16 <bittin> i will continue to test for the 6 days more :)
15:42:18 <bittin> too
15:42:20 <jforbes> I am going to be doing 5.13.2 later in the week, and would like to see some testing on it too.  800 patches
15:42:30 <bittin> jforbes: will help out :)
15:42:46 <sumantro> jforbes, will make announcements as needed :D
15:42:48 <jforbes> I will replace the links to the kernel and put up fresh images when it is available
15:42:50 <andi89gi> But for me 1st Kernel test run well
15:43:14 <bittin> yeah has worked good for me to on a HP 655 laptop and in Virtualbox
15:43:20 <sumantro> andi89gi, you will get a badge :)
15:43:21 <bittin> jforbes: sumantro: +1
15:43:45 <sumantro> That's all for Test Days from my side
15:43:49 <adamw> thanks folks
15:43:55 <andi89gi> sumantro: thx!
15:44:06 <sumantro> andi89gi, np!
15:44:18 <adamw> #info Kernel 5.13 Test Week is on now, thanks to sumantro and jforbes for organizing that, please join in if you can!
15:45:59 <bittin> Open floor?
15:46:44 <adamw> yup
15:46:45 <adamw> #topic Open floor
15:47:55 <bittin> I will miss the meeting next week if there is any as i will help out rigging up GUADEC Gnomes Europe Conference Online http://guadec.org then and need to do some personal food and drinks shopping before then but will be back in 2 weeks
15:48:09 <bittin> if you have any time your welcome to visit when GUADEC Online 2021 opens up for visitors in 8 days
15:48:59 <bittin> and like GNOME
15:49:10 <adamw> #info GUADEC 2021 (online) starts July 21, check it out if you're free!
15:49:51 <bittin> Fedora is one of the sponsors and will have something one of the days
15:50:18 <bittin> https://events.gnome.org/event/9/page/29-sponsors
15:53:20 <bittin> anyone else have anything?
15:53:43 <adamw> apparently not!
15:53:48 <adamw> thanks for coming folks
15:53:49 <tablepc> Have a Great Day/Evening Everyone!
15:53:58 <bittin> tablepc: same to you and everyone else
15:54:04 <bittin> cya in two weeks (for me)
15:54:24 <frantisekz> bye, thanks for the meeting!
15:54:33 <lruzicka2> have a nice time
15:55:23 <bittin> #endmeeting
15:55:24 <adamw> #endmeeting