15:00:11 #startmeeting Fedora QA meeting 15:00:11 Meeting started Mon Jun 15 15:00:11 2015 UTC. The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:11 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:00:13 tnx 15:00:15 #meetingname fedora-qa 15:00:15 The meeting name has been set to 'fedora-qa' 15:00:21 #topic Roll call 15:00:26 * roshi is here 15:00:29 ahoyhoy folks, who's around for some QA meetiness? 15:00:30 wb, adamw 15:00:37 * satellit listening 15:00:38 hi randomuser 15:01:57 * tflink is here, wrapping up the qadevel meeting 15:03:26 * kparal is here 15:03:32 .hello dmossor 15:03:33 danofsatx: dmossor 'Dan Mossor' 15:03:41 greetings, folks. 15:03:55 hi tflink, kparal, dan 15:04:25 * FranciscoD sits in a corner to watch :) 15:05:11 this room is shaped like an egg. 15:05:18 * adamw watches franciscod cease to exist in a puff of logic 15:05:24 :o 15:06:03 qa-devel meeting done yet? 15:06:06 oh, yes. 15:06:08 let's go then! 15:06:20 #topic Previous meeting follow-up 15:06:30 I dug out the minutes from last time, doesn't look like there were any action items 15:06:38 #info there appear to be no action items from last time 15:06:44 was there anything else that requires follow-up? 15:07:39 I can't think of anything 15:07:59 okely dokely 15:08:10 #info there's nothing requiring follow-up from 2015-06-01 meeting 15:08:17 #topic Fedora 23 schedule / status / planning 15:08:35 #info the current F23 schedule is available at https://fedorapeople.org/groups/schedule/f-23/f-23-quality-tasks.html 15:09:10 action adamw: create action items 15:09:33 as it stands, we have the first blocker meeting scheduled 07-01, Alpha TC1 07-14, Alpha freeze 07-28, and go/no-go 08-06 15:09:39 * adamw disappears in a puff of recursion 15:09:46 wow, that soon? 15:09:48 oof 15:10:04 i think this is another short cycle to try and get back on october releases 15:10:09 I won't be at that meeting. I'll be in the middle of the Carribean ;) 15:10:10 Final release is scheduled for 10-27 15:10:42 danofsatx: i wish I could use that excuse :) 15:11:08 tflink: I'm pretty sure that was danofsatx's way of inviting us 15:11:22 my FIL paid for it. I can't afford it :/ 15:11:23 did that get moved up or is it that we usually slip until after US thanksgiving? 15:11:43 so, i've been kinda kicking around with dgilmore some ideas about changing the alpha/beta/final process, but we want to discuss that in more detail at flock and it looks like we'll be well into f23 by then 15:12:14 so we might change it up for F24, then 15:12:25 tflink: we slipped gradually backwards over i think 17-20 15:12:35 could we get a tl:dr of what you guys have discussed? 15:12:37 we kept slipping a few weeks then scheduling another 6 months 15:12:44 roshi: if the idea holds up, yeah 15:13:13 roshi: basically my thought is that the whole idea of alpha/beta milestone releases is a bit antiquated these days 15:13:27 we put a lot of effort into them and then mostly people just go and install TCs or nightlies 15:13:27 roshi: I'm game for a trip to the Caribbean :) 15:13:35 adamw: :) dropping alpha would be awesome 15:13:53 so i'd like to look into the possibility of a more nightly-based process 15:13:55 we should have nightly TC composes for F23 15:14:16 call 'em TCs or whatever, but just focus on continual improvement on the nightlies 15:14:17 in releng we are working to make branched and rawhide look just like a TC 15:14:26 similar to what we did between F21 and F22, but with the branched tree instead of rawhide, then? 15:14:30 * tflink would be interested in seeing how much traffic the alpha composes see after alpha release (knowing that the stats from dl.fp.o aren't the whole picture) 15:14:37 i suspect that it's minimal 15:15:06 roshi: well, what we did for 21 and 22 was do more nightly testing before alpha, but once we hit alpha we did more or less the same as before 15:15:17 right 15:15:24 my rough idea would be to just not do alpha or beta at all, and rework release validation around the nightlies 15:15:53 well us going through all the blockers every meeting fits that pretty well 15:15:58 there's a lot of questions of course, like how do we make sure the nightlies are at a reasonable minimal level of quality throughout, or at least major showstoppers are quickly fixed, without the alpha/beta blocker process 15:16:00 and then we'd just have "blockers" 15:16:02 but we can look at it at least 15:16:07 right 15:16:24 the one thing I can think of that removing alpha/beta will affect would be the "press" 15:16:27 obviously it's kind of a CI-ish model so we'd want it tied into taskotron/openqa testing somehow 15:16:30 milestone releases have marketing value 15:16:37 as in, no we have alpha/beta announcements and so on 15:16:38 * randomuser highfives FranciscoD 15:16:43 FranciscoD: yeah, that's another point, alpha/beta have PR value too 15:16:44 lol 15:16:45 adamw: taskotron can fire stuff based on composes 15:16:45 that was my thought as well 15:16:50 so, there's lots to think about there 15:16:52 randomuser: you stole my sentence :P 15:17:04 hopefully we'll have a better beaker system before too much longer 15:17:11 * tflink doesn't know how openqa schedules stuff 15:17:23 adamw: tflink: down the magical rod that sees into the future, I want to fire off composes when pieces that are in it change 15:17:31 so we do a CI based compose 15:17:39 tflink: dumbly, of course! basically you can run a cron job that'll look and see if it has something new to test. :) 15:17:51 fair enough 15:17:56 which would then go through a bunch of automated testing before going elsewhere 15:18:02 the marketing problem shouldn't be too difficult to handle though, stuff can be labelled alpha/beta even if the milestones aren't being used for QA 15:18:07 dgilmore: what do you want/need from us to make that happen? 15:18:20 the compose parts, not the automated testing parts :) 15:18:25 FranciscoD: yeah, the good old tried and true PR strategy of 'lying' :) 15:18:32 * tflink has a decent idea of what's needed there 15:18:34 tflink: we need to know what is actually in the pieces to know when to trigger composes 15:18:57 tflink: then we need tooling to dynamically know what to make 15:19:04 so we only make things that will be changed 15:19:26 dgilmore: one thought would be to use taskotron to analyze what's changed and submit signals to fire off a compose 15:19:35 but we can discuss that outside the meeting, if it makes sense 15:19:37 * adamw notes we're a bit off the f23 topic but doesn't want to stop the train as it's a useful discussion 15:19:44 i'll fix it in the #info later, we can get back to 23 discussion in a bit 15:19:46 * tflink isn't sure what all has changed in releng plans recently 15:19:54 tflink: lets chat later 15:19:57 dgilmore: how's work coming along on reducing compose times? 15:20:12 adamw: for f23 we should have a nightly compose that looks like a TC/RC 15:20:20 at least that is what releng is working towards 15:20:28 dgilmore: that'd be great if we can get that soon 15:20:45 adamw: its getting there, slower than we want, but we are making progress 15:20:46 dgilmore: sounds like a plan 15:21:44 #info we have some thoughts on overhauling the release validation process to be more nightly-based and CI-ish and removing or downplaying the significance of Alpha and Beta releases, but these are probably F24 or later stuff with the shortish F23 cycle 15:22:02 #info releng is working on having full TC/RC-style nightly composes during F23 cycle (not just lives and boot.iso) 15:22:35 so for F23 - do we want to move up the first blocker review meeting, carrying on the idea of 'do it early and often'? 15:22:53 there are 2 proposed alpha blockers atm 15:23:08 we could do a quick meeting next week to start the process off 15:23:14 "want"? not so much, "would it be a good idea?", probably :) 15:24:27 don't everyone jump for joy all at once :) 15:24:42 * roshi will send the announce 15:25:41 * tflink plants feet firmly on floor, will do no more jumping for joy 15:26:09 #info we'll do the first blocker review meeting on 06-22 to get out in front of the proposed blockers 15:26:44 otherwise i think we're more or less in shape for 23, right? all the tools and things seem to be pretty much ticking over 15:26:56 can anyone think of any outstanding test case / criteria issues? 15:27:46 kinda related: what are our expectations for mid-release updated images? 15:28:17 ah, yeah, that's another active thing 15:28:27 if it's gonna happen in f23 cycle we certainly need to discuss it, thanks roshi 15:28:42 it's still at the 'being kicked around in fesco' stage right now, right? 15:28:55 np - I suspect we'll need to have some criteria discussion with it 15:28:59 afaik, yeah 15:29:16 I mean, there's been discussion about it since F21 15:29:22 it was part of the Grand Plan 15:30:25 Are we talking Alpha criteria, or all of it? 15:30:27 https://fedorahosted.org/fesco/ticket/1444 15:30:40 danofsatx: for an updated release image, I'd say all of it 15:30:46 hmmm.... 15:30:57 what good is a new released image with beta and final blockers on it? 15:30:57 * danofsatx comes up with a plan 15:31:24 Final has to have no blockers of any kind, so an updated image/media/whathaveyou would too 15:31:32 I was speaking specifically in regards to the criteria itself, not whether the images met the criteria 15:32:00 #info roshi notes that there is a current discussion around shipping post-release respins for some images: https://fedorahosted.org/fesco/ticket/1444 . This obviously has significant QA implications 15:32:19 I was thinking of going over the criteria with a fine tooth comb to determine what is still relevant, and what, if anything, was missing. 15:32:34 danofsatx: well, i'd say at present all we have is a higher-level question than that: how do we ensure respun media are of acceptable quality? 15:32:57 danofsatx: or were you going back to my earlier 'can anyone think of any criteria issues?' question? 15:33:05 I was going back 15:33:09 sorry, I'm distracted 15:33:31 * danofsatx puts on headphones to ignore the office 15:33:50 danofsatx: that's totally fine, sorry, just got wires crossed :) 15:33:56 there are already updated respins, but they're not officially done by fedora infra 15:34:18 or tested by us, afaik 15:34:20 yeah 15:34:26 Updated Live isos: http://tinyurl.com/Live-respins 15:34:30 danofsatx: i was meaning the whole of the criteria. it's pretty much always a good idea to go through the whole set and see if you find anything that's outdated or incorrect or missing 15:34:31 * satellit soas is not part of respins though 15:34:40 been in the #fedora topic for quite a long time now 15:34:41 danofsatx: i try to do it every so often but it's always a good idea 15:34:50 danofsatx: so if you want to do that, more power to you 15:34:59 anything you see, send out to the list as a proposal 15:35:02 I know you do, but it seems a Herculean effort for one busy person 15:35:06 * tflink thought that one idea was to get the WGs to do most of the testing 15:35:11 FranciscoD: right, the current live respins are unofficial and we don't do any testing on them 15:35:17 yeah 15:35:22 for respins, anyways 15:35:42 #info existing live respins - http://tinyurl.com/Live-respins - are semi- (or quarter-?) official and have no official QA process 15:35:50 I don't reckon they'll need too much testing - we don't push radical changes to stable fedora release anymore. 15:36:01 But yes, one has to install and test them, so that is quite a bit of work 15:36:13 eh. there are kernel updates and dracut updates to name just two... 15:36:15 another question is how to handle breakage 15:36:21 right 15:36:37 for example (not so likely but an example) say some update breaks anaconda's ability to install the respins 15:36:49 there's the 'obvious' idea of just re-running the whole existing validation process for respun images...which obviously would probably work fine, but seems like a lot of work 15:37:26 does that mean the anaconda devs are "on the hook" to fix it, does the update at fault get reverted or some other method for mitigation? 15:37:38 then there's a whole spectrum of options between that and 'say we don't have time to do anything and it's up to WGs to test them or just kick them out the door' 15:37:45 that's why in fesco ticket #1444 I suggested that the WGs do the testing, and bring their results to QA to sign off on 15:37:48 right, that's another point 15:38:04 hrm, but then the WGs need their own "QA" teams? 15:38:04 though seems like it might not be ours to decide (tflink's note) 15:38:22 FranciscoD: they should already have folks willing to help test 15:38:26 or will QA work with WGs? 15:38:34 the issue in my mind is that the big issue is personpower 15:38:44 FranciscoD: from a QA standpoint, I would think QA can do best effort help 15:38:57 they probably do, but those folks will need to get up to speed with the QA validation process and things 15:39:00 ++ 15:39:06 since we generally have our hands full with the current release 15:39:15 if the WGs want respins, they need to help :) 15:39:39 is the respin idea coming out of WGs? 15:40:04 afaict, QA could really care less in our day to day lives if there are updated images - because we're always working on the next thing 15:40:10 the ticket is from dennis, and it sounds like it's with his releng hat on 15:40:10 adamw: yeah, it is 15:40:13 won't making netinstallers primary sort of take care of this? Normal testing continues, and people get newer packages? 15:40:24 FranciscoD: the obvious candidates for respins are the cloud imagesd 15:40:32 my understanding is that it's cloud that wanted first, then atomic and whatnot 15:40:36 which is kinda outside that scope 15:40:45 Ah, OK. Makes sense 15:41:10 and there was talk from the workstation WG about wanting respun live images as well 15:41:45 cloud images don't worry me as much since they're on the simpler side of deliverables 15:41:51 if one WG wants them, everyone will want them too :) 15:41:56 so, this has been mostly a kick-around discussion so far 15:42:12 do we want to take any definite points into the fesco discussion at this point, or do we want to wait for the proposal to take a bit more shape? 15:42:13 * tflink notes that the cloud WG doesn't exactly have the best track record for testing stuff 15:42:32 adamw: yeah - it's been kicked around for a long time 15:42:43 it's getting better :) 15:42:46 trusting them to do the testing just because they say they will ... may not turn out so well 15:42:53 they're even coming to blocker review meetings 15:43:05 well, if no testing happens, no updated image 15:43:05 * tflink will believe it when he sees it 15:43:10 seems easy enough to me 15:43:18 yeah, that's what i was getting at 15:43:25 and there's that automated testing thing for cloud images too 15:43:25 a "must be this tall to ride" kinda thing 15:44:00 * satellit are netinstall tests = to repins for DE's? 15:44:05 the only thing that concerns me about a WG-based approach is that it's kinda inconsistent; do we wind up with a future where some chunks of fedora QA work run through us and some run through WGs and the border between the two makes no logical sense? 15:44:08 don't commit to "yes we're OK with respun images on X date" without a "without testing done by a certain date, no respins" 15:44:27 as things stand we're supposed to be 'the' fedora QA organization and in theory if we need more resources they should just magically appear and route through us, i think 15:44:35 adamw: do you think we have the spare cycles to commit to respin testing? 15:44:41 but on a practical level, requiring the WGs to bring the resources makes sense 15:44:53 it depends on what the meaning of the word 'we' is ;) 15:45:17 adamw: due to a lack of personpower, having the WGs do testing on updated images makes sense I think, so we can still focus on the next release 15:45:46 I'd love it where the WGs assigned people to work as their QA monkeys with us 15:45:51 ok, so let me take a couple of notes 15:46:49 to address both concerns, maybe set the bars for the required testing for respins without committing time from folks who are usually doing Fn+1 work? 15:46:55 #info we have a consensus that respins shouldn't be released without some kind of testing commitment in place, and the WGs responsible for the media to be respun need to help provide the resources to do the testing and demonstrate that it's actually occurred 15:47:01 or some less akward/divisive way of phrasing that 15:47:07 does that sound OK? 15:47:08 that -> what I said 15:47:26 adamw: sounds good to me 15:47:28 yeah, that works 15:47:50 ok, i'll try and summarize this for the fesco ticket and we can do more concrete discussion when the fesco proposal is more concrete 15:47:56 sgtm 15:49:10 so let's move on quick... 15:49:14 #topic Fedora 22 retrospective 15:49:44 I didn't really have much for this topic and we don't have much time, but does anyone have any reflections on the 22 cycle? any thoughts on how the things we changed worked out? 15:50:24 i thought the idea to try and get blockers found and discussed earlier worked out pretty well overall, seems like there was less drama with anaconda team and we got the release done with only 1 week slip overall 15:50:39 ^^ that 15:50:47 I thought the blocker process went really well 15:51:01 we almost never went over time, the meetings were generally shorter 15:51:20 we did still wind up with some fairly long meetings near the end of the cycle, but i think it was better overall 15:51:29 don't think we had any 6-hours-over-two-days epics 15:51:41 not that I recall 15:52:29 I think openQA helped as well 15:52:43 how's the packaging of it going? need extra hands for it? 15:52:48 I'm still slightly disgruntled about the KDE xcb error that got practically ignored. 15:53:40 which bug is that, dan? 15:53:45 hang on... 15:54:03 .bug 1193742 15:54:06 danofsatx: Bug 1193742 (intel) Plasma5 Freezes (often due to notifications while locked), xcb/dri3 deadlock? - https://bugzilla.redhat.com/1193742 15:54:30 roshi: it's not going atm, still in post-vacation-clearout mode. but it's not super difficult. i do have an os-autoinst package that builds; that was about as far as i got before my vacation. didn't check if it actually works as i don't know how to make os-autoinst do something useful outside the openqa framework :) 15:54:45 roshi: http://paste.fedoraproject.org/232219/34383679 15:55:58 * roshi will take a look 15:56:32 danofsatx: hum, afaict that one never really hit any of the release validation processes, right? 15:56:33 it' 15:56:42 doesn't look like it was proposed as a blocker or FE at any point 15:56:51 It was. 15:57:56 ah, it was proposed as https://bugzilla.redhat.com/show_bug.cgi?id=1217844 15:57:59 that's what I was just looking at 15:58:14 that's the other one I was looking for 15:58:31 seems like it was punted on 05-11 and then never re-discussed? 15:58:36 at least from the comments 15:59:25 yeah 15:59:37 I was just dissapointed in the process where a release-blocking desktop locking up did not qualify as a blocker. 15:59:41 * roshi wonders how this got missed in other meetings? 16:00:41 I apparently wasn't eloquent (or pushy) enough to get this acted on, either by QA in general, the KDE Sig, or upstream. 16:01:02 danofsatx: it looks more like a process breakdown 16:01:08 looks like in comment #15 the blocker tracker got removed 16:01:10 danofsatx: though the KDE sig does seem to be actively working on it 16:01:26 ah, roshi's right 16:01:26 so it never showed up in the irc list after that 16:01:27 yeah, rdieter is driving it. 16:01:56 so we never discussed it again because it had gone into stealth mode 16:01:58 danofsatx: so the problem is that when doing the secretarialization you took it off the proposed blocker list, so it didn't show up for subsequent meetings 16:02:11 probably my fault for not noticing when i reviewed the secretarialization :( sorry 16:02:28 and my fault for screwing it up (again) 16:02:33 ah well, we live and learn :) 16:02:50 * danofsatx vows to never volunteer again (NAVY - Never Again Volunteer Yourself) 16:03:06 that seems a bit harsh :p 16:03:32 this is a case where a respun live image could make sense 16:03:34 but accurate :P 16:03:43 potentially i guess yeah 16:03:54 but live respins seem like the hardest case to me as you kinda wanna re-do all the validation 16:04:03 they're desktops and they're installers 16:04:11 anyhow, we're getting over time... 16:04:20 I'm kinda working from the assumption that a respin of anything will need a whole run of the matrix 16:05:41 * roshi has nothing else to add 16:06:16 #agreed F22 process tweaks like more nightly validation, earlier blocker review and openQA all worked pretty well, we'll continue with them for F23 16:06:56 #info https://bugzilla.redhat.com/show_bug.cgi?id=1217844 was a victim of a secretarialization mistake during blocker review, let's remember to double-check our work for F23 :) 16:07:01 #topic Open floor 16:07:07 anyone have anything for open floor, quickly? 16:07:27 just this: great work on F22, folks :) 16:07:55 yep! thanks for all the work on f22, everyone! 16:08:51 nothing here. 16:08:59 see roshi's heroes of fedora posts: part 1 is up at https://roshi.fedorapeople.org/horoes-of-fedora-hof-fedora-22-part-1.html 16:08:59 * danofsatx votes for close. 16:09:14 part 2 and 3 are forthcoming 16:09:17 awesome job satellit on all the validation tests 16:09:31 roshi: i note that at present it's actually a Horoes of Fedora post. :P 16:09:40 eh 16:09:43 * roshi fixes it 16:10:14 * satellit_e :) 16:10:30 ok, thanks again for coming everyone 16:10:36 * adamw sets fuse 16:10:37 gotta fire my copy editor... 16:10:47 * danofsatx is looking for a job 16:11:31 https://roshi.fedorapeople.org/heroes-of-fedora-hof-fedora-22-part-1.html 16:12:42 danofsatx: roshi offers competitive pay and benefits, if by 'competitive' you mean 'none' :P 16:12:53 call it an internship! 16:13:03 haha 16:13:14 #endmeeting