16:00:05 <adamw> #startmeeting Fedora QA meeting 16:00:05 <zodbot> Meeting started Mon Nov 23 16:00:05 2015 UTC. The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:05 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:05 <zodbot> The meeting name has been set to 'fedora_qa_meeting' 16:00:09 <adamw> #meetingname fedora-qa 16:00:09 <zodbot> The meeting name has been set to 'fedora-qa' 16:00:13 <adamw> #topic Roll call 16:00:25 <adamw> ahoyhoy folks, who's around for the QA meeting? 16:00:37 * garretraziel is here 16:01:03 <garretraziel> pschindl will hopefully join us soon 16:01:05 * tflink is here 16:01:20 * kparal is here 16:01:26 * roshi is here 16:01:31 * satellit_e listening 16:01:46 * adamw abandons his fantasy of no-one saying anything so he can go back to bed 16:02:18 <adamw> how's everyone doing? 16:03:16 <adamw> bright and bushy-tailed, huh 16:03:17 <adamw> alrighty 16:03:21 <adamw> #topic Previous meeting follow-up 16:03:32 <kparal> we didn't want to say anything so that you can go to bed 16:03:55 <adamw> :P 16:03:58 <kparal> your fault :) 16:04:01 <garretraziel> yeah, but kparal ruined it 16:04:02 <kparal> I'm fine, thanks 16:04:13 <adamw> #info "adamw to draft up proposal for better handling of 'special blockers'" - yep, did that, and we'll be discussing the draft later in the meeting 16:04:20 <adamw> garretraziel: doesn't he always? jeez, that guy. 16:04:21 <kparal> garretraziel: do you think he would realize after 10-15 minutes? 16:05:46 * pschindl is here (late from train) 16:05:50 <garretraziel> oh, here he comes, my job here is done 16:06:18 <adamw> #info "adamw to finish off the 'installer help' criteria / test case changes" - did that: can't find the link in new hyperkitty mailing list archive, but trust me, I did it 16:07:03 <adamw> #info "adamw to work with releng to get rawhide nightly boot.iso compose working and create an initial f24 nightly validation event" - did that too, nirik fixed all the things stopping nightly composes and we have nightly validation events going now 16:07:06 <adamw> any other follow-up? 16:07:15 * adamw does his one-man show again 16:07:59 * satellit_e lives still failing in koji and boot.iso fails in plasma today 16:09:03 <adamw> satellit: probably more dep failures, i guess? did you check the logs? 16:09:10 <adamw> also what do you mean by 'boot.iso fails in plasma'? 16:09:23 <satellit_e> open office seems broken with depends 16:10:03 <satellit_e> plasma in boot.iso boots but ends up with black screen with cursor 16:10:10 <adamw> ah, yeah, the python 3.5 rebuild failed. 16:10:13 * satellit_e virt manager 16:10:19 <adamw> satellit: you mean it fails post-install ? 16:10:24 <satellit_e> yes 16:11:08 <satellit_e> correction no...at boot 16:11:15 <satellit_e> of live 16:11:50 <satellit_e> boot.iso install finishes and logs in then black screen and cursor 16:12:26 <adamw> right, OK. did you report the boot.iso failure? 16:12:52 <satellit_e> not today 16:13:41 <satellit> https://fedoraproject.org/wiki/Test_Results:Fedora_24_Rawhide_20151120_Installation 16:13:44 <adamw> ok, we should probably get a bug report in, i can see if i can reproudce the problem later...thanks 16:14:08 <adamw> #info satellit reports ongoing problems with Rawhide image compose and install 16:14:09 <adamw> so, onto our first topic, I guess... 16:14:28 <adamw> #topic Non-media blocker process 16:15:48 <adamw> so for background, I submitted an initial dump of ideas for changing the process for handling blocker bugs that don't need fixing on the release media 16:16:05 <adamw> man, i picked a bad time to learn hyperkitty, live in the middle of a meeting...:P 16:17:07 * kparal already struggled with the admin interface 16:17:29 <adamw> alright, so, er, no link to the first mail, let's just hope everyone has it in their mail client :) 16:17:43 <adamw> there's been some wide-ranging discussion on the thread so it seemed like a good topic to kick around in a meeting 16:17:45 <roshi> aw, man, I have to load my mail client... 16:17:49 <roshi> such a bother... 16:17:55 <adamw> SORRY ROSHI 16:18:00 * adamw plays tiny violin 16:18:09 <roshi> sheesh, demanding canadians... 16:18:12 <adamw> =) 16:18:16 <adamw> #info roshi is lame 16:18:38 <roshi> I'm certainly not feeling spry, that's for sure :p 16:19:04 <adamw> so i'd say the big discussion has been on the question of what we should do in terms of 'enforcing' non-image blockers 16:19:29 <adamw> currently we kind of wave our hands at them and say 'lo! a fix shall appear before release!' and then we forget about them, which seems like it could stand improvement 16:19:57 <kparal> I'm not aware whether releng has some processes to go along with this, but my comments were mostly about the fact that we'll probably need those as well 16:20:38 <adamw> nirik: dgilmore: around? 16:21:01 <roshi> I haven't had a chance to look at the PDC project much yet, but perhaps that has some bits and bytes that can help track all this? 16:21:06 <adamw> so aiui, releng definitely makes sure a 0-day update push *happens* 16:21:19 <kparal> ok, that's good 16:21:19 <adamw> roshi: i don't know much about it either 16:21:28 <roshi> Product Definition Center - which will have lists of all our images and whatnot 16:21:35 <adamw> roshi: i know what it *is*, yes. 16:21:37 <kparal> I'd be interested to know what went wrong in our system-upgrade.fc22 case 16:21:45 * roshi was going to work with threebean on getting it populated with data 16:21:50 <kparal> so that we can avoid it next time 16:21:52 <nirik> yes, as soon as the release is tagged in koji, we can do a updates push... usually thats the next day... 16:21:59 <roshi> adamw: that was more for others than for you :p 16:22:06 <adamw> =) 16:22:08 <dgilmore> adamw: what is up? 16:22:32 <adamw> dgilmore: we're considering the topic of non-media release blockers ('special blockers') 16:22:56 <adamw> kparal: was interested in what policies releng has for update pushes around stable releases (right kparal?) 16:23:21 <kparal> nirik: dgilmore: do you have a process that makes sure that 0day updates are pushed (and mirrored, and mirrormanager does no longer point to outdated mirrors) before the release announcement is published? 16:23:51 <dgilmore> adamw: we have no policies outside of making sure that everything used in teh release is pushed stable 16:23:52 <kparal> I'm talking about that 1-2 day period when mirrormanager can still return an outdated mirror (the previous metadata) 16:23:53 <adamw> kparal: note, 0-day applies only to the release itself; I don't know whether there's any rule about getting updates for the previous stable releases pushed 16:24:07 <kparal> ah 16:24:08 <adamw> dgilmore: so 0-day updates are basically a best-effort? 16:24:18 <dgilmore> adamw: we try and do 0 day pushes, but if they do not happen they do not happen 16:24:23 <kparal> ok, I was talking about previous stable releases 16:24:25 <dgilmore> adamw: they are 16:25:04 <dgilmore> kparal: if we need say a dnf update in a previous release for upgrades to work, they are on a best effort basiss 16:25:16 <kparal> would you be open to some changes, to make sure important updates in previous stable releases are pushed before the new release announcement goes live? 16:26:00 <dgilmore> kparal: we would not be. too much can go wrong in updates pushing. updates pushing are a best effort basis and that is not going to change without major changes in how we push updates 16:26:02 <kparal> because we experience a very unfortunate mess-up with system-upgrade.fc22 this time 16:26:08 <kparal> *experienced 16:26:29 <dgilmore> we can not guarantee that we can get the update pushed 16:26:31 <kparal> half of the people upgraded with an old broken version of system-upgrade 16:27:45 <kparal> ok, in that case I'd Go/No-Go call to consider whether the 0day blockers for previous stable releases are already *pushed*, not just scheduled for pushing as sgallagh proposed 16:28:22 <adamw> that seems like the safest case, but comes with an obvious risk of additional delays 16:28:31 <kparal> I know 16:29:02 <dgilmore> kparal: it would need karma and all sorts of things 16:29:03 <sgallagh> Right, but delays generally cost us less than broken upgrades in terms of user goodwill 16:29:03 <kparal> and we need to learn how exactly mirrormanager decides which mirror is "ok" and which is outdated 16:29:15 <dgilmore> kparal: we can not guarantee when things will get oout 16:29:30 <sgallagh> dgilmore: When he says "scheduled for pushing", he means "Stable push is requested" 16:29:40 <sgallagh> Don't we always do a stable 0day push before release? 16:29:41 <kparal> dgilmore: that's why I'd like to be sure that Go is voted only once the packages are really pushed 16:29:51 <dgilmore> sgallagh: "scheduled for pushing" does not mean that 16:29:52 <sgallagh> I mean, I can understand that it's always possible that push could break 16:29:58 <dgilmore> sgallagh: it can meen to testing 16:30:09 <sgallagh> dgilmore: I'm translating it because the terminology was confusing 16:30:15 <dgilmore> sgallagh: so you need to say "scheduled for pushing to stable" 16:30:28 <kparal> dgilmore: that's what I meant, sorry 16:30:30 <sgallagh> My proposal was specifically that at go/no-go, it has to be requested for stable push already 16:31:02 <sgallagh> Apologies if terminology got in the way of clarity 16:31:07 <dgilmore> sgallagh: we still can not guarantee that it will go out in time 16:31:25 <sgallagh> dgilmore: We don't strictly *guarantee* anything 16:31:28 <dgilmore> shit happens and pushes can take days to get out 16:31:31 * danofsatx is unable to participate today and sends regrets. 16:31:35 <sgallagh> But it's still more consistent than what we have now 16:31:41 <dgilmore> sgallagh: kparal is asking for a guarantee 16:31:58 <sgallagh> kparal is used to being disappointed, I think ;-) 16:32:00 <dgilmore> I am saying we can not give it 16:32:01 <kparal> well, I'm asking for delaying the announcement until everything is ready 16:32:14 <kparal> whether we delay go/no-go decision or the announcement is not important 16:32:43 <kparal> but we should wait a few days until it is really pushed and available and not served by outdated mirrors 16:32:54 <kparal> but blessed by mirrormanager 16:33:10 <adamw> i think that under the current process, the go/no-go decision is treated as a guarantee of a release on tuesday by various parties 16:33:50 <adamw> changing that would be a significant thing we'd need to check is OK with them (mirrors, press, maybe some other folks - mattdm would likely know in more detail) 16:33:51 <kparal> that's why I proposed the outcome should be no-go in this case, but some parts of releng process would be granted to be processed 16:34:02 <kparal> like pushing the composes to correct trees and such 16:34:14 <linuxmodder> kparal, how much of a delay tho 16:34:20 <adamw> that would be safe as a process change, i think 16:34:25 <linuxmodder> how much is okay rather 16:34:28 <kparal> until the blocker updates are pushed 16:34:42 <kparal> we can re-evaluate every one or two days, if needed 16:35:08 <kparal> unless thursday and tuesday is set in stone 16:35:29 <kparal> or important for some reason I don't know about 16:36:11 <adamw> when the idea of <1 week slips has come up before there's always been some reason why we can't do it 16:36:16 <adamw> but i'm not up on the details 16:36:37 <tflink> the biggest reason I know of is churn 16:36:56 * linuxmodder thinks he missed the larger issue at hand ....reading back it seems that the issue is having mirrors not well managed for things like release days and 0-days 16:36:58 <tflink> er, load on qa and releng 16:37:36 <kparal> that doesn't seem to apply in this case 16:37:56 <kparal> because everything would be tested and ready, and we would just wait for some specific updated to be pushed to mirrors 16:38:01 <tflink> sure but that wasn't the question :) 16:38:01 <kparal> *updates 16:38:28 <adamw> linuxmodder: the basic problem we're trying to fix is "shipping before updates needed to fix blocker bugs have been released" 16:38:41 <adamw> we are good at making sure all blocker fixes that need to be on the release media are on them 16:38:53 <adamw> we're not so good at making sure all blocker fixes that need to be in updates repositories are there in time 16:38:56 <linuxmodder> but not to the mirrors? 16:40:28 <kparal> linuxmodder: there is some additional time that it takes for updates to reach the mirrors, and the mirrormanager to say whether they are up-to-date or not 16:40:59 <linuxmodder> kparal, I have a private mirror I'm familiar with that 16:41:00 <kparal> afaik, the previous push metadata are still considered up-to-date 16:41:13 <kparal> not sure if there's a time constraint 16:41:16 <linuxmodder> s/have/keep one 16:41:18 <adamw> so let's see, i'm, trying to think what we can move forward in this meeting 16:41:25 <kparal> so either we need to pushes to occur, or 2 days, or something along that lines 16:41:38 <kparal> *two pushes 16:41:47 <kparal> adamw: yep 16:42:05 <adamw> i guess we can ask: are we generally in favour of changing the process to include some kind of check on the status of blocker-fixing updates as an input to go/no-go ? 16:42:27 <tflink> +1 16:42:32 <kparal> +1 here 16:42:33 <adamw> i'm a bit worried about the prospect for additional delays, but it does seem like we need something heavier than the current process 16:42:58 <linuxmodder> +! 16:43:28 <linuxmodder> maybe make it a seperate metadata check (not sure how that would be coded in tho) 16:43:36 <pschindl> +1 16:43:54 <adamw> linuxmodder: these decisions are made by squishy humanware 1.0, i'm afraid 16:43:56 <kparal> I'll try to investigate the details about mirrormanager 16:44:05 <kparal> :) 16:44:22 <linuxmodder> or only for those hosting the updates that carry it (we all know not all mirrors carry the full enchildada) 16:44:24 <roshi> +1 16:44:35 <adamw> ok 16:44:44 <linuxmodder> willing to help look myself 16:44:49 <adamw> #agreed QA is generally in favour of changing the process to include some kind of check on the status of blocker-fixing updates as an input to go/no-go 16:45:21 <adamw> kparal: do you want an action item to look into the details of what change is actually feasible and come up with a specific proposal for that part? 16:45:33 <kparal> can try 16:45:50 <kparal> give me an action item 16:47:16 <adamw> #action kparal to look further into the details of go/no-go process and propose a practical policy for changing it to cover blockers fixed by updates 16:47:40 <adamw> alrighty 16:47:58 <adamw> let's go on to open floor, where we can maybe quickly talk about kparal's new proposals 16:48:01 <adamw> #topic Open floor 16:48:56 <adamw> so kparal posted a couple of proposals about criteria/tests today 16:49:13 <adamw> i'm definitely +1 to dropping the UEFI/BIOS distinction for upgrade testing 16:50:08 <pschindl> make sense to me too. 16:50:28 <adamw> on N+2 upgrades - i'm OK with it if the tools folks are 16:50:35 <adamw> did you ask them, kparal? 16:50:55 <kparal> adamw: nope, not yet 16:51:09 <kparal> is there something else than system-upgrade? 16:51:26 <adamw> don't think so 16:51:39 <adamw> i guess systemd is possibly involved, but it seems a fairly simple design 16:51:39 <kparal> I think wwoods_ is operating in the same vacuum as we are - no clear rules whether we support N+2 or not 16:51:57 <adamw> kparal: well the question would be 'are they OK with us specifying that it *is* supported' 16:52:27 <kparal> I kind of believe fedora project supports it as a whole, we just don't test it. at least that was my impression all those years 16:52:36 <kparal> but we can run this through fesco to have clear rules 16:53:04 <adamw> my memory is that we didn't used to support it because we felt it wasn't really viable, but with dnf-system-upgrade i suspect that might be different 16:53:04 <kparal> I just didn't want to start the whole bureacratic process if not needed 16:53:11 <kparal> ok 16:53:43 <kparal> so I'll discuss it with wwoods_, and if he agrees to support it, there's no problem. if he wants fesco's confirmation, I'll file a fesco ticket 16:53:59 <adamw> sounds good 16:54:19 <kparal> if he doesn't want to support it at all, we should warn the users or disable that option right away 16:54:21 <adamw> anyone opposed to supporting N+2 upgrades, speak now or forever hold your peace (or, you know, just write an email, I guess) 16:55:21 <kparal> adamw: more action items for me, please 16:55:36 <adamw> yay, it's a kparal action week 16:55:51 <adamw> #action kparal to check with dnf-system-upgrade maintainers if they're OK with blocking on N+2 upgrades 16:56:25 <kparal> adamw: it's to make you feel not lonely next week during previous week follow-up :) 16:56:39 <kparal> it was boring when it was all adamw and no kparal 16:56:46 <adamw> yaaay 16:57:00 * adamw looks forward to it 16:57:08 <adamw> OK, we have blocker review in 3 minutes over in #fedora-blocker-review 16:57:11 <adamw> it'll just be a mini meeting 16:57:18 <adamw> anything else for open floor? 16:57:24 * roshi has nothing 16:57:49 <kparal> nope 16:58:34 <adamw> alrighty then! 16:58:38 <adamw> thanks for coming, everyone 16:58:43 * adamw sets the big fuse 17:00:00 <adamw> see you next week, everyone 17:00:04 <adamw> #endmeeting