16:00:05 <adamw> #startmeeting Fedora QA meeting
16:00:05 <zodbot> Meeting started Mon Nov 23 16:00:05 2015 UTC.  The chair is adamw. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:05 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:05 <zodbot> The meeting name has been set to 'fedora_qa_meeting'
16:00:09 <adamw> #meetingname fedora-qa
16:00:09 <zodbot> The meeting name has been set to 'fedora-qa'
16:00:13 <adamw> #topic Roll call
16:00:25 <adamw> ahoyhoy folks, who's around for the QA meeting?
16:00:37 * garretraziel is here
16:01:03 <garretraziel> pschindl will hopefully join us soon
16:01:05 * tflink is here
16:01:20 * kparal is here
16:01:26 * roshi is here
16:01:31 * satellit_e listening
16:01:46 * adamw abandons his fantasy of no-one saying anything so he can go back to bed
16:02:18 <adamw> how's everyone doing?
16:03:16 <adamw> bright and bushy-tailed, huh
16:03:17 <adamw> alrighty
16:03:21 <adamw> #topic Previous meeting follow-up
16:03:32 <kparal> we didn't want to say anything so that you can go to bed
16:03:55 <adamw> :P
16:03:58 <kparal> your fault :)
16:04:01 <garretraziel> yeah, but kparal ruined it
16:04:02 <kparal> I'm fine, thanks
16:04:13 <adamw> #info "adamw to draft up proposal for better handling of 'special blockers'" - yep, did that, and we'll be discussing the draft later in the meeting
16:04:20 <adamw> garretraziel: doesn't he always? jeez, that guy.
16:04:21 <kparal> garretraziel: do you think he would realize after 10-15 minutes?
16:05:46 * pschindl is here (late from train)
16:05:50 <garretraziel> oh, here he comes, my job here is done
16:06:18 <adamw> #info "adamw to finish off the 'installer help' criteria / test case changes" - did that: can't find the link in new hyperkitty mailing list archive, but trust me, I did it
16:07:03 <adamw> #info "adamw to work with releng to get rawhide nightly boot.iso compose working and create an initial f24 nightly validation event" - did that too, nirik fixed all the things stopping nightly composes and we have nightly validation events going now
16:07:06 <adamw> any other follow-up?
16:07:15 * adamw does his one-man show again
16:07:59 * satellit_e lives still failing in koji  and boot.iso fails in plasma  today
16:09:03 <adamw> satellit: probably more dep failures, i guess? did you check the logs?
16:09:10 <adamw> also what do you mean by 'boot.iso fails in plasma'?
16:09:23 <satellit_e> open office seems broken with depends
16:10:03 <satellit_e> plasma in boot.iso boots but ends up with black screen with cursor
16:10:10 <adamw> ah, yeah, the python 3.5 rebuild failed.
16:10:13 * satellit_e virt manager
16:10:19 <adamw> satellit: you mean it fails post-install ?
16:10:24 <satellit_e> yes
16:11:08 <satellit_e> correction no...at boot
16:11:15 <satellit_e> of live
16:11:50 <satellit_e> boot.iso install finishes and logs in then black screen and cursor
16:12:26 <adamw> right, OK. did you report the boot.iso failure?
16:12:52 <satellit_e> not today
16:13:41 <satellit> https://fedoraproject.org/wiki/Test_Results:Fedora_24_Rawhide_20151120_Installation
16:13:44 <adamw> ok, we should probably get a bug report in, i can see if i can reproudce the problem later...thanks
16:14:08 <adamw> #info satellit reports ongoing problems with Rawhide image compose and install
16:14:09 <adamw> so, onto our first topic, I guess...
16:14:28 <adamw> #topic Non-media blocker process
16:15:48 <adamw> so for background, I submitted an initial dump of ideas for changing the process for handling blocker bugs that don't need fixing on the release media
16:16:05 <adamw> man, i picked a bad time to learn hyperkitty, live in the middle of a meeting...:P
16:17:07 * kparal already struggled with the admin interface
16:17:29 <adamw> alright, so, er, no link to the first mail, let's just hope everyone has it in their mail client :)
16:17:43 <adamw> there's been some wide-ranging discussion on the thread so it seemed like a good topic to kick around in a meeting
16:17:45 <roshi> aw, man, I have to load my mail client...
16:17:49 <roshi> such a bother...
16:17:55 <adamw> SORRY ROSHI
16:18:00 * adamw plays tiny violin
16:18:09 <roshi> sheesh, demanding canadians...
16:18:12 <adamw> =)
16:18:16 <adamw> #info roshi is lame
16:18:38 <roshi> I'm certainly not feeling spry, that's for sure :p
16:19:04 <adamw> so i'd say the big discussion has been on the question of what we should do in terms of 'enforcing' non-image blockers
16:19:29 <adamw> currently we kind of wave our hands at them and say 'lo! a fix shall appear before release!' and then we forget about them, which seems like it could stand improvement
16:19:57 <kparal> I'm not aware whether releng has some processes to go along with this, but my comments were mostly about the fact that we'll probably need those as well
16:20:38 <adamw> nirik: dgilmore: around?
16:21:01 <roshi> I haven't had a chance to look at the PDC project much yet, but perhaps that has some bits and bytes that can help track all this?
16:21:06 <adamw> so aiui, releng definitely makes sure a 0-day update push *happens*
16:21:19 <kparal> ok, that's good
16:21:19 <adamw> roshi: i don't know much about it either
16:21:28 <roshi> Product Definition Center - which will have lists of all our images and whatnot
16:21:35 <adamw> roshi: i know what it *is*, yes.
16:21:37 <kparal> I'd be interested to know what went wrong in our system-upgrade.fc22 case
16:21:45 * roshi was going to work with threebean on getting it populated with data
16:21:50 <kparal> so that we can avoid it next time
16:21:52 <nirik> yes, as soon as the release is tagged in koji, we can do a updates push... usually thats the next day...
16:21:59 <roshi> adamw: that was more for others than for you :p
16:22:06 <adamw> =)
16:22:08 <dgilmore> adamw: what is up?
16:22:32 <adamw> dgilmore: we're considering the topic of non-media release blockers ('special blockers')
16:22:56 <adamw> kparal: was interested in what policies releng has for update pushes around stable releases (right kparal?)
16:23:21 <kparal> nirik: dgilmore: do you have a process that makes sure that 0day updates are pushed (and mirrored, and mirrormanager does no longer point to outdated mirrors) before the release announcement is published?
16:23:51 <dgilmore> adamw: we have no policies outside of making sure that everything used in teh release is pushed stable
16:23:52 <kparal> I'm talking about that 1-2 day period when mirrormanager can still return an outdated mirror (the previous metadata)
16:23:53 <adamw> kparal: note, 0-day applies only to the release itself; I don't know whether there's any rule about getting updates for the previous stable releases pushed
16:24:07 <kparal> ah
16:24:08 <adamw> dgilmore: so 0-day updates are basically a best-effort?
16:24:18 <dgilmore> adamw: we try and do 0 day pushes, but if they do not happen they do not happen
16:24:23 <kparal> ok, I was talking about previous stable releases
16:24:25 <dgilmore> adamw: they are
16:25:04 <dgilmore> kparal: if we need say a dnf update in a previous release for upgrades to work, they are on a best effort basiss
16:25:16 <kparal> would you be open to some changes, to make sure important updates in previous stable releases are pushed before the new release announcement goes live?
16:26:00 <dgilmore> kparal: we would not be. too much can go wrong in updates pushing.  updates pushing are a best effort basis and that is not going to change without major changes in how we push updates
16:26:02 <kparal> because we experience a very unfortunate mess-up with system-upgrade.fc22 this time
16:26:08 <kparal> *experienced
16:26:29 <dgilmore> we can not guarantee that we can get the update pushed
16:26:31 <kparal> half of the people upgraded with an old broken version of system-upgrade
16:27:45 <kparal> ok, in that case I'd Go/No-Go call to consider whether the 0day blockers for previous stable releases are already *pushed*, not just scheduled for pushing as sgallagh proposed
16:28:22 <adamw> that seems like the safest case, but comes with an obvious risk of additional delays
16:28:31 <kparal> I know
16:29:02 <dgilmore> kparal: it would need karma and all sorts of things
16:29:03 <sgallagh> Right, but delays generally cost us less than broken upgrades in terms of user goodwill
16:29:03 <kparal> and we need to learn how exactly mirrormanager decides which mirror is "ok" and which is outdated
16:29:15 <dgilmore> kparal: we can not guarantee when things will get oout
16:29:30 <sgallagh> dgilmore: When he says "scheduled for pushing", he means "Stable push is requested"
16:29:40 <sgallagh> Don't we always do a stable 0day push before release?
16:29:41 <kparal> dgilmore: that's why I'd like to be sure that Go is voted only once the packages are really pushed
16:29:51 <dgilmore> sgallagh: "scheduled for pushing" does not mean that
16:29:52 <sgallagh> I mean, I can understand that it's always possible that push could break
16:29:58 <dgilmore> sgallagh: it can meen to testing
16:30:09 <sgallagh> dgilmore: I'm translating it because the terminology was confusing
16:30:15 <dgilmore> sgallagh: so you need to say "scheduled for pushing to stable"
16:30:28 <kparal> dgilmore: that's what I meant, sorry
16:30:30 <sgallagh> My proposal was specifically that at go/no-go, it has to be requested for stable push already
16:31:02 <sgallagh> Apologies if terminology got in the way of clarity
16:31:07 <dgilmore> sgallagh: we still can not guarantee that it will go out in time
16:31:25 <sgallagh> dgilmore: We don't strictly *guarantee* anything
16:31:28 <dgilmore> shit happens and pushes can take days to get out
16:31:31 * danofsatx is unable to participate today and sends regrets.
16:31:35 <sgallagh> But it's still more consistent than what we have now
16:31:41 <dgilmore> sgallagh: kparal is asking for a guarantee
16:31:58 <sgallagh> kparal is used to being disappointed, I think ;-)
16:32:00 <dgilmore> I am saying we can not give it
16:32:01 <kparal> well, I'm asking for delaying the announcement until everything is ready
16:32:14 <kparal> whether we delay go/no-go decision or the announcement is not important
16:32:43 <kparal> but we should wait a few days until it is really pushed and available and not served by outdated mirrors
16:32:54 <kparal> but blessed by mirrormanager
16:33:10 <adamw> i think that under the current process, the go/no-go decision is treated as a guarantee of a release on tuesday by various parties
16:33:50 <adamw> changing that would be a significant thing we'd need to check is OK with them (mirrors, press, maybe some other folks - mattdm would likely know in more detail)
16:33:51 <kparal> that's why I proposed the outcome should be no-go in this case, but some parts of releng process would be granted to be processed
16:34:02 <kparal> like pushing the composes to correct trees and such
16:34:14 <linuxmodder> kparal,  how much of a delay tho
16:34:20 <adamw> that would be safe as a process change, i think
16:34:25 <linuxmodder> how much is  okay rather
16:34:28 <kparal> until the blocker updates are pushed
16:34:42 <kparal> we can re-evaluate every one or two days, if needed
16:35:08 <kparal> unless thursday and tuesday is set in stone
16:35:29 <kparal> or important for some reason I don't know about
16:36:11 <adamw> when the idea of <1 week slips has come up before there's always been some reason why we can't do it
16:36:16 <adamw> but i'm not up on the details
16:36:37 <tflink> the biggest reason I know of is churn
16:36:56 * linuxmodder thinks he missed the larger issue at hand ....reading back it seems that the issue is having mirrors not  well managed for things like release days and 0-days
16:36:58 <tflink> er, load on qa and releng
16:37:36 <kparal> that doesn't seem to apply in this case
16:37:56 <kparal> because everything would be tested and ready, and we would just wait for some specific updated to be pushed to mirrors
16:38:01 <tflink> sure but that wasn't the question :)
16:38:01 <kparal> *updates
16:38:28 <adamw> linuxmodder: the basic problem we're trying to fix is "shipping before updates needed to fix blocker bugs have been released"
16:38:41 <adamw> we are good at making sure all blocker fixes that need to be on the release media are on them
16:38:53 <adamw> we're not so good at making sure all blocker fixes that need to be in updates repositories are there in time
16:38:56 <linuxmodder> but not to the mirrors?
16:40:28 <kparal> linuxmodder: there is some additional time that it takes for updates to reach the mirrors, and the mirrormanager to say whether they are up-to-date or not
16:40:59 <linuxmodder> kparal,  I  have a private mirror I'm familiar with that
16:41:00 <kparal> afaik, the previous push metadata are still considered up-to-date
16:41:13 <kparal> not sure if there's a time constraint
16:41:16 <linuxmodder> s/have/keep one
16:41:18 <adamw> so let's see, i'm, trying to think what we can move forward in this meeting
16:41:25 <kparal> so either we need to pushes to occur, or 2 days, or something along that lines
16:41:38 <kparal> *two pushes
16:41:47 <kparal> adamw: yep
16:42:05 <adamw> i guess we can ask: are we generally in favour of changing the process to include some kind of check on the status of blocker-fixing updates as an input to go/no-go ?
16:42:27 <tflink> +1
16:42:32 <kparal> +1 here
16:42:33 <adamw> i'm a bit worried about the prospect for additional delays, but it does seem like we need something heavier than the current process
16:42:58 <linuxmodder> +!
16:43:28 <linuxmodder> maybe make it a seperate metadata check (not sure how that would be coded in tho)
16:43:36 <pschindl> +1
16:43:54 <adamw> linuxmodder: these decisions are made by squishy humanware 1.0, i'm afraid
16:43:56 <kparal> I'll try to investigate the details about mirrormanager
16:44:05 <kparal> :)
16:44:22 <linuxmodder> or only for those  hosting the  updates that carry it (we all know not all mirrors carry the full enchildada)
16:44:24 <roshi> +1
16:44:35 <adamw> ok
16:44:44 <linuxmodder> willing to help look myself
16:44:49 <adamw> #agreed QA is generally in favour of changing the process to include some kind of check on the status of blocker-fixing updates as an input to go/no-go
16:45:21 <adamw> kparal: do you want an action item to look into the details of what change is actually feasible and come up with a specific proposal for that part?
16:45:33 <kparal> can try
16:45:50 <kparal> give me an action item
16:47:16 <adamw> #action kparal to look further into the details of go/no-go process and propose a practical policy for changing it to cover blockers fixed by updates
16:47:40 <adamw> alrighty
16:47:58 <adamw> let's go on to open floor, where we can maybe quickly talk about kparal's new proposals
16:48:01 <adamw> #topic Open floor
16:48:56 <adamw> so kparal posted a couple of proposals about criteria/tests today
16:49:13 <adamw> i'm definitely +1 to dropping the UEFI/BIOS distinction for upgrade testing
16:50:08 <pschindl> make sense to me too.
16:50:28 <adamw> on N+2 upgrades - i'm OK with it if the tools folks are
16:50:35 <adamw> did you ask them, kparal?
16:50:55 <kparal> adamw: nope, not yet
16:51:09 <kparal> is there something else than system-upgrade?
16:51:26 <adamw> don't think so
16:51:39 <adamw> i guess systemd is possibly involved, but it seems a fairly simple design
16:51:39 <kparal> I think wwoods_ is operating in the same vacuum as we are - no clear rules whether we support N+2 or not
16:51:57 <adamw> kparal: well the question would be 'are they OK with us specifying that it *is* supported'
16:52:27 <kparal> I kind of believe fedora project supports it as a whole, we just don't test it. at least that was my impression all those years
16:52:36 <kparal> but we can run this through fesco to have clear rules
16:53:04 <adamw> my memory is that we didn't used to support it because we felt it wasn't really viable, but with dnf-system-upgrade i suspect that might be different
16:53:04 <kparal> I just didn't want to start the whole bureacratic process if not needed
16:53:11 <kparal> ok
16:53:43 <kparal> so I'll discuss it with wwoods_, and if he agrees to support it, there's no problem. if he wants fesco's confirmation, I'll file a fesco ticket
16:53:59 <adamw> sounds good
16:54:19 <kparal> if he doesn't want to support it at all, we should warn the users or disable that option right away
16:54:21 <adamw> anyone opposed to supporting N+2 upgrades, speak now or forever hold your peace (or, you know, just write an email, I guess)
16:55:21 <kparal> adamw: more action items for me, please
16:55:36 <adamw> yay, it's a kparal action week
16:55:51 <adamw> #action kparal to check with dnf-system-upgrade maintainers if they're OK with blocking on N+2 upgrades
16:56:25 <kparal> adamw: it's to make you feel not lonely next week during previous week follow-up :)
16:56:39 <kparal> it was boring when it was all adamw and no kparal
16:56:46 <adamw> yaaay
16:57:00 * adamw looks forward to it
16:57:08 <adamw> OK, we have blocker review in 3 minutes over in #fedora-blocker-review
16:57:11 <adamw> it'll just be a mini meeting
16:57:18 <adamw> anything else for open floor?
16:57:24 * roshi has nothing
16:57:49 <kparal> nope
16:58:34 <adamw> alrighty then!
16:58:38 <adamw> thanks for coming, everyone
16:58:43 * adamw sets the big fuse
17:00:00 <adamw> see you next week, everyone
17:00:04 <adamw> #endmeeting