17:30:04 <jkurik> #startmeeting F24 Beta Go/No-Go meeting - 2
17:30:04 <zodbot> Meeting started Thu May  5 17:30:04 2016 UTC.  The chair is jkurik. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:30:04 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:30:04 <zodbot> The meeting name has been set to 'f24_beta_go/no-go_meeting_-_2'
17:30:06 <jkurik> #meetingname F24-Beta-Go_No_Go-meeting
17:30:06 <zodbot> The meeting name has been set to 'f24-beta-go_no_go-meeting'
17:30:07 <jkurik> #topic Roll Call
17:30:09 <jkurik> .hello jkurik
17:30:10 <zodbot> jkurik: jkurik 'Jan Kurik' <jkurik@redhat.com>
17:30:36 * kparal is around
17:30:45 * garretraziel is lurking
17:30:46 <linuxmodder> .hello linuxmodder
17:30:48 <zodbot> linuxmodder: linuxmodder 'Corey W Sheldon' <sheldon.corey@openmailbox.org>
17:30:59 * linuxmodder is on  iffy connect  may  drop out at times
17:30:59 <jkurik> adamw, sumantro, roshi, nirik, dgilmore - anyone around ?
17:31:07 <nirik> morning
17:31:09 <adamw> .hello adamwill
17:31:10 <zodbot> adamw: adamwill 'Adam Williamson' <awilliam@redhat.com>
17:31:18 <coremodule> .hello coremodule
17:31:19 <zodbot> coremodule: coremodule 'Geoffrey Marr' <geoff-marr@hotmail.com>
17:32:11 <jkurik> kparal, garretraziel, linuxmodder: hi
17:32:46 <linuxmodder> o/
17:32:47 <jkurik> hi adamw
17:32:59 <jkurik> coremodule: wellcome
17:33:04 * pschindl is here
17:33:14 <adamw> don't mind me, i'm just combining openvswitch crap i don't understand with ansible crap i don't understand
17:33:17 <adamw> i am confident this will end well
17:33:22 <linuxmodder> and now everyone comes out of the woodwork :)
17:33:39 <jkurik> so, we need someone from FESCo and releng
17:33:52 * nirik is here for both or either
17:33:58 <jkurik> great
17:34:12 <jkurik> wellcome everyone here
17:34:37 <jkurik> #chair dgilmore nirik adamw kparal
17:34:37 <zodbot> Current chairs: adamw dgilmore jkurik kparal nirik
17:34:47 <jkurik> #topic Purpose of this meeting
17:34:53 <jkurik> #info Purpose of this meeting is to check whether or not F24 Beta is ready for shipment, according to the release criteria.
17:34:58 <jkurik> #info This is determined in a few ways:
17:35:07 <jkurik> #info No remaining blocker bugs
17:35:08 <jkurik> #info Release candidate compose is available (check https://fedorahosted.org/rel-eng/ticket/6405 )
17:35:18 <jkurik> #info Test matrices for Beta are fully completed
17:35:19 <jkurik> #link https://qa.fedoraproject.org/blockerbugs/milestone/24/beta/buglist
17:35:21 <jkurik> #link https://fedoraproject.org/wiki/Test_Results:Fedora_24_Beta_1.6_Summary
17:35:22 <jkurik> #link https://fedoraproject.org/wiki/Test_Results:Fedora_24_Beta_1.6_Installation
17:35:24 <jkurik> #link https://fedoraproject.org/wiki/Test_Results:Fedora_24_Beta_1.6_Base
17:35:25 <jkurik> #link https://fedoraproject.org/wiki/Test_Results:Fedora_24_Beta_1.6_Server
17:35:27 <jkurik> #link https://fedoraproject.org/wiki/Test_Results:Fedora_24_Beta_1.6_Cloud
17:35:28 <jkurik> #link https://fedoraproject.org/wiki/Test_Results:Fedora_24_Beta_1.6_Desktop
17:35:30 <jkurik> #link https://fedoraproject.org/wiki/Test_Results:Fedora_24_Beta_1.6_Security_Lab
17:35:48 <jkurik> adamw, kparal: just to be sure, can you please confirm the 1.6 compose is the latest one ?
17:35:52 <adamw> it is
17:35:56 * nirik nods
17:35:58 <adamw> that does not necessarily mean we are shipping it...
17:36:01 <dgilmore> hi all
17:36:02 <jkurik> the requested 1.7 has never been done, right ?
17:36:04 <adamw> but let's get to that in a minute!
17:36:05 <adamw> no indeed
17:36:15 <jkurik> ok, thanks
17:36:27 <dgilmore> jkurik: it was closed as invalid, we can not do what was asked
17:36:31 <sgallagh> .hello sgallagh
17:36:32 <zodbot> sgallagh: sgallagh 'Stephen Gallagher' <sgallagh@redhat.com>
17:36:47 <jkurik> dgilmore: yes, I just wanted to be sure, thanks
17:36:53 <jkurik> #topic Current status
17:37:14 <jkurik> #info The last compose (1.6) is missing XFce images. (note: XFce images are not release blocking)
17:37:21 <jkurik> #link https://fedorahosted.org/rel-eng/ticket/6405#comment:9
17:37:27 <jkurik> #info We also have four proposed blockers
17:37:32 <jkurik> #link https://qa.fedoraproject.org/blockerbugs/milestone/24/beta/buglist
17:37:38 <nirik> it's also missing a few other images too sadly.
17:38:15 <jkurik> :(
17:38:33 <nirik> well, SoAS at least.
17:38:53 <jkurik> ok, so can we start with blocker review or are we going to discuss the missing images first ?
17:38:57 <adamw> blocker review first i think
17:39:04 <adamw> if we're blocked we're blocked, images don't matter
17:39:21 <sgallagh> jkurik: XFCE images are release blocking for arm, aren't they?
17:39:26 <jkurik> as far as I can say, all the release blocking images are available
17:39:27 <nirik> sgallagh: yes, and we have those.
17:39:32 <sgallagh> ok
17:39:49 <jkurik> ok, so...the blocker review
17:39:59 <jkurik> #topic Mini-Blocker Review
17:40:15 <jkurik> adamw, kparal: can anyone of you please chair it ?
17:40:19 <adamw> sure
17:40:22 <jkurik> thanks
17:40:39 <adamw> note: these first two bugs are quite similar
17:40:44 <adamw> #topic (1259953) _ped.IOException: Partition(s) 8 on /dev/mmcblk0 have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before ...
17:40:44 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=1259953
17:40:44 <adamw> #info Proposed Blocker, anaconda, NEW
17:41:02 <adamw> pschindl: are you around?
17:41:13 <adamw> ok, so both of these bugs are RAID related
17:41:25 <adamw> they are both triggered in quite similar ways and have similar results
17:41:35 <pschindl> adamw: I'm here.
17:41:43 <adamw> basically the test is to do an LVM on RAID install, then immediately try and do it again over the top
17:41:49 <pschindl> I tried to reproduce it today.
17:41:51 <sgallagh> bugs. RAID-related. The puns just write themselves.
17:41:55 <adamw> sgallagh: =)
17:42:00 <nirik> ha
17:42:24 <adamw> this first bug is what happens when you do this on *firmware* RAID
17:42:38 <pschindl> And even though I was able to reproduce it from time to time it doesn't happen everytime I tried to install with deleting old lvm on raid.
17:42:52 <adamw> pschindl: i hit it starting from scratch
17:42:59 <sgallagh> adamw: Presumably hardware RAID is sufficiently abstracted that it doesn't happen?
17:43:01 <adamw> but this error is inherently a bit racey, i think
17:43:14 <adamw> sgallagh: yes, hardware RAID so far as the kernel is concerned is just a simple block device.
17:43:18 <adamw> it shows up as sda or whatever.
17:43:53 <adamw> so this basically happens if you start with two clean disks, create a firmware RAID array of them, do a default install to it, then reboot and try to do a default install deleting the previous install
17:44:13 <sgallagh> adamw: I know that's how it should be, but then crap like fakeraid gets into "real" hardware RAID controllers and ruins lives.
17:44:27 <pschindl> But as I said in the bug. Failed installation destroys those disks, so user can restart and try again (this time using free space) as workaround.
17:44:33 <adamw> which is a fairly reasonable scenario (e.g. you had a default install of F23 to your firmware RAID and you want to blow it away and do an F24 install)
17:44:35 <adamw> yes
17:44:39 <adamw> i confirmed that too
17:45:00 <adamw> after hitting the crash if you reboot and try again, it works (at least it did for both me and pschindl when we hit this)
17:45:18 <pschindl> + it doesn't happen everytime I tried. So I'm more -1 for beta.
17:45:35 <adamw> just as a note, testing this stuff is a nightmare because both RAID and LVM tend to leave stale metadata lying around disks; it's very easy to get into a scenario where some random old metadata is screwing stuff up. i had to completely zero both my test disks to get a clean reproducer of this yesterday.
17:45:49 <sgallagh> Is that because the failure the first time deletes whatever is causing it to fail?
17:45:54 <adamw> yes, in this case
17:46:05 <adamw> when you boot the third time, anaconda sees the raid set as empty (no partitions)
17:46:19 <adamw> so the first reinstall attempt seems to crash after it's wiped the partitions
17:46:20 <sgallagh> OK, so as long as it's "Retry once" not "Just keep trying, eventually it will work"
17:46:33 <adamw> with these bugs nothing is entirely guaranteed, but that's our understanding so far
17:46:44 <sgallagh> In that case, I feel like for Beta I could be comfortable handwaving this as a CommonBug
17:46:50 <adamw> so i agree that i think this one is livable-with for Beta, we can document the 'just reboot and try again' workaround
17:46:59 <dgilmore> ugly but I think given the cicumstances rare and should be Beta FE and Final Blocker
17:47:06 * nirik nods. ok by me
17:47:08 <adamw> heck, in this non-ideal world i might even say that's OK for final, given how freaking messy fwraid/lvm are.
17:47:11 <pschindl> The disks seem to be destroyed but the kernel doesn't listen. So after reboot it is free.
17:47:33 <jkurik> we can consider the scenarion "Retry once" - not - "Just keep trying" as a workaround :)
17:47:35 <sgallagh> /me wonders if we could kexec mid-install to get around it :-D
17:47:37 <garretraziel> (not sure whether "destroyed" is proper term :-) )
17:47:40 <adamw> the more i test this, the more i reckon the only reliable approach is to reuse the existing devices or nuke the entire goddamn thing with dd before attempting to install...=)
17:48:33 <pschindl> garretraziel: eh, partitions are destroyed, not the disks :)
17:48:34 <jkurik> sounds like we can live with this for Beta and do not block on this , right ?
17:48:55 <adamw> yeah, that seems like the majority opinion
17:49:06 * kparal agrees
17:49:06 <adamw> let's not deal with final blocker status at this meeting
17:49:14 <dgilmore> adamw: that is fine
17:49:16 <dgilmore> +1 FE
17:49:33 <pschindl> +1 FE and repropose it as final?
17:49:41 <dgilmore> sure
17:49:46 <sgallagh> -1 blocker, +1 FE in case we do another spin anyway.
17:49:56 <adamw> proposed #agreed 1259953 - RejectedBlocker (Beta) - this is annoying but it's a fairly specific scenario with an apparently safe workaround that we can document, which we reckon is sufficiently good enough for Beta
17:50:02 <nirik> ack
17:50:03 <jkurik> -1 blocker, +1 FE
17:50:04 <adamw> patch
17:50:14 <adamw> proposed #agreed 1259953 - RejectedBlocker (Beta) AcceptedFreezeException (Beta) - this is annoying but it's a fairly specific scenario with an apparently safe workaround that we can document, which we reckon is sufficiently good enough for Beta
17:50:20 <pschindl> ack
17:50:21 <nirik> sure. ack
17:50:30 <dgilmore> ack
17:50:37 <jkurik> ack
17:50:51 <sgallagh> ack
17:51:44 <kparal> ack
17:51:47 <garretraziel> ack
17:52:10 <adamw> #agreed 1259953 - RejectedBlocker (Beta) AcceptedFreezeException (Beta) - this is annoying but it's a fairly specific scenario with an apparently safe workaround that we can document, which we reckon is sufficiently good enough for Beta
17:52:18 <adamw> #topic (1333131) gi.overrides.BlockDev.MDRaidError: Process reported exit code 256: mdadm: Cannot get exclusive access to /dev/md/pv00:Perhaps a running process, mounted filesystem or active volume group?
17:52:18 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=1333131
17:52:18 <adamw> #info Proposed Blocker, anaconda, NEW
17:52:29 <adamw> this is a similar one, only in this case the scenario is LVM on *software* RAID
17:52:40 <adamw> it's also worth noting that when pschindl tried to reproduce this he didn't hit the same bug, but a different one
17:52:49 <adamw> i did hit this one twice in my testing, though
17:52:54 <nirik> ditto from last one (although this is likely more common than firmware raid)
17:53:03 <adamw> for me it similarly worked after a restart, even though anaconda showed the partitions as still present
17:53:09 <adamw> i meant to re-run this a few times this morning but got distracted :/
17:53:16 <adamw> pschindl: how many times have you tried this case? just once?
17:53:34 <pschindl> adamw: just once.
17:53:49 <adamw> kk
17:54:01 * adamw runs it again, what the heck
17:54:27 <adamw> i'm gonna try and put this case into openqa For The Future.
17:55:02 <dgilmore> #action adamw to add test case to openqa covering this
17:55:10 <jkurik> adamw++
17:55:20 <adamw> dgilmore: hah, joke's on you - no-one ever looks at those action items
17:55:20 <adamw> :P
17:55:59 <adamw> pschindl: did you do RAID-0 or RAID-1?
17:56:08 <pschindl> raid 1
17:56:27 * jkurik puts on his todo list to chase adamw to finish the actions from this meeting
17:56:27 <pschindl> when I tried it with raid 0 yesterday it doesn't appear.
17:56:42 <dgilmore> adamw++
17:57:01 <adamw> #action jkurik to...
17:57:01 <adamw> :P
17:57:27 <jkurik> :)
17:57:51 <adamw> ok, so i just ran two more times and got the same sequence
17:58:05 <adamw> so i've had clean install -> reinstall 1 crash -> reinstall 2 succeed -> reinstall 3 crash -> reinstall 4 succeed
17:58:09 <adamw> so that seems kinda reproducible
17:58:29 <dgilmore> adamw: I would say the same outcome as the previous bug
17:58:32 <adamw> yeah
17:58:35 * nirik too
17:58:44 <pschindl> adamw: And you uses RAID-1? or 0
17:58:48 <adamw> it just makes me wonder *why* it works on the second attempt since the first doesn't seem to wipe the devices, but whatever
17:58:50 <adamw> pschindl: 1
17:59:26 <pschindl> maybe it erases at lest something (metadata?)
17:59:53 <adamw> could be
18:00:02 <adamw> who knows, this is all terrible and i need whisky
18:00:04 <pschindl> anyway I'm -1 too. I looks very simmilar to previous bug.
18:00:28 <garretraziel> I'm also -1
18:00:53 <adamw> proposed #agreed 1333131 - RejectedBlocker (Beta) AcceptedFreezeException (Beta) - as per 1259953 this is a fairly well-identified case with an apparently reliable and simple workaround, we find that acceptable for Beta
18:01:05 <jkurik> ack
18:01:11 <nirik> ack
18:01:52 <pschindl> ack
18:01:54 <sgallagh> ack
18:01:56 <garretraziel> ack
18:02:01 <adamw> #agreed 1333131 - RejectedBlocker (Beta) AcceptedFreezeException (Beta) - as per 1259953 this is a fairly well-identified case with an apparently reliable and simple workaround, we find that acceptable for Beta
18:02:18 <adamw> #topic (1331630) Anaconda crashed during installation
18:02:18 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=1331630
18:02:18 <adamw> #info Proposed Blocker, python-blivet, VERIFIED
18:02:24 <adamw> bit of a preamble here
18:02:34 <adamw> this is a similar LVM/RAID-related bug and it was fixed in 1.6
18:02:37 <adamw> hence VERIFIED
18:02:52 <adamw> however, we do have a reason to vote on it: if we decided this was not a blocker we *could* choose to ship 1.4
18:03:21 <jkurik> adamw: what will be the benefit of shipping 1.4 ?
18:03:37 <sgallagh> adamw: The missing images from 1.6: those are present in 1.4?
18:04:06 <jkurik> sgallagh: ah, good point
18:04:16 <adamw> the diff is this:
18:05:01 <adamw> 1.4 has Xfce, 1.6 does not
18:05:13 <adamw> 1.4 has design suite, 1.6 does not
18:05:23 <adamw> 1.6 has security lab, 1.4 does not
18:05:26 <adamw> i don't think either has SoaS>
18:05:37 <sgallagh> ...
18:05:47 <kparal> I'd rather ship 1.6 and the missing images from 1.4, if available
18:05:54 <kparal> with a note in common bugs
18:05:54 <adamw> we can't really do that
18:05:56 <dgilmore> SoaS gets hit a lot by the lmc bug
18:06:06 <adamw> the composes are units, they have metadata and get registered into PDC
18:06:38 <dgilmore> we could lie
18:06:40 <adamw> we can't say 'F24 Beta is Beta-1.6 with a bit of Beta-1.4 sprinkled in'. well i guess we CAN, but the metadata nerds amongst us (koff koff me) would hate it.
18:06:41 <nirik> perhaps we could have websites put a note on them and point to the nightly? or ?
18:06:43 <sgallagh> Well, we should probably rule on this bug without pressure
18:06:48 <dgilmore> it just means what we ship does not match PDC :(
18:07:08 <adamw> i already hate that the compose gets split up between the main mirror space and alt
18:07:10 <adamw> ah, well
18:07:26 <adamw> still, all that aside: i think I'm +1 on this one because you *can't* get around this one by trying again
18:07:35 <adamw> it just blows up every time
18:07:40 <adamw> pschindl, was that your experience too?>
18:07:58 <sgallagh> Yeah, if this is a consistent crash, it seems like a pretty clear violation of the criteria
18:08:08 <dgilmore> I am not sure how much a pain it would be for websites if we mixed and matched
18:08:13 <pschindl> I vote +1 on this blocker. I was able to reproduce it quite often. And the anaconda crashed totaly (sigsegv) and no partitions deleted. So it means that next time you boot it, it will fail again.
18:08:17 <adamw> oh yeah it was: "I tried to run it again and it seems that I have computer in state when it happens in every attempt to install"
18:08:29 * nirik voted +1 in bug, and is still that.
18:09:07 <adamw> this one we hit on both lvm-on-fwraid and lvm-on-swraid, i think.
18:09:20 <jkurik> if we can not get around it, then +1 to block on this
18:09:51 <pschindl> I wasn't able to reproduce it with swraid, but I tried just raid 0 (twice).
18:10:01 <dgilmore> adamw: even a dd to wipe the disks hits it?
18:10:20 <adamw> dgilmore: no, it doesn't happen on a fresh install, it's only when anaconda has to wipe existing LVM bits
18:10:46 <adamw> but it seems pretty consistent when trying to install over an existing LVM-on-RAID install and when it happens it keeps happening, you can't just reboot and succeed
18:11:41 <pschindl> pvdelete fails and the bug was that anaconda tried to run it in never ending loop.
18:11:57 <adamw> proposed #agreed 1331630 - AcceptedBlocker (Beta) - this was another crasher in the case of installing over LVM-on-RAID, but it does not seem to have a workaround besides manually destroying the existing layout, so we consider it to be significant enough to constitute a criteria violation for Beta as you cannot complete the install
18:12:02 <dgilmore> adamw: so a workaround, albeit ugly is to wipe the disks
18:12:18 <adamw> dgilmore: right, but as i said that's actually pretty hard
18:12:19 <dgilmore> +1
18:12:38 <pschindl> ack
18:12:39 <kparal> ack
18:12:39 <jkurik> ack
18:12:44 <sgallagh> I'm perfectly comfortable saying that a "workaround" involving `dd` is not a workaround
18:12:49 <sgallagh> ack
18:12:57 * adamw would pay quite a lot for a magic 'find all raid/lvm metadata on this disk and delete it all, quickly' tool
18:13:01 <dgilmore> ack
18:13:05 <adamw> #agreed 1331630 - AcceptedBlocker (Beta) - this was another crasher in the case of installing over LVM-on-RAID, but it does not seem to have a workaround besides manually destroying the existing layout, so we consider it to be significant enough to constitute a criteria violation for Beta as you cannot complete the install
18:13:20 <pschindl> you could use wipefs in fact (that's what anaconda do now)
18:13:24 <adamw> dding my 500GiB test disks took like 5 hours, for the record
18:13:47 <adamw> pschindl: i'm not sure if that catches everything, but hey, next time i'll try it...
18:14:10 <dgilmore> adamw: dd if=/dev/zero of=/dev/sda bs=4M count=1
18:14:18 <cmurf> hdparm ata security erase, goes faster
18:14:20 <dgilmore> partprobe
18:14:45 <nirik> ack
18:14:55 <pschindl> adamw: if I understand to it correctly that is fallback in anaconda if pvremove doesn't work. They use wipefs -f. It should fail too but it doesn't :)
18:14:56 <sgallagh> dgilmore: I am terrified of someone reading this meeting log and copy-pasting that command verbatim.
18:15:04 <kparal> dgilmore: for gpt you also need to erase the backup table
18:15:17 <adamw> dgilmore: nope, that's not enough
18:15:23 <adamw> dgilmore: some metadata goes at the end of the disk
18:15:32 <adamw> and i have no idea where the hell lvm puts its bits
18:15:55 <adamw> i have a dd command to wipe the *last* few kb/mb/whatever you like of the disk, but i still wasn't 100% sure that'd get it all
18:15:56 <sgallagh> adamw: It hides them with steganography :-P
18:15:58 <cmurf> yes imsm metadata is at the end of the physical device
18:16:00 <nirik> newer mdraid is at the end of the disk with it's metadata yeah
18:16:42 <nirik> wipefs is your best bet IMHO...
18:16:58 <adamw> aaaanyhoo
18:17:03 <adamw> where were we
18:17:06 <adamw> oh yeah, one more
18:17:13 <dgilmore> sgallagh: they should know it is about destroying data
18:17:23 <adamw> #topic (1332623) breaks upgrade path
18:17:23 <adamw> #link https://bugzilla.redhat.com/show_bug.cgi?id=1332623
18:17:23 <adamw> #info Proposed Blocker, python-dateutil, ON_QA
18:18:52 <pschindl> -1 due to last two comments.
18:18:57 <sgallagh> Doesnt' look like a blocker to me.
18:19:40 <sgallagh> Or, I guess maybe it's a "special blocker"?
18:19:43 <kparal> garretraziel tried quite a lot to reproduce this
18:19:50 <kparal> -1 since we can't reproduce it
18:19:50 <sgallagh> As long as the fixed package was in the stable set by launch time
18:19:59 <jkurik> -1 as well
18:20:03 <sgallagh> But I'm -1
18:20:12 <dgilmore> -1
18:20:41 <kparal> also, it does not seem to be in a default package set
18:20:55 <garretraziel> I'm also -1 since I couldn't reproduce it
18:21:09 <adamw> -1, seems pretty clear from testing
18:21:28 <nirik> we don't use upgrade do we?
18:22:03 <sgallagh> nirik: EPARSE
18:22:20 <nirik> dnf system-upgrade uses distro-sync right?
18:22:51 <sgallagh> nirik: I'm not sure how that's relevant
18:23:24 <adamw> proposed #agreed 1332623 - RejectedBlocker (Beta) - no-one could reproduce this with one of the release-blocking package sets in multiple attempts
18:23:31 <jkurik> ack
18:23:40 <kparal> nirik: yes, but it stops as file conflicts same as regular dnf does
18:23:43 <kparal> *at
18:23:43 <nirik> then perhaps I don't understand the bug.
18:24:42 <kparal> this is not about a broken dep, but about a file conflict between two packages with ok deps
18:24:44 <sgallagh> nirik: I think what it's saying is that you couldn't install both python2-dateutil and python3-dateutil on the same machine.
18:24:46 <pschindl> ack
18:25:01 <sgallagh> We've established that this isn't a situation you could get into on upgrade, because python3-dateutil didn't exist on F23?
18:25:08 <nirik> ok, but I don't see what release critera this hits then
18:25:14 <nirik> anyhow, -1
18:25:16 <kparal> sgallagh: it's just not installed by default
18:25:29 <kparal> but even if we installed it manually, we didn't reproduce this
18:25:48 <sgallagh> kparal: OK, that last part is what leaves me puzzled.
18:25:54 <sgallagh> But I'm -1 anyway
18:25:57 <adamw> sgallagh: the question is pretty simply: does this error happen if you install any of the f22 or f23 package sets we support upgrades from, and run an upgrade
18:26:05 <adamw> the answer appears to be 'no'
18:26:06 <adamw> thus, not a blocker.
18:26:07 <kparal> -1
18:26:17 <adamw> any more acks?
18:26:23 <nirik> ack
18:26:23 <sgallagh> ack
18:26:25 <dgilmore> ack
18:26:37 <adamw> #agreed 1332623 - RejectedBlocker (Beta) - no-one could reproduce this with one of the release-blocking package sets in multiple attempts
18:26:57 <adamw> alright, that's all the proposed blockers
18:27:09 <jkurik> thanks adamw
18:27:13 <adamw> #info there are no outstanding blockers in Beta-1.6. there is one outstanding blocker in Beta-1.4
18:28:14 <jkurik> so if we are not going to ship 1.6 due to bug 1331630, can we ship 1.4 ?
18:28:21 <jkurik> or the 1.4 is blocked as well ?
18:28:40 <pschindl> 1.4 is blocked, 1.6 is not.
18:29:03 <adamw> it's 1.4 we can't ship due to 1331630.
18:29:06 <adamw> we can ship 1.6.
18:29:08 <jkurik> pschindl: bug #1331630 applies on 1.4 only ?
18:29:11 <adamw> yes.
18:29:13 <adamw> 1.6 fixed it.
18:29:15 <jkurik> ok, great
18:29:18 <jkurik> thanks
18:29:19 <adamw> the only sad thing about 1.6 is the missing xfce lives.
18:29:29 <jkurik> right
18:29:40 <sgallagh> Sad, but not release-blocking
18:29:43 <jkurik> so moving on ...
18:29:48 <jkurik> #topic Test Matrices coverage
18:29:57 <jkurik> #link https://fedoraproject.org/wiki/Test_Results:Fedora_24_Beta_1.6_Summary
18:30:21 <jkurik> can we now please check the Test matrices ?
18:30:48 <adamw> 1.6 had only one change compared to 1.4 (the 1331630 fix)
18:31:02 <adamw> so we can consider most 1.4 results valid for 1.6 where necessary
18:31:51 <dgilmore> adamw: do we have a merging of the somewhere?
18:32:09 <adamw> no, it has to be done manually and i haven't done it
18:32:47 <adamw> looking at them side-by-side, between 1.4 and 1.6 we are missing only most of the server tests, and 'audio basic' for ARM
18:33:10 <dgilmore> looks like none of the cloud deliverables have been tested at all
18:33:16 <adamw> we do however have a full set of server results for https://fedoraproject.org/wiki/Test_Results:Fedora_24_Branched_20160424.n.0_Server
18:33:25 <nirik> I think jds2001 did the AD test with server, but I could be wrong...
18:33:30 <adamw> and there was no significant change in the relevant packages between that compose and 1.6 i don't think
18:33:39 <adamw> oh yeah, still missing the AD tests :/
18:33:56 <adamw> it would be nice if someone could sanity check the 1.6 cloud images before we sign off
18:34:04 <pwhalen> adamw, can confirm audio on arm is working, will update
18:34:11 <adamw> ok thanks
18:34:27 * nirik can try a quick test in openstack...
18:34:39 <adamw> yeah, just a sanity check would be good
18:34:44 <adamw> make sure the images didn't somehow blow up
18:35:52 <adamw> jds2001: danofsatx: did either of you do the AD tests?
18:36:27 <adamw> per https://www.happyassassin.net/testcase_stats/24/Server.html the last time they got done was Alpha 1.6
18:37:45 <adamw> #info Active Directory client tests have not been run since Alpha
18:39:17 <jkurik> AD/IPA tests are release blocking right ?
18:39:32 <adamw> i'd be ok with a one-time exemption for those tests on the basis that the arrangements to run them fell through and we can't just do it at the drop of a hat due to the setup required. but we need to find a way to get them done for Final or we need to drop the relevant release criteria and no longer guarantee that functionality
18:39:32 <nirik> $ uname -a
18:39:33 <nirik> Linux kevintest.novalocal 4.5.2-302.fc24.x86_64 #1 SMP Wed Apr 27 14:22:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
18:39:41 <adamw> jkurik: yes
18:39:43 <nirik> x86_64 image came up fine.
18:39:47 <adamw> nirik: coolbeans
18:39:52 <adamw> can you throw a result at the matrix?
18:40:11 <sgallagh> adamw: As noted previously, I still have an AD forest inside the RHT VPN I can provide access to.
18:40:24 <nirik> adamw: sure.
18:40:47 <adamw> sgallagh: yeah, we should be able to set something up with that, but there does need to be some kinda arrangement...
18:41:12 <sgallagh> adamw: There needs to be a long-term solution, but for F24, maybe that's enough.
18:41:29 <adamw> sure, so long as we make sure someone's actually got time and access to run the tests
18:41:32 <sgallagh> (I can't guarantee I won't wipe the hypervisor that's running on for other purposes eventually)
18:42:20 <adamw> yeah, that's what worries me long term, plus since it's inside the firewall it means it has to be an RH person who does the tests
18:42:21 <adamw> anyhow
18:43:14 <adamw> i can try and run at least one or two of the tests on sgallagh's cluster right now i guess
18:43:19 <adamw> sgallagh: did you send me the details?
18:43:46 <sgallagh> adamw: No, you told me you'd ask for them if you decided to use it.
18:43:53 <sgallagh> I'll PM you
18:43:54 <adamw> sgallagh: can you send 'em now? :) thanks
18:44:20 <jkurik> ok, so can we consider the sanity test of the cloud image nirik did, as sufficient for the cloud images (for the purpose of this meeting) ?
18:44:38 <nirik> i386 cloud isn't blocking right?
18:45:23 <jkurik> nirik: no it is not
18:45:53 <adamw> jkurik: sure
18:46:01 <jkurik> so, the only check we need to do is the AD/IPA
18:46:16 <adamw> jkurik: we have the tests run with 1.4, there was no change that would affect cloud stuff between 1.4 and 1.6
18:46:20 <adamw> IPA tests are done
18:46:27 <jkurik> wow
18:46:39 <adamw> at least with 20160424, and the openQA tests with 1.6
18:46:41 <adamw> it's AD that's missing
18:46:54 <adamw> i'm OK with counting the 20160424 results for IPA, nothing relevant changed since then
18:47:08 <nirik> hey, cloud-init sucks less than it used to. Cool.
18:47:09 <adamw> i can do some wiki result transfer after the meeting to make everything look tidy
18:47:31 <dgilmore> nirik: there is no i386 cloud images
18:47:47 <dgilmore> I forgot to reenable them in release composes
18:47:52 <jkurik> ok, so we have the minimal matrices coverage we need, right ?
18:47:57 <nirik> dgilmore: great. ;)
18:48:22 <dgilmore> jkurik: no
18:48:26 <dgilmore> jkurik: missing AD
18:48:49 <adamw> so we either have to wait while me and sgallagh try to get set up for me to run the AD tests against his server
18:49:13 <jkurik> adamw: what is the ETA ? 10 minutes ?
18:49:14 <adamw> or we fudge it this one time and come up with something permanent for Final - either a reliable way to get the tests run, or we stop blocking/supporting AD
18:50:27 <adamw> jkurik: maybe? maybe 30, since i've never done this before
18:50:28 <adamw> just trying it now
18:52:19 <jkurik> ok, I would wait a while for the results
18:53:23 <jkurik> if it will take too long, than we can think of skipping the AD tests for now
18:53:27 <cmurf> Fedora-Cloud-Base-24_Beta-1.6.x86_64.qcow2 sanity checks OK
18:53:46 <jkurik> cmurf: thanks
18:55:08 <jkurik> lets continue in 20 minutes and we will see if the results will be available ...
18:55:13 <adamw> we're working on it
18:55:25 <jkurik> adamw: thanks
18:58:13 <sgallagh> Sorry, my fault. I typoed the IP address to adamw, causing hilarity.
19:01:01 <adamw> ok, the enrolment is running
19:01:17 <cmurf> for what it's worth, Cloud decided to do away with 32-bit images and are emminently publishing a Fedora Magazine article to that effect
19:01:32 <nirik> right
19:01:59 <cmurf> or rather imminently, :P
19:07:20 <adamw> just running the checks now
19:13:41 <adamw> hmm, seems like there may be a bug :(
19:13:53 <adamw> the enrolment mostly worked but user lookup doesn't seem to be working...
19:16:18 <jkurik> I can not find it now, but I believe we have a criteria for it...
19:17:17 <sgallagh> adamw: I just enrolled my F24 laptop against that AD server and it worked.
19:17:20 <nirik> adamw: you are using username@domainname right?
19:17:27 <adamw> it more or less violates "the system must respect the identity, authentication and access control configuration provided by the domain"
19:17:31 <adamw> sgallagh: huh.
19:17:33 <adamw> nirik: yes.
19:17:40 <adamw> sgallagh: missing package, maybe? i started from a vanilla server install
19:17:58 <sgallagh> adamw: Not impossible.
19:18:10 <sgallagh> I'd like to see the SSSD logs.
19:18:23 <sgallagh> It's also possible I have a fix from u-t that you don't
19:18:27 <sgallagh> What version of SSSD?
19:18:28 <adamw> working on it
19:19:42 <sgallagh> Hmm, you probably have 1.13.3 while I have 1.13.4
19:20:37 <adamw> the fuck
19:20:49 <sgallagh> adamw: Once you get those logs, see if updating SSSD fixes it.
19:20:53 <adamw> i'm trying to get your DNS into /etc/resolv.conf permanently so i can be sure everything works on startup
19:21:00 <adamw> /etc/resolv.conf keeps freaking disappearing
19:21:05 <adamw> even thoguh i told NM not to touchj it
19:21:13 <sgallagh> ...
19:22:25 <adamw> ok, there
19:23:05 <adamw> hah, ok, and now getent works
19:23:34 <jkurik> what has changed ?
19:23:35 <sgallagh> OK, so it was probably a DNS lookup issue
19:23:36 <adamw> and i can log in as him too
19:23:40 <adamw> jkurik: probably the dns thing
19:23:47 <sgallagh> I bet the resolv.conf got swapped before SSSD tried to reach AD
19:23:59 <sgallagh> That's a relief
19:23:59 <adamw> on boot the system was still using a DNS server which knew nothing about the AD controller
19:24:04 <adamw> i was manually editing it after boot
19:24:19 <adamw> i would've thought it'd work right after enrolment, since resolv.conf was cahnged already, but whatever. it works
19:24:38 <adamw> so let's see, i can enter some results
19:24:51 <jkurik> does not it break the "     When configured to use the domain controller for DNS services, the system must be able to use DNS to discover the domain controller address using SRV records. " ?
19:25:03 <sgallagh> adamw: It should have worked right after enrollment, but it's possible you caught a race where it got replaced in the middle.
19:25:30 <sgallagh> jkurik: No, because the system wasn't configured to use the domain controller for DNS services.
19:25:46 <sgallagh> (Or rather, the configuration wasn't applied persistently)
19:25:51 <adamw> jkurik: no, because that assumes the DNS server in question is actually serving the right records.
19:26:09 <jkurik> ok, got it
19:26:13 <adamw> anyhow, take it from us, we're okay calling this a test config issue. :P
19:26:20 <adamw> in a proper deployment, this...would not happen.
19:26:28 <sgallagh> adamw: No, the specific wording there is about a client explicitly pointing at the DC for DNS, which this wasn't (by accident)
19:26:33 <sgallagh> Yes
19:26:40 <adamw> sgallagh: yeah, true.
19:26:51 <adamw> so i can affirm QA:Testcase_realmd_join_sssd for AD
19:27:18 <nirik> \o/
19:27:26 <adamw> that leaves join_kickstart and join_cockpit un-covered, but since those are ultimately just driving realmd, i think we can accept the combination of this test + alpha tests of those + recent tests of kickstart/cockpit for IPA as sufficient coverage for the requirements there
19:27:41 <adamw> with the caveat that for final we should test properly
19:28:31 <jkurik> adamw, sgallagh: thanks for the verification
19:28:53 <jkurik> lets move to the final part - the go/no-go voting
19:29:17 <jkurik> #topic Go/No-Go decision
19:29:48 <adamw> so I think QA can vote go, though it requires a bit more fudging of the test coverage situation than I'd like...but that's mostly our fault and i'm OK with it as a one-time thing
19:29:52 <adamw> anyone else in QA object?
19:30:16 <jkurik> we also need someone from FESCo as well as from Releng
19:30:31 <sgallagh> I'll vote Go from FESCo
19:30:33 <nirik> I think we are good to go...
19:30:38 <nirik> dgilmore: ^ ?
19:30:45 * kparal is go
19:30:56 * jkurik is go as well
19:32:13 <jkurik> proposed #agreed The Fedora_24_Beta_1.6 compose has been agreed as the GOLD to be released as planned on May-10.
19:32:44 <jkurik> anyone would like to change the wording ?
19:33:13 <nirik> ack
19:33:22 <adamw> ok by me
19:33:56 <dgilmore> releng is go
19:34:04 <jkurik> ok, thanks
19:34:10 <jkurik> #agreed The Fedora_24_Beta_1.6 compose has been agreed as the GOLD to be released as planned on May-10.
19:34:18 <nirik> so, do we want to discuss missing images? Or just ship 1.6 as is and just miss those missing images?
19:34:53 <jkurik> nirik: my understanding was that we ship what we have (without the missing images)
19:35:12 <jkurik> are there any other proposal what we can do with it ?
19:35:42 <adamw> the question i guess is whether we very messily ninja the 1.4 xfce images in
19:35:45 <adamw> i would be opposed to that
19:35:54 * jkurik does not feel very comfortable to mix 1.4 and 1.6 composes
19:35:56 <adamw> it sucks to be missing xfce but i don't think we should mess with the composes, they are units
19:36:18 <nirik> well, or whats the delta between nightly and beta right now?
19:36:35 <nirik> just the anaconda with that one bug fix for 1.6 isn't stable?
19:37:50 <adamw> blivet
19:37:51 <nirik> so I was thinking if websites would buy in, we could have just those missing images have a short note that they didn't compose for beta, but you could use X nightly compose ? not sure how much I like that either tho
19:37:55 <adamw> not sure other than that
19:38:16 <adamw> but yeah, i wouldn't object to something along those lines
19:38:22 <adamw> it'd be up to websites if it can be reasonably worked in tough
19:38:39 <nirik> we would likely need to put them somewhere in alt...
19:38:53 <nirik> and make checksums and sign them
19:39:02 <nirik> so there would be releng work too
19:39:42 <sgallagh> That seems like a lot of work for a non-blocking deliverable
19:40:02 <nirik> yeah, true.
19:40:31 <nirik> dgilmore / robyduck
19:40:37 <nirik> thoughts? ^
19:40:39 <jpigface> (just download the "Security live" if ya want Xfce on Beta 1.6 :-D)
19:40:50 <adamw> heh, point.
19:41:10 <nirik> sure, or install via netinstall iso
19:41:28 <linuxmodder> and add the xfce goodies or whatever its called for the  full suite
19:41:30 <nirik> but SoAS is in the same boat.
19:41:31 <jkurik> I do not have an explicit reason, but I do not like the idea of mixing this together. I would prefer what we have in 1.6
19:41:37 * robyduck reads backlog
19:42:19 <dgilmore> signing CHECKSUMS we can not do
19:42:30 <dgilmore> not for nightly composes
19:42:39 <linuxmodder> reading  backlog  -- we ARE shipping  beta  just now talking about the non blocking images ?
19:42:45 <robyduck> uhmm, I would rather slip than doing all this stuff, making all this is a lot of work as we have automatic paths and checksums
19:42:46 <dgilmore> linuxmodder: yes
19:43:08 <nirik> dgilmore: well, we could... we may choose not to, but we could.
19:43:13 <jkurik> robyduck: we do not need to slip - these images are non-blocking
19:43:13 <nirik> robyduck: ok, fair enough
19:43:22 * nirik drops the idea.
19:43:30 <robyduck> then let's skip them
19:43:47 <nirik> it does still mean some work, as you will have to somehow indicate they are missing...
19:43:56 <nirik> but we had to do that for alpha too..
19:44:05 <dgilmore> nirik: if we did we would have to hold up the compose until it was signed every day
19:44:09 <linuxmodder> as is what would the  proposed  non blockers that would not  ship be?
19:44:10 <dgilmore> then manually rsync
19:44:20 * satellit point to Fedora-24-20160505.n.0/compose/
19:44:21 <linuxmodder> xfce and soas only?
19:44:38 <robyduck> we have already some labs which failed the alpha fina compose and just dropped them from the prerelease
19:44:42 <nirik> dgilmore: no, I was talking about copying some days composes of _just those missing isos_ and putting them in alt/somewhere and creating/signing a checksum for them...
19:44:54 <nirik> not the entire nightly
19:44:57 <robyduck> and soas also failed IIRC
19:45:04 <robyduck> (for alpha)
19:45:09 <nirik> but I am fine dropping the idea, just wanted to explore it.
19:46:04 <jkurik> ok, so looks like we are done
19:46:04 <nirik> so, lets close up/move on/etc.
19:46:05 <jkurik> #topic Open floor
19:46:13 <jkurik> anything else to discuss today on this meeting ?
19:46:27 <jkurik> #action jkurik to publish the Go/No-Go result
19:47:02 <jkurik> I am going to close the meeting in approx. 1 minute ...
19:48:04 <jkurik> #endmeeting