14:33:05 <dgilmore> #startmeeting RELENG (2016-06-20)
14:33:05 <zodbot> Meeting started Mon Jun 20 14:33:05 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:33:05 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:33:05 <zodbot> The meeting name has been set to 'releng_(2016-06-20)'
14:33:05 <dgilmore> #meetingname releng
14:33:05 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
14:33:05 <zodbot> The meeting name has been set to 'releng'
14:33:05 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
14:33:08 <dgilmore> #topic init process
14:33:38 <nirik> morning
14:33:52 * pbrobinson is here
14:34:01 * sharkcz is here
14:34:38 <dgilmore> going to try keep this short today
14:34:56 <dgilmore> planning to skip the secondary arch updates, and just cover the one issue
14:35:18 <pbrobinson> well secondaries are all done with F-24 so that's fine with me :)
14:35:24 <sharkcz> ack :-)
14:35:30 <nirik> hurray. ;) congrats
14:35:42 <masta> hiya guys
14:37:02 <dgilmore> #topic spins that missed the boat
14:37:40 <pbrobinson> they didn't miss the boat, all their bits were there just there was issues with their compose
14:37:49 <dgilmore> they missed the boat
14:38:00 <dgilmore> but it was because of anaconda that they did
14:38:07 <dgilmore> well except for Jam_KDE
14:38:35 <dgilmore> #info Jam_KDE was not installable due to not all of its packages being resolvable
14:38:37 <pbrobinson> missing the boat makes it sound like it was something they didn't do which isn't the case
14:39:04 <dgilmore> #info SoaS and Design_Suite missed the release due toa  bug in anaconda
14:39:16 <dgilmore> pbrobinson: in the Jam KDE case it was
14:39:29 <dgilmore> missing teh boat means they did not make the release
14:39:30 <pbrobinson> dgilmore: sure, but for the other two it's a different issue
14:39:45 <dgilmore> it does not differenciate why
14:39:58 <dgilmore> https://fedoraproject.org/wiki/Spins_SIG
14:40:16 <dgilmore> the information in the spins sig pages is woefully out of date
14:40:29 <dgilmore> even the list of spins in F24 is innacuare
14:40:32 <nirik> the spins sig is pretty much nonexistant. ;)
14:40:35 <maxamillion> dgilmore: yes ... >.>
14:40:43 <maxamillion> .hello maxamillion
14:40:44 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
14:40:51 <maxamillion> sorry I'm late, was on the phone
14:41:18 <dgilmore> #info Spins documatation is innacuate and incomplete.
14:41:31 <dgilmore> #info Spins SIG is not existant
14:41:43 <dgilmore> maxamillion: no problem
14:41:53 <nirik> we should clean that all up, but thats not really the issue here. ;)
14:41:59 <dgilmore> I did respond to mattdm's email
14:42:15 <pbrobinson> yes, but that has been the case for as long as I can remember, going back to around F-13 when I added the Netbook spin it wasn't great
14:42:49 <nirik> the issue is if we can or should or want to do anything for those 2 spins for f24 release... :)
14:43:27 <dgilmore> #info the issue is if we can or should or want to do anything for those 2 spins for f24 release...
14:43:31 <pbrobinson> I think we should, mattdm emailed to the rel-eng list about an hour or so about it
14:43:46 <nirik> all the options are hacky. ;(
14:44:04 <dgilmore> since I forgot to use #info last week. and was called out on it going to make sure that I do better this week
14:44:14 <dgilmore> you all can do it for important things also
14:44:24 <pbrobinson> yes, but I don't see why those spins should be disadvantaged for something out of their control and for doing the work
14:44:36 <nirik> I guess it's: 1) do nothing, they don't exist. 2) use the last nightly composes and put them somewhere perm and have websites point to them with a note, 3) something else?
14:44:43 <pbrobinson> especially since the anaconda/lorax bug was opened beginning of march
14:44:49 <dgilmore> I think the only viable option is to take the last known working nightly and put it in an unofficial name space in alt
14:45:47 <nirik> that will have the prerelease notice on it right?
14:45:49 <pbrobinson> 4) resubmit the failed build which should create the identical F-24 GA compose option, put it on alt.fp.o, sign checksums, let websites know
14:45:50 <dgilmore> mostly I think that if we do something that its is viable to do a nightly, that if we keep resubmitting the failed task, it will look like it should have been part of the release
14:45:57 <dgilmore> It needs to be clear it is not
14:46:14 <dgilmore> it is more an issue for websites and marketting to deal with
14:46:47 <dgilmore> pbrobinson: that ignores the compose metadata, particullary when it comes to PDC
14:47:05 <dgilmore> if we want to trust the data in PDC we need to ensure it does not lie
14:47:21 <pbrobinson> dgilmore: where is PDC even used yet?
14:47:25 <dgilmore> doing anything is going to get increeasingly difficulat as we automate everything
14:47:35 <dgilmore> pbrobinson: does not matter, we want it to be used
14:47:58 <dgilmore> we want to move forward with it being an accurate source of information
14:48:02 <masta> interesting point about pdc
14:48:08 <nirik> would resubmitting even work? some of that data might not be in the same places as it was when the compose was running?
14:48:22 <dgilmore> if we know from the start it lies because we ignored things we are setting ourselves up badly
14:48:24 <pbrobinson> I still think it's bullshit that we just drop stuff for things completely out of the spins control, with no warning and without even taking it to FESCo
14:48:43 <nirik> hum, I guess it would
14:48:58 <dgilmore> nirik: it would
14:49:14 <nirik> how about: 5) we fire off new composes of those with the rc2 stuff and use those...
14:49:29 <masta> any risk doing that?
14:49:39 <nirik> by use I mean put in a place in alt like proposed for nightly
14:49:42 <dgilmore> nirik: you mean just the two failed spins?
14:49:47 <nirik> yes.
14:49:50 <pbrobinson> I tested resumitting the SoaS one and the first time I tried it worked http://koji.fedoraproject.org/koji/taskinfo?taskID=14522162
14:50:26 <dgilmore> pbrobinson: that may have been a bad thing to do
14:50:28 <nirik> that would be better than a nightly as it wouldn't have the prerelease notice and such
14:50:48 <nirik> does koji keep track of the old failed build, or it replaces it?
14:50:48 <dgilmore> if people find the build they may wonder what is going on and why it was not in the release
14:50:58 <dgilmore> it replaces it
14:51:20 <dgilmore> nirik: the last nightlys will not have the prerlease notice
14:51:21 <nirik> yeah, thats the problem there. ;(
14:51:26 <pbrobinson> well people are already wondering why a SoaS image isn't in the release even without them know about that
14:52:11 <dgilmore> anaconda looks at fedora-release and if the release of teh build is 1 or higher it turns betanag off
14:52:17 <nirik> ok.
14:52:59 <dgilmore> if need be we could do a new nightly compose, with just the final content
14:53:13 <dgilmore> and just the two images
14:53:38 <dgilmore> that way we could be sure that the content matches GA
14:53:39 <pbrobinson> why do all that extra work?
14:54:00 <dgilmore> it would give us the metadata in pdc
14:54:02 <nirik> can we "adjust" pdc ?
14:54:18 <maxamillion> everything about this feels like a giant one-off which makes me worry that this will lead to a slippery slope of making exceptions for blocker bugs in the future, "but you did it last time...."
14:54:20 <nirik> ie, tell it about the resubmitted jobs?
14:54:21 <pbrobinson> I mean the anaconda/lorax issue is intermittent so it might pass/fail, statistically it would be just as likely to pass the resubmitting of the failed jobs
14:54:23 <dgilmore> would have to ask threebean, but I do not think so
14:54:27 <dgilmore> at least not easily
14:54:27 <pbrobinson> and less work
14:55:01 <dgilmore> maxamillion: I agree, I also think that as pbrobinson is the spin owner of SoaS his view is a littel biassed
14:55:23 <dgilmore> pbrobinson: less work how exactly?
14:55:32 <pbrobinson> maxamillion: if you read the BZ on it (https://bugzilla.redhat.com/show_bug.cgi?id=1315541#c12) it was meant to be an Alpha blocker
14:56:20 <pbrobinson> dgilmore: you might think that, if we'd gone with RC 1.1 which had a working SoaS but had other failures at that point my opinion was still the same
14:56:21 * nirik doesn't see any reason to be unreasonable here. We should find a way to get people those 2 spins, since they aren't broken due to any fault of theres and we do have ways to provide them.
14:56:24 <maxamillion> pbrobinson: I'm not sure I follow what you mean by that ... where were things missed? what was the blocker against
14:56:27 <maxamillion> ?*
14:56:39 <pbrobinson> dgilmore: I am more than capable of splitting the two opinions
14:56:48 <nirik> threebean: you happen to be around for a PDC question? ;)
14:56:55 <pbrobinson> maxamillion: read adamw's comment
14:57:28 <maxamillion> pbrobinson: so it was an alpha blocker against either anaconda or pungi4 that never god fixed before alpha went out?
14:57:34 <maxamillion> got*
14:58:03 <nirik> maxamillion: no, it didn't get accepted as a blocker, because the workaround was "meh, we will just re-run it until we have all the release blocking parts, and too bad about the rest"
14:58:17 <pbrobinson> maxamillion: not sure if it was never proposed or what, my point was that there's no reason that something like this should ever be anything but a one off exception once the bug is fixed
14:58:34 <maxamillion> pbrobinson: that's my point, it's a one-off
14:58:37 <maxamillion> I don't like one-offs
14:58:51 <pbrobinson> maxamillion: but it didn't appear to get the appropriate escalation/attention early enough in the cycle so that we'd never have to have this discussion
14:59:07 <maxamillion> I've seen them be used as precendent that leads to 100 hour weeks in every job I've ever worked at since getting into IT as a profession
14:59:23 <threebean> nirik: I'm here now
14:59:25 * threebean reads up
15:00:00 <nirik> threebean: 2 of the spins failed to compose (due to no fault of their own). We could resubmit those jobs (one of them already has been)... but then pdc knows nothing.
15:00:16 <nirik> how hard would it be to import/change the info for those in pdc after the fact.
15:00:42 <threebean> if we have a complete compose.json and rpms.json productmd files from pungi, very easy.
15:00:50 <threebean> if not.. I'm not sure how hard.  but, harder.
15:00:54 * dgilmore notes that will meanmanually munging the compose tree
15:00:59 <dgilmore> after it is on the mirrors
15:01:14 <threebean> yeah, this sounds hard.
15:01:26 <threebean> not impossible.
15:01:28 <adamw> i think we'd have to mung the metadata, then manually upload it to PDC.
15:01:36 * threebean nods
15:01:44 <dgilmore> its nmot impossibe, it is a lot of work to do right
15:02:25 <dgilmore> pungi records the variant an image is in
15:02:47 <dgilmore> so we would have to put the images in Spins/ and Labs/
15:02:53 <dgilmore> change the CHECKSUMS
15:02:56 <nirik> I guess then I am just in favor of pulling the last nightly (and hopefully getting a quick smoketest that they aren't broken horribly) and put them in alt and have websites point to them with a note.
15:02:58 <dgilmore> resign them
15:03:37 <dgilmore> nirik: I think that is the only viable option. but that is more up to websites and marking how they communicate it
15:03:47 <dgilmore> marketing
15:03:53 <nirik> "Due to a compose bug, this spin wasn't produced for the final release, however, the last nightly version should work fine and contains the same package set, you can find it here... "
15:03:55 <nirik> or something
15:04:17 <dgilmore> sure
15:04:36 <dgilmore> no idea how hard it will be to have websites point elsewhere
15:05:16 <adamw> yeah, we should probably open a ticket and ask them or something
15:06:02 <dgilmore> adamw: I think mattdm should, since he is pushing for something to be done
15:06:06 <nirik> robbyduck was open to the idea
15:06:19 <adamw> i was using 'we' in my usual sense of 'someone who is not me' :P
15:06:26 <dgilmore> adamw: :P
15:06:36 <adamw> it's my favourite f/oss word
15:06:40 <adamw> it's like a social 'sudo'
15:06:42 <dgilmore> It will not be me, after the meeting I am going back to sleep
15:06:45 <nirik> but we (releng we) needs to put them in place and sign checksums, etc.
15:07:17 <dgilmore> nirik: sure, I can do that in 8-9 hours when I get to the office
15:07:32 <nirik> ok.
15:07:45 <dgilmore> assuming by then we know that it is something that is okay, and which exact isos to ship
15:07:52 <pbrobinson> I can probably do something if it needs to be done before then
15:09:08 * nirik too.
15:09:22 <nirik> anything else, everyone happy with that? ;)
15:09:47 <dgilmore> Proposed #agreed, the least amount of 1 off work is to ship some tested nightlies in /pub/alt/unoffical/release/24/ just the isos, signed checksums and some manifest files that list all the srpms to keep legal happy
15:11:19 <dgilmore> #action mattdm to work with websites and marketing over communication
15:11:35 <nirik> ack
15:11:40 <dgilmore> since no one voted I will assume that everyone agrees
15:11:58 <dgilmore> #agreed The least amount of 1 off work is to ship some tested nightlies in /pub/alt/unoffical/release/24/ just the isos, signed checksums and some manifest files that list all the srpms to keep legal happy
15:12:51 <maxamillion> ¯\_(ツ)_/¯
15:13:00 <dgilmore> #info keeping the images seperate and named differently will ensure that the metadata and what is in PDC about the compose and release is correct
15:13:49 <dgilmore> #info someone needs to step up and manage spins, update documentation, ensure that policies are properlly documented
15:14:42 <nirik> well, we just need to fix docs IMHO.
15:14:55 <dgilmore> possibly
15:15:06 <nirik> new spins -> change process, fixes for spins -> "owner" of them.
15:15:18 <nirik> just treat them like any other package.
15:16:16 <nirik> but thats a larger topic for another day. ;)
15:16:50 <dgilmore> we made the new spins change a few release back
15:16:59 <dgilmore> but thats not well documented :(
15:17:02 <pbrobinson> it's no different to workstation or docker or any of the other artifacts
15:17:49 <dgilmore> no one really owns any of the compose time built artifacts
15:17:50 <maxamillion> nirik: +1
15:18:08 <dgilmore> docker base image is possibly the worst owned thing
15:18:25 <dgilmore> as technically the Base WG, that no longer exists owns it
15:18:36 <dgilmore> so I guess modularity WG should
15:18:42 <nirik> having owners might be good...
15:18:50 <dgilmore> but the cloud WG tries to claim ownership some times
15:19:19 <dgilmore> #action dgilmore to ensure all artifacts have known owners
15:19:52 <nirik> it might also be nice to have all things have a bz component or known place to file bugs where people who care about it can see them
15:20:25 <dgilmore> I wonder if we can get them added in pkgdb
15:20:51 <nirik> possibly, or have another component...
15:21:11 <nirik> Fedora -> releasecomponents
15:21:13 <dgilmore> then ownership in bugzilla nd koji could be set, FMN can deal with sending task notifications
15:21:25 <dgilmore> aybe
15:22:17 <nirik> might be worth pondering on.
15:22:17 <dgilmore> we just do not need a git repo for them
15:22:51 <dgilmore> So we will need to have follow on conversations about dealing with the compose artifacts
15:23:19 <maxamillion> dgilmore: puiterwijk and I kind of own the Docker Base image
15:23:25 <maxamillion> dgilmore: at least in terms of Docker Hub release
15:23:43 <maxamillion> dgilmore: https://github.com/docker-library/official-images/blob/99e785d91e5392e6e9059d8c67286a6404efab69/library/fedora#L1-L2
15:23:48 <dgilmore> maxamillion: but that comes after the thing is built
15:23:53 <maxamillion> correct
15:24:01 <maxamillion> which is why I said, "kind of"
15:24:19 <dgilmore> there is no one that actually owns defining what is in it, nor to call on when it fails to fix things
15:24:21 <maxamillion> we direct people to the kickstart for changes, but we don't really do much ... it just kidn of exists
15:24:38 <dgilmore> just like the Spins
15:24:56 <maxamillion> I've chased down things to fix with it just because I normally get contacted about it since my email is on the Hub listing
15:25:09 <maxamillion> but I'm not sure I'm the "owner" in any officially defined context
15:25:24 <maxamillion> so maybe we should sort that at some point
15:25:31 <dgilmore> maxamillion: well technically its the BaseWG
15:25:37 <dgilmore> they just no longer exist
15:25:39 <maxamillion> right
15:25:54 <dgilmore> it is a zombie artifact
15:26:10 <nirik> I just would prefer that we have a easy way to contact a group of people who care about a artifact and want to fix it/keep it working... right now we don't always, it
15:26:18 <nirik> s very hit or miss finding people
15:26:37 <dgilmore> I think anything we do to ensure that people are notified has to be self manageable and automated
15:26:48 <nirik> agreed
15:27:09 <dgilmore> which kinda falls into pkgdb
15:27:27 <maxamillion> nirik: +1
15:27:46 <dgilmore> but maybe we need some name spacing, or maybe we can take advantage of the work that will be needed to handle modularity
15:28:08 <nirik> yeah.
15:28:24 <nirik> artifacts/ instead of rpms/
15:28:43 * nirik shrugs. not sure
15:29:08 <dgilmore> indeed
15:29:12 <dgilmore> so hours about up
15:29:42 <dgilmore> #info there is much more that needs discussed. we will have to ensure conversations happen
15:29:58 <dgilmore> #info need to find owners for all artifacts
15:30:23 <dgilmore> #info need to make sure that owners know when things do not work in a simple automated way
15:30:27 <dgilmore> #open floor
15:30:31 <dgilmore> gahh
15:30:41 <dgilmore> #topic Open Floor
15:30:56 <dgilmore> we could probably send all day discussing how to deal with things
15:31:07 <dgilmore> does anyone have something they want to bring up?
15:32:39 <maxamillion> dgilmore: we definitely could :)
15:32:39 * nirik has nothing off hand. Hoping for a smooth release tomorrow.
15:33:01 <dgilmore> nirik: indeed.
15:33:22 <dgilmore> I am going to do the torrents when I get to the office
15:33:49 <nirik> dgilmore: FYI, if you want to save some sync time, everything should be up to date on download-ib01...
15:34:56 <dgilmore> nirik: thats where I have been pulling from for awhile now
15:35:02 <nirik> ok, cool.
15:36:06 <dgilmore> okay, if nothing else. lets wrap up
15:38:20 <dgilmore> #endmeeting