15:30:51 <masta> #startmeeting releng
15:30:51 <zodbot> Meeting started Mon Jun  2 15:30:51 2014 UTC.  The chair is masta. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:30:51 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:30:51 <masta> #meetingname releng
15:30:51 <masta> #topic === Releng Roll Call ===
15:30:51 <masta> #chair dgilmore dwa nirik tyll sharkcz bochecha
15:30:51 <zodbot> The meeting name has been set to 'releng'
15:30:51 <zodbot> Current chairs: bochecha dgilmore dwa masta nirik sharkcz tyll
15:31:02 <nirik> morning
15:31:08 * masta waves
15:31:36 * sharkcz is here
15:31:46 <masta> we have a few folks running late, so please grab coffee or whatever you have to do...
15:33:55 <dgilmore> hola
15:34:30 <tyll_> Hi
15:36:19 <masta> hi
15:36:33 <masta> ok I guess we can get started
15:36:51 <masta> #topic === releng Announcements ===
15:38:14 <tyll_> what should be announced here?
15:38:20 <masta> hehe
15:38:46 <masta> ok, lets move on to tickets then
15:38:50 <tyll_> next monday is a public holiday in Germany
15:38:56 <tyll_> maybe we need to move next meeting?
15:39:05 <tyll_> this could be an announcement
15:39:34 <tyll_> I don't know whether it is an holiday in the USA as well
15:39:54 <dgilmore> tyll_: its not, last monday was a us holiday
15:40:02 <dgilmore> win 40
15:40:16 <masta> #topic === releng Tickets ===
15:40:16 <masta> #link https://fedorahosted.org/rel-eng/report/10
15:41:15 <masta> https://fedorahosted.org/rel-eng/ticket/5877   (Schedule and coordinate mass rebuild for F21)
15:41:47 <nirik> are we all set for it?
15:41:57 <tyll_> last meeting it came up that the scripts need to be looked at
15:42:12 <tyll_> I remember some problems reporting FTBFS during last cycle
15:42:46 <tyll_> maybe links to build logs were wrong
15:42:48 <dgilmore> we should try and make the reporting better
15:42:52 <nirik> tyll_: yeah, me too, but I don't recall details. ;)
15:43:07 <dgilmore> the issue with the ftbfs bugs was that the urls to the logs were wrong
15:43:10 <nirik> ah yeah, links.
15:43:22 <dgilmore> and they only live a short time anyway
15:43:36 <tyll_> btw. will be rebuild in a side tag?
15:43:38 <dgilmore> really we shouldnt put links to koji tasks in
15:43:40 <tyll_> will we
15:43:52 <dgilmore> tyll_: yes it will all land in f21-rebuild
15:43:59 <dgilmore> ill merge it to f21 when done
15:44:09 <nirik> mass_rebuild_file_bugs.py ?
15:44:13 <dgilmore> that way we dont get hundreds of newRepos
15:44:17 <tyll_> we could copy failed build logs to fedorapeople or attach them to bugzilla
15:44:20 <tyll_> I see
15:44:21 <dgilmore> yeah
15:44:39 <dgilmore> tyll_: scripting attaching to bugzilla is really really hard
15:45:02 <dgilmore> you need to get back the bug number and attach them seperately
15:45:04 <tyll_> #info rebuild will happen to f21-rebuild and will me merged into f21 to avoid newRepo tasks
15:45:19 <dgilmore> the api doesnt allow you to upload attachments when creatinga  bug
15:45:21 <nirik> it would be nice to attach them, because then they would always exist.
15:45:29 <nirik> but yeah.
15:45:37 <tyll_> mdomsch always attached them in the past
15:45:42 <dgilmore> it could be done, just likely a fair bit of work
15:46:04 <dgilmore> maybe we can find his scripts and see how he did it
15:46:11 <tyll_> maybe we can first move them to fedorapeople to save them and attach them to the bugs if we find the time
15:46:26 <dgilmore> an option
15:46:29 <sharkcz> bugzilla --user=foo '--password=bar' attach -f "$filename" -d "$filename" -t text/plain $bug_id
15:46:39 <sharkcz> ^^ this used to work to attach files in bz
15:47:14 <dgilmore> sharkcz: but you need to know the bug number
15:47:25 <dgilmore> which i  think was kinda hard to get
15:47:32 <dgilmore> we should try do it
15:47:34 <nirik> mass_rebuild_file_bugs.py is the script right?
15:47:37 <dgilmore> at least look
15:47:42 <dgilmore> nirik: yeah
15:48:06 <nirik> would anyone have time to work on it? I could see if any of the infra folks might...
15:49:23 <dgilmore> can try and make time, though i suspect that I wont have any until next week
15:49:39 * nirik also ask around.
15:49:41 <dgilmore> we dont need to run it until we merge the tag back in
15:49:48 <dgilmore> which i expect to do next monday
15:49:59 <nirik> we also need a f21 mass rebuild wiki page right?
15:50:08 <dgilmore> yeah
15:50:15 <dgilmore> I usually do that last minute :(
15:50:49 <tyll_> #action fix mass_rebuild_file_bugs.py to attach build logs to bugs
15:51:00 <tyll_> #action add f21 mass rebuild wiki page
15:51:58 <tyll> #info rebuild will happen to f21-rebuild and will me merged into f21 to avoid newRepo tasks
15:52:35 <tyll> shouldn'z zodbot say something if I use a #keyword?
15:52:42 <nirik> on the plus side we have all builders up and running now... so we should have even quicker time than last time.
15:52:49 <dgilmore> tyll: i believe it doesnt
15:52:58 <nirik> tyll: it doesn't... design decision from the plugin author.
15:53:03 <dgilmore> last time was 46 hours
15:53:49 <masta> do we re-run the script on the failed builds, to sorta shake-out bad builds?
15:53:50 <dgilmore> #info mass rebuild will start friday evening US Central time
15:54:09 <dgilmore> masta: nope
15:54:29 <dgilmore> the maintainers are responsible for dealling with all failures
15:55:04 <tyll> #action add f21 mass rebuild wiki page
15:55:06 <masta> ok, then afterwards I'll try to triage with the maintainers of any failed builds, or just try them myself
15:55:13 <tyll> #action fix mass_rebuild_file_bugs.py to attach build logs to bugs
15:56:00 <nirik> so I pulled a f20 ftbfs bug up, but the links seem ok in it...
15:56:10 <nirik> oh wait, I see where they are wrong.
15:56:29 <dgilmore> nirik: they were pointing at parent tasks not the child that had the logs
15:56:54 <nirik> right. the base one was ok, but the logs were all wrong.
15:57:37 <nirik> but ideally we would just attach them.
15:58:20 <dgilmore> yeah, just need to fetch them and get the bug number to attach them to
15:59:17 <tyll> I believe createbug() already returns a bug object with the number
16:00:05 <masta> ok folks, I'd like to steer the meeting topic to the next ticket soon. Any final things to discuss on the mass-rebuild?
16:01:07 <masta> https://fedorahosted.org/rel-eng/ticket/5870   (rawhide signing)
16:02:10 <tyll> The current status is in the ticket
16:02:36 <tyll> I still need access to secondary sigul and become koji admin to be able to do everything
16:02:56 <masta> =)
16:02:58 <dgilmore> #topic
16:03:14 <dgilmore> gahh
16:03:42 <dgilmore> #topic https://fedorahosted.org/rel-eng/ticket/5870
16:03:52 <tyll> the recent builds on Rawhide should already be signed
16:04:01 <tyll> But there some design decissions left
16:04:33 <tyll> Since fedmsg is not meant to be reliable, I believe it would be best to use fedmsg messages only as a trigger to sign all unsigned builds
16:04:53 <tyll> this is currently not easy to do, because there are too many builds
16:04:56 <dgilmore> tyll: ill try get the secondary sigul bits sorted out later,  we will need  Red Hat IT to open up some ports for us
16:05:24 <dgilmore> tyll: never going to be a good way to sign the whole tag
16:05:45 <dgilmore> tyll: but one way to do it would be to use calls to koji to get builds since the last run
16:05:46 <nirik> well, if we tell rawhide mash to check the key it can tell us if we missed any...
16:05:46 <tyll> also the idea is to use a gating tag, e.g (f21-pending or f21-unsigned) and then move signed builds to f21
16:05:53 <dgilmore> then sign all those builds
16:06:06 <tyll> dgilmore: it might be easier with the gating tag I guessed
16:06:13 <dgilmore> tyll: could be
16:06:25 <tyll> Ideally it is only one build all the time
16:06:33 <dgilmore> should be a smaller number of builds
16:07:05 <nirik> but if we miss a fedmsg, we wouldn't know to move that build would we?
16:07:33 <dgilmore> nirik: well if when you get teh next one you list the builds in teh staging tag you would
16:07:45 <tyll> nirik: if always all builds from f21-unsigned are signed and then moved to f21, nothing would be missed
16:08:18 <tyll> can we create a f21-unsigned tag already but use inheritance to not disturb rawhide compose?
16:08:24 <nirik> yeah, but if we do that I would like a log/warning. ;)
16:08:29 <tyll> this would allow to develop and test this
16:08:41 <dgilmore> since when you move builds you need to do it per build you will have to list all builds in the tag with inhertance off to get the nvrs to move
16:08:48 <nirik> ie, in our past tests we have not had any fedmsg lossage, but would be good to know if it started happening.
16:08:55 <dgilmore> so you would sign those nvrs then move them
16:09:20 <tyll> nirik: it is not so easy to detect, because there might be builds were a message is not processed
16:09:53 <nirik> tyll: well, we have datalogger/datagrepper listening/recording them all too... so we could check against thaat
16:10:19 <tyll> nirik: yes, some kind of delayed reporting is possible
16:10:30 * nirik nods.
16:11:14 <threebean> in the past, tflink and I ran a script that 1) listened for koji messages and counted them, and then 2) once an hour, asked koji for all the latest builds.
16:11:16 <tyll> ah, I just rememeber, I asked threebean whether we could add a fedmsg message for signed builds, to detect problems
16:11:39 <threebean> it compared the two lists and if there was any discrepancy, it warned us.  aside from some early hiccups where we had typos and stuff, there were no dropped messages over a two week period.
16:11:47 <threebean> but -- that's a strategy for testing if you want to try it
16:11:50 * nirik nods.
16:11:59 <tyll> then the tag messages could be compared with the signing messages and discrepancies reported
16:12:29 <dgilmore> tyll: 
16:12:37 <dgilmore> gahh
16:12:47 <nirik> well, if we just sign/move the build(s) we just got a message for and then check that the tag has nothing left in it, that would check too right?
16:13:00 <dgilmore> nirik: yep
16:13:10 <dgilmore> plenty of ways to skin the cat
16:13:38 <tyll> nirik: not really, because there might be a message in the queue for the left builds
16:13:43 <nirik> ie, sign foobar-1.0, move it to the f21 tag, then see that there's baz-1.0 still in f21-unsigned, but we didn't see a fedmsg for it... alert!
16:13:43 <dgilmore> if we list the builds so we know what to sign and move we would know also
16:13:55 <tyll> nirik: at least currently the script only does stuff if it processes a message
16:14:01 <dgilmore> if there was two builds in the tag but one on the fedmsg
16:14:17 <nirik> ah I see... if 10 landed and you were processing... hum.
16:15:31 <nirik> especially for a first cut, I would be happy with something that just let us know and we could manually fix things...
16:15:55 <nirik> perhaps time based...
16:15:57 <dgilmore> nirik: should be easy to just get t all signed
16:16:18 <dgilmore> list the tag, sign the builds, move them
16:17:00 <dgilmore> i guess some of it will come down to do we get race conditions from lots of builds landing at once
16:17:21 <tyll> nirik: time based would work, e.g. sign all builds a message was received for and check for builds that are older than e.g. 15 minutes still left in the tag and report them
16:17:48 <tyll> I will figure something out
16:18:20 <nirik> yeah. we also could do some nagios checks on it...
16:18:37 <tyll> but back to the gating tag, can we introduce it already to be able to add the tag moving feature?
16:19:06 <nirik> I don't think we can set it up as we will want it to be... until we have something that moves them.
16:19:47 <dgilmore> tyll: we can
16:19:58 <nirik> but could we have buildrawhide move them?
16:20:00 <dgilmore> it will effect rawhide
16:20:04 <nirik> until the script is ready?
16:20:21 <tyll> nirik: I hoped inheritance can make moving uneccesary
16:20:27 <dgilmore> nirik: maybe yeah
16:20:37 <tyll> e.g. if f21 inherits from f21-unsigned
16:20:44 <nirik> tyll: but then we need to change it later when the script is ready...
16:20:53 <dgilmore> tyll we do not use inheritance in composing
16:21:13 <tyll> can we tag stuff in two tags at once?
16:21:21 <dgilmore> and in this case it wouldnt work
16:21:30 <tyll> e.g. both into f21 and f21-unsigned?
16:21:42 <dgilmore> we would have to have f21-signed inherit from f21 and compose from f21-unsigned
16:21:53 <dgilmore> we can tag a build in as many tags as we want
16:22:14 <nirik> tyll: you just want it to test against? perhaps we could make the tag and just manually have you tag builds into it for testing?
16:22:25 <nirik> and then hook it up later.
16:23:52 <masta> Looks like we got one more ticket, and then open floor.
16:24:19 <tyll> nirik: hm, it could test the single method for this manually, but since it is triggered by fedmsg it would be inconsistent during a complete run
16:24:23 <dgilmore> masta: tickets, secondary arch updates and open floor is the order
16:24:50 <masta> dgilmore: ah, thx... will update my notes
16:24:56 <nirik> tyll: true. Perhaps we can get our staging koji usefull... thats one thing I wanted to work on in the fad this week. ;)
16:25:45 <dgilmore> nirik: that would be useful. I need to write  some more koji patches for images. need a test koji to test them on
16:27:05 <tyll> nirik: yes, but then some builds and sigul need to get there too to make it complete
16:27:07 <nirik> dgilmore: yeah, lots of questions around how to do it, but we can talk about them when you get here. ;)
16:27:24 <nirik> tyll: yep
16:27:48 <masta> #topic https://fedorahosted.org/rel-eng/ticket/5914
16:27:53 <dgilmore> tyll: the build side is easy, the sigul harder
16:28:06 <masta> (Move fedmsg based blocking service to Fedora Infrastructure)
16:29:06 <dgilmore> for this we should setup a user in koji or reuse one like masher
16:29:23 <dgilmore> probably have it run as a service on releng04
16:29:31 <tyll> the the fedmsg based blocking service I wanted to discusss whether we can make it based on dead.package commits
16:29:33 <dgilmore> unless we want to have it isolated
16:30:42 <dgilmore> tyll: id honestly rather not. but only because it means we have to watch the contents of commits
16:31:01 <dgilmore> there is 3 things that need to happen to retire a package
16:31:11 <tyll> dgilmore: all information is already in the fedmsg message
16:31:16 <dgilmore> dead.package, retire n pkgdb and block in koji
16:31:46 <tyll> people tend to forget to retire in pkgdb, because it requires addtitional credentials
16:32:05 <tyll> dgilmore: btw. the fedpkg patches need to be merged to fix this
16:32:19 <dgilmore> tyll: i merged your patches on saturday
16:32:37 <tyll> dgilmore: \o/
16:32:48 <dgilmore> just need to build an update
16:33:07 <tyll> however, I have code to detect dead.package commits already ready
16:33:11 * dgilmore will drop off soon. descent just started
16:34:15 <dgilmore> tyll: so im less concerned how it all happens than that it all happens. but I feel like koji blocking should be tied to pkgdb retiring
16:34:59 <dgilmore> tyll: so how would you envison it works if it was dead.package based
16:35:04 <tyll> dgilmore: ah, I would also retire the package in pkgdb if a dead.package commit is detected
16:35:08 <dgilmore> would teh service to pkgdb also?
16:35:13 <tyll> yes
16:35:49 <dgilmore> tyll: id rather that the owner do that. as a check that the owner is the one doing it and not someone deciding to go retire the package
16:36:37 <tyll> dgilmore: in case there is a dispute, it is very cheap to undo the retirement
16:36:55 <tyll> dgilmore: Also I don't think it will happen that often
16:37:03 <dgilmore> maybe we can convince pkgdb to allow koji certs for auth for retiring packages
16:37:27 <abadger1999> dgilmore: It would be fas_openid... and I'm not sure that's a good idea.
16:37:34 <abadger1999> We still can't encrypt koji certs on disk.
16:37:37 <dgilmore> not sure how hard that would be to implement
16:37:47 <abadger1999> I mean -- without annoying maintainers out of their minf.
16:38:02 <dgilmore> abadger1999: even if we could people will always be able to undo it
16:38:11 <abadger1999> Yewah
16:38:43 <abadger1999> what about a cron job?
16:39:23 <dgilmore> abadger1999: well tyll wanst to have it so if  somoene commits a dead.package to git that the service would block in koji and retire in pkgdb
16:39:24 <abadger1999> I guess the trigger would have to be in pkgdb if we want to be assured of who requested the retirement.
16:40:42 <dgilmore> my concern is that someone could come along and retire somoeone elses package
16:41:12 <tyll> that someone could also commit bad contents
16:41:21 <dgilmore> true
16:41:25 <tyll> e.g. it is alreay privileged acccess
16:41:35 <tyll> and retiring is not final but undoable
16:41:44 <dgilmore> they would have to be a proven packager
16:41:53 <abadger1999> <nod>  Maybe if fedmsg is detailed enough, have  something watch for a message about dead.package being added.  Then trigger a retirement in pkgdb and block in koji.  The triggered script could check that the git commit was from a pkgdb owner.
16:42:26 <tyll> abadger1999: but what do we want to do if it was not the pkgdb owner?
16:42:27 <abadger1999> We'd also want an infrequent cron job that checked that everything was consistent (in case fedmsg drops a notification.)
16:42:33 <dgilmore> i guess emails will eb sent and someone could speak up
16:42:47 <dgilmore> yeah
16:43:03 <tyll> the cron job is required anyhow, because it is all inconsistent already
16:43:26 <dgilmore> end goal is to make sure retirement is one step and works
16:44:03 <tyll> (at least regarding pkgdb, dist-git and koji)
16:44:13 <dgilmore> tyll:  so using dead.package the person starting the service will have to enter their credentials for pkgdb?
16:44:58 <tyll> dgilmore: this would be a possibility
16:45:16 <dgilmore> just wondering how it would all work
16:45:35 <tyll> dgilmore: it would also be possible to create a designated account for it
16:45:57 <tyll> but the service needs credentials to pkgdb for it - but it also needs a certificate for koji
16:47:04 <dgilmore> tyll: the koji bit is pretty easy to handle
16:47:09 <abadger1999> tyll: So.. if we're requiring pkgdb credentials... maybe it makes more sense to submit it to pkgdb and have pkgdb manage the git and koji retirement.
16:47:13 <dgilmore> the pkgdb side is trickier
16:47:13 <tyll> I would actually prefer to have a designated account that has only hte necessary pkgdb cretdentials
16:47:55 <tyll> abadger1999: yes, it would require more coding, though
16:48:02 <abadger1999> <nod>
16:48:08 <dgilmore> i need to run now. laptop has to be turned off
16:48:13 <masta> in a few moments... let us move on to Secondary arch updates, and then open floor. (we are running over already)
16:48:33 <masta> dgilmore: thanks and have a great day =)
16:48:58 <masta> tyll: any final thoughts on this ticket?
16:49:20 <tyll> masta: no, thank you, we need to find a consent
16:49:56 <masta> alright, then we can move along....
16:50:10 <masta> #topic === releng Secondary Architectures ===
16:50:24 <masta> #topic PPC64
16:50:43 <masta> dwa or sharkcz: any things to update?
16:50:51 <dwa> yeah
16:51:00 <sharkcz> ppc64 is catching up primary to get to the state when we can merge with ppc64le
16:51:07 <dwa> #info ppc64 is playing whack-a-mole to catch up to primary with deps to prepare for merging ppc64le
16:51:23 <dwa> #info ppc64 may though delay doing the mass rebuild so we can get ppc64le imported and do it all at once
16:51:56 <dwa> #info ppc64 has completed updating all builders to Fedora 20 now, huzzah
16:52:37 <dwa> also for any who didn't already know, this is my last week at Red Hat. masta (and dennis and nirik) will have the keys to the ppc64 kingdom :)
16:52:38 <nirik> cool
16:53:14 <nirik> dwa: good luck with your endevors... it's been great working with you!
16:53:35 <dwa> nirik: thanks, same here :)
16:54:02 <dwa> I think that does it for me
16:54:16 <masta> dwa: try not to charge too high a consulting fee if I come looking for you to ask questions. ;-)
16:54:35 <dwa> lol. I've told johnf that I'll happily answer questions :)
16:55:01 <masta> happy and sad all at once.
16:55:31 <masta> #topic aarch64
16:55:39 <masta> ok, so that is me.
16:56:26 <masta> we are set for the rebuild, I think maybe the usual scramble to patch things just prior where possible.
16:56:57 <masta> alright...
16:57:00 <tyll> I found out aarch64 koji is missing fedmsg integration, are there any plans to change it soo?
16:57:37 <masta> tyll: sure, we can probably do that. I was unaware.
16:58:02 <masta> tyll: I'll syncup with you later or tomorrow about that. ok?
16:58:17 <tyll> masta: threebean can help you more proabably
16:58:26 <tyll> masta: I noticed it only because of the Rawhide signing
16:58:28 <threebean> tyll: there are plans.  dgilmore has asked me about it recently but we just haven't gotten time to do it.
16:58:39 <masta> tyll: ack
16:58:50 <threebean> it needs to be handled 'by hand' since the secondary arch koji instances are not under infrastructure ansible control (afaik)
16:59:04 <tyll> I see
16:59:08 <masta> #topic === releng Open Floor ===
16:59:23 <sharkcz> masta: you missed s390 :-)
16:59:28 <masta> what!
16:59:29 <masta> ok
16:59:32 <sharkcz> s390 is catching up primary after the glibc ABI breakage, we went from the end of January to end of April during May, so it goes quite well, naturally some packages don't like us and needs some attention :-)
16:59:45 <masta> #topic S390
17:00:02 <sharkcz> #info s390 is catching up primary after the glibc ABI breakage, we went from the end of January to end of April during May, so it goes quite well, naturally some packages don't like us and needs some attention :-)
17:00:17 <sharkcz> eom
17:00:34 <masta> #topic === releng Open Floor ===
17:00:41 <masta> sharkcz: my sry's
17:00:46 <sharkcz> masta: np
17:01:08 <tyll> I want to propose to handle add a list of the usual meeting topics to the wiki
17:01:42 <masta> good idea
17:01:54 <tyll> also I have ideas for two more generic topics: recap important changes since the last meeting
17:02:28 <tyll> and to announce current/upcoming tasks to make it more transparent what is happening
17:02:53 * nirik has to leave now... will catch the rest in logs.
17:03:07 <masta> our meetings seems to consume much time already.
17:03:20 <masta> but these are good ideas tyll
17:03:23 <tyll> and as a short last item, discuss the next meeting date, e.g. when the next one will happen
17:04:27 <masta> great idea
17:04:56 <tyll> hm, then we should decide what we want to use the meetings for - maybe discussing the tickets in too much detail is not so good
17:05:03 <masta> I've added all of these to my meeting notes... the doc I go by to conduct a meeting... which I just wrote as we held this meeting... it could become the wiki doc tyll
17:06:39 <masta> tyll: well the problem is there was good discussion... it's hard to predict how good our meetings will be... so best not to schedule things immediately after releng irc.
17:07:31 <masta> alrighty, good stuff... I'm going to shut it down.
17:07:38 <tyll> ok
17:07:38 <masta> thanks all!
17:07:45 <masta> #endmeeting