14:36:53 <dgilmore> #startmeeting RELENG (2016-07-18)
14:36:53 <zodbot> Meeting started Mon Jul 18 14:36:53 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:36:53 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:36:53 <zodbot> The meeting name has been set to 'releng_(2016-07-18)'
14:36:53 <dgilmore> #meetingname releng
14:36:54 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
14:36:54 <zodbot> The meeting name has been set to 'releng'
14:36:54 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
14:36:56 <dgilmore> #topic init process
14:37:21 <dgilmore> #chair mboddu
14:37:21 <zodbot> Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll
14:37:29 * sharkcz is here
14:37:34 <nirik> morning
14:37:41 * pbrobinson o/
14:40:41 * masta is here
14:40:58 <dgilmore> #topic #6420 Hypothetical maximum speed mirroring
14:41:05 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6420
14:41:18 <dgilmore> not sure we have much else to say
14:41:51 <dgilmore> there was teh one reported issue where it seems that the filielist file was pulled while being updated
14:42:38 <dgilmore> other than that we need to ensure that the filelist files are updated appropriately for each rsync module
14:42:47 <masta> I guess that file is written to in place
14:43:15 <nirik> thats been fixed.
14:43:40 <nirik> there's now a EOD marker at the end, so if you pull the file and it's not there, you know it's incomplete.
14:45:23 <dgilmore> #info issue with pulling file while being updated has been addressed
14:45:51 <dgilmore> have we identified all the areas that write to /pub?
14:45:55 <dgilmore> do we need to?
14:46:44 <masta> it would be nice to know
14:47:06 <nirik> I think we have... the only one that needs updating is the secondary stuff
14:47:39 <dgilmore> #info secondary update processes need updating to update the filelist
14:47:41 <nirik> rawhide, branched composes, bodhi for epel/fedora updates, secondary01 for alt stuff, seconday
14:49:30 <dgilmore> alt stuff is written to from more places than secondary01
14:49:49 <dgilmore> compose-x86-01 at the least also
14:50:05 <nirik> yeah, it's a tough one because it's not all script driven
14:50:14 <dgilmore> should we have some kind of messaging and a service that updates based on recieveing messages
14:50:16 <nirik> the kernel-nodebug, etc could update anytime
14:50:23 <dgilmore> yep
14:50:31 <nirik> right now it's on a hourly cron
14:50:41 <nirik> which isn't ideal
14:51:03 <dgilmore> it is not
14:51:20 <dgilmore> do we only update it if it changes?
14:51:29 <dgilmore> do we publish messages when it is updated?
14:51:53 <nirik> yes, only if something changes.
14:51:59 <nirik> no, but we should do a fedmsg for it
14:52:26 <nirik> we just need to pick a name/message and add it to the scripts
14:53:18 <dgilmore> okay
14:53:42 <dgilmore> it would be nice to have a message, the mirrors could then listen for it and pull when something changes
14:53:49 <nirik> yep.
14:54:01 <nirik> alt might need certs... but composers and bodhi have em already
14:54:15 <dgilmore> they probably all need certs
14:54:33 <dgilmore> since we probably want them for a mirror user or some such
14:54:39 <dgilmore> or ftpsync
14:54:56 <dgilmore> #info need to get fedmsg integration
14:55:01 <nirik> yeah, I was going to use ftpsync/the same whatever sends the 'rsynced updates contents' one
14:55:09 <nirik> but I haven't looked closely.
14:55:23 <dgilmore> nirik: thats releng
14:55:31 <dgilmore> i would rather not use that cert
14:55:52 <dgilmore> actually I think branched/rawhide is releng and updates is bodhi
14:56:15 <nirik> we can sort it out. perhaps file a ticket to discuss names/what cert out of meeting?
14:56:21 <dgilmore> sure
14:56:28 <dgilmore> lets move on for now
14:56:33 <dgilmore> #topic #6437 Update documentation
14:56:43 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6437
14:57:03 <dgilmore> mboddu: did you manage to get anywhere with adding the git hash of some other identifier?
14:58:41 <mboddu> yep, I figured a way to use hash for versioning
14:59:41 <dgilmore> cool
14:59:49 <mboddu> and also I got the code which generates the docs on pagure, I will change it so that it can pick up the version # directly from the hash
15:00:26 <mboddu> first thing first, we need to identify what are all the changes and start documenting them, we can do it in our KT process
15:01:19 <dgilmore> #info mboddu to update the docs to use git hash for versioning
15:01:28 <dgilmore> mboddu: they all need review and updating
15:01:39 <dgilmore> I do not think we need to indentify specifc ones
15:01:47 <dgilmore> as we find issues we need to fix them
15:01:54 <dgilmore> and add new docs for defiencies
15:02:28 <mboddu> KT- knowledge transfer
15:03:11 <mboddu> sure thing
15:05:43 <dgilmore> #info we assume all docs need updated and as we go though things will update them
15:06:38 <dgilmore> #info need a way to ensure all documents are reviewed
15:07:43 <dgilmore> lets move on
15:07:44 <dgilmore> #topic Secondary Architectures updates
15:07:45 <dgilmore> #topic Secondary Architectures update - ppc
15:07:52 <dgilmore> pbrobinson: how is ppc this week?
15:09:04 <sharkcz> builds are blocked by 1356919 (emacs vs glibc malloc), but otherwise no problems
15:09:47 <pbrobinson> yea, not bad, there's a few minor issues
15:10:04 <pbrobinson> the one that was mentioned by sharkcz
15:10:45 <dgilmore> #info build blocked by rhbz#1356919
15:10:52 <dgilmore> cool
15:11:06 <dgilmore> #topic Secondary Architectures update - s390
15:11:15 <dgilmore> how is s390 sharkcz ?
15:11:49 <sharkcz> better than ppc :-) still just hours behind primary from the builds point of view
15:11:59 <dgilmore> cool
15:12:06 <dgilmore> #info no blockers currently
15:12:08 <sharkcz> and the introduction of s390x golang should bring us even closer
15:12:38 <pbrobinson> there was a side tag requested for golang, I created it this morning
15:12:45 <dgilmore> #info s390x introduced golang so will remove more differences
15:12:52 <dgilmore> okay
15:13:57 <dgilmore> #topic Secondary Architectures update - arm
15:14:04 <dgilmore> hows aarch64 pbrobinson ?
15:14:16 <pbrobinson> builds are mostly up to date
15:14:25 <pbrobinson> the composes were failing for the last week or so
15:14:47 <pbrobinson> I submitted a lorax pull request that should fix that shortly
15:15:06 <dgilmore> okay
15:15:26 <dgilmore> rawhide in general was failing due to lorax bugs in the last week
15:16:04 <pbrobinson> yea, not surprised there really looking at some of the recent commits/changes
15:17:29 <dgilmore> #info composes have been failing due to lorax bug
15:17:46 <dgilmore> #topic f25 status
15:17:49 <pbrobinson> moonshot is moving forward, discovered a firmware bug
15:17:56 <dgilmore> #undo
15:17:56 <zodbot> Removing item from minutes: <MeetBot.items.Topic object at 0x39b73110>
15:17:58 <pbrobinson> awaiting confirmation around it
15:18:01 <dgilmore> pbrobinson: fun
15:18:37 <pbrobinson> hoping to have confirmation/fix shortly
15:18:43 <pbrobinson> that's it for aarch64
15:18:52 <dgilmore> cool
15:18:54 <dgilmore> #topic f25 status
15:19:06 <dgilmore> so lets quickly go over f25
15:19:21 <dgilmore> https://fedoraproject.org/wiki/Releases/25/Schedule
15:20:42 <dgilmore> branching in in 8 days
15:21:07 * nirik nods.
15:22:06 <dgilmore> mboddu: will be handling the mass branching
15:22:17 <dgilmore> pbrobinson: will be handling enabling branched composes
15:22:30 <nirik> cool. Happy to help out with anything if anyone gets stuck on anything.
15:22:34 * dgilmore will be on PTO without computer access
15:22:42 <nirik> you out all next week?
15:22:58 <dgilmore> I will be most likely working monday
15:23:07 <pbrobinson> I'm around but will be on GMT+8
15:23:16 <dgilmore> but I may have to take PTO to look after Mason
15:23:30 <mboddu> thanks nirik, I hope I wont face any issues
15:23:48 <nirik> yeah, hopefully it will go smoothly.
15:23:54 <dgilmore> pingu offered to help with pkgdb side of things
15:24:03 <pbrobinson> mboddu: co-ordination is key, and don't be afraid to ask questions
15:24:18 <dgilmore> hopefully he fixed the bug that caused the pkgdb side of things to take about 10 hours to run
15:24:29 <mboddu> thanks pbrobinson
15:24:55 <nirik> dunno, we should ask. ;)
15:25:01 <dgilmore> he said he had
15:25:09 <dgilmore> but not sure how well its tested
15:25:59 <dgilmore> any releng features for f25 need to be deployed by monday next week
15:26:16 <nirik> yeah, it's hard to test things that happen only 2x/year
15:27:48 <dgilmore> maxamillion: so we have a week to ensure OSBS is deployed and whatever we are going to deliver in f25 is done
15:27:54 <dgilmore> nirik: indeed
15:28:13 <dgilmore> it does mean at least that it is one of the better documented pieces
15:28:39 <maxamillion> dgilmore: yeah, there's still a few items on the list but we're mostly done
15:29:20 * dgilmore does not want us adding new features after alpha
15:30:08 <maxamillion> dgilmore: it's not features, just finishing up some items like the BZ component and fedpkg blacklisting
15:30:17 <dgilmore> maxamillion: okay
15:30:17 <nirik> FYI, I have been updating builders slowly. arm01/arm02 are done. Going to do arm04 today and some buildvm's... then will hopefully finish buildvm's and buildhw in the next few days.
15:30:22 <maxamillion> dgilmore: it's still in the same scope, we're not adding anything that we didn't originally say we would
15:30:38 * dgilmore wants to have all the deliverables done and defined for alpha
15:31:06 <dgilmore> maxamillion: sure. we just need to have it done
15:31:20 <maxamillion> dgilmore: indeed we do
15:31:23 <pbrobinson> nirik: is that updating them to F-24?
15:31:27 <maxamillion> along with all the things
15:31:27 <nirik> yes
15:31:39 <nirik> seems to be working fine. I updated stg a few weeks ago...
15:31:44 <pbrobinson> cool, will start to do the same for ppc/aarch64 this week then too
15:32:58 <dgilmore> #info nirik updating builders to Fedora 24
15:33:32 <pbrobinson> is F-22 last push today or tomorrow?
15:33:58 <dgilmore> it is today
15:34:04 <nirik> tomorrow is eol
15:34:25 <dgilmore> nirik: right
15:34:36 <dgilmore> the last push has to go out before then
15:34:39 <dgilmore> so it will be today
15:34:42 <nirik> I'm starting a push in a bit. You want me to do another f22 one after this one finishes? or just call it dead?
15:34:49 <pbrobinson> cool, once that's done I'll close out secondaries too
15:34:50 <dgilmore> up to you
15:35:08 <nirik> will see when it finishes and such...
15:38:05 <nirik> anything else on f25?
15:38:30 <dgilmore> #info nirik to do the last push for f22 soon
15:38:38 * dgilmore has nothing
15:38:53 <dgilmore> #topic Open Floor
15:39:06 <dgilmore> doea anyone have naything they want to raise?
15:39:11 <dgilmore> or call it a day?
15:39:13 <nirik> I was wondering about https://fedorahosted.org/rel-eng/ticket/6432
15:39:47 <dgilmore> they will have to be done this week
15:40:00 * nirik nods
15:40:02 <dgilmore> I am not sure that they are ready for it
15:41:04 <pbrobinson> dgilmore: yes, they are, I was going to use the partial script to push builds for the list tomorrow morning
15:41:07 <nirik> well, I think rpm is ready.
15:41:26 <nirik> this would done in a side tag? or ?
15:41:37 <pbrobinson> yep
15:41:39 <dgilmore> pbrobinson: okay, to me at least its not clear that things were ready for the rebuild to happen
15:41:47 <dgilmore> it will have to be a side tag
15:42:02 <pbrobinson> dgilmore: they asked about it on the internal rel-eng channel a few hours ago
15:42:03 * dgilmore thinks that it needs some better communication
15:42:14 <dgilmore> pbrobinson: thats not very useful
15:42:32 <dgilmore> not anyones fault
15:42:46 <pbrobinson> dgilmore: I don't disagree, I directed them in future to the external channel and they did update the ticket
15:42:48 <dgilmore> just not very useful to ask in the wrong places
15:42:58 <dgilmore> anyway
15:43:05 * nirik has nothing more off hand.
15:43:14 <dgilmore> #info pbrobinson to do partial mass rebuild for python
15:43:19 <nirik> dgilmore: you going to do the eol on f22 tomorrow? or want someone else to via the SOP?
15:43:35 <dgilmore> pbrobinson: you going to do it on compose-x86-01 with the releng cert?
15:43:46 <dgilmore> nirik: I was going to
15:44:05 <nirik> ok. fine with me.
15:44:11 <dgilmore> I will do it with mboddu
15:44:15 <pbrobinson> there's also the golang 1.7 rebuild that needs to be done by branch too
15:44:22 <pbrobinson> dgilmore: yes
15:44:42 <dgilmore> we should send out an email today stating there is a week for any side tags to be done
15:45:21 <pbrobinson> I think it's currently just those two still open, haskell is merged in and done
15:48:16 <dgilmore> yeah it is
15:48:20 <dgilmore> just had a look
15:48:34 <dgilmore> anything else?
15:48:53 <nirik> oh, FYI, on the new buildhw's... we ran into a snag with rack sizes (again). Hopefully we will get them up in the next few weeks tho...
15:54:54 <dgilmore> okay
15:54:59 <dgilmore> what is the issue
15:56:39 <nirik> they are too long to fit in our normal racks, so we had to order a longer one.
15:57:40 <dgilmore> fun
15:57:46 <dgilmore> what is the hardware?
15:58:15 <pbrobinson> nirik: sounds like the issue with the moonshot
15:58:15 <nirik> they are dell ... uh, don't recall the model, but bladecenters... 8 machines per x 2 (16 total)
15:58:27 <nirik> pbrobinson: yep, its the exact same issue.
15:58:38 <nirik> my bad for not doublechecking it. ;(
15:58:48 <nirik> anyhow, we will get em up and running.
15:59:59 <nirik> and retire the old buildhw's finally.
16:02:07 <dgilmore> okay
16:04:32 <dgilmore> anything else?
16:04:59 <dgilmore> #endmeeting