16:36:15 <dgilmore> #startmeeting RELENG (2015-11-23)
16:36:15 <zodbot> Meeting started Mon Nov 23 16:36:15 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:36:15 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:36:15 <zodbot> The meeting name has been set to 'releng_(2015-11-23)'
16:36:23 <dgilmore> #meetingname releng
16:36:23 <zodbot> The meeting name has been set to 'releng'
16:36:23 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
16:36:23 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
16:36:26 <dgilmore> #topic init process
16:36:44 * nirik is sort of here, but also swamped in piles of emails.
16:36:45 * sharkcz is here today
16:36:52 * pbrobinson is here
16:37:47 <dgilmore> there i no ticket items on the agenda
16:37:55 <dgilmore> but I wante dto go over some things
16:37:58 * linuxmodder listening in for learning purposes
16:38:17 <dgilmore> #topic qpid ticket
16:39:00 <dgilmore> while I was on PTO there was a ticket https://fedorahosted.org/rel-eng/ticket/6291
16:39:04 <maxamillion> .hello maxamillion
16:39:07 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
16:39:31 <dgilmore> there was no data in the ticket on what problem was trying to be solved.
16:39:42 <dgilmore> In the end I reached out and figured out the issue
16:40:46 <dgilmore> pulp upstream were building on top of rhel and epel. the qpid people updated qpid-proton which broke a dep in their stack
16:41:02 <dgilmore> as the qpid packages are still shipped in RHEL we can not replace them
16:41:26 <dgilmore> so they need to package up compat packages to give them the functionality that they needed
16:41:47 <dgilmore> anyone could have found out what was going on and made the same suggestion
16:42:02 <dgilmore> does anyone have any questions about it?
16:42:39 * nirik tried to get more info from the reporter... didn't have time to ask anyone else. thanks for figuring it out.
16:43:03 <dgilmore> It took more time than it should have :(
16:43:42 <dgilmore> Not only did they file the releng ticket, they filed a FESCo ticket and started a thread on my internal email adress pulling in all sorts of people
16:43:47 <pbrobinson> I saw the ticket but was so overloaded that we I hoped someone else would deal with it :-/
16:43:57 <dgilmore> They were making a lot of noise and had no substance
16:44:00 <dgilmore> :(
16:44:30 <dgilmore> that they made so much noise is the only reason I dig into it
16:44:51 <dgilmore> I am tempted to write to their manager
16:45:06 <dgilmore> It does however make me wonder if we need to have some releng FAQ
16:45:20 <dgilmore> and if we did would it have even helped
16:46:04 <maxamillion> +1 for making a FAQ
16:46:11 <nirik> this was a kinda weird corner issue, I don't think a faq would have helped it... but other things it could.
16:46:25 <pbrobinson> agreed
16:46:54 <linuxmodder> +1 for FAQ and helping author it
16:47:33 <dgilmore> okay
16:48:06 <dgilmore> #action All, figure out where to put a FAQ, and what to put in it.
16:48:33 * maxamillion votes to put it in the releng git repo under docs/ .... not sure what all should go into it at first pass though
16:48:53 <dgilmore> maxamillion: I think that is where it likely should go
16:48:58 <dgilmore> #topic Secondary Architectures updates
16:49:08 <dgilmore> any generic secondary arch updates?
16:49:22 <dgilmore> nirik: have we started on the arm and ppc hubs?
16:49:46 <pbrobinson> dgilmore: not yet
16:49:53 <nirik> started yes, finished no. I have the arm stuff built, but it needs db transfered.
16:49:59 <nirik> and I have had 0 time to do so yet
16:50:05 <dgilmore> okay
16:50:31 <dgilmore> #topic Secondary Architectures update - ppc
16:50:37 <dgilmore> pbrobinson: how is ppc going?
16:50:39 <pbrobinson> nor have I and I have a few bits I want to do first, I need to send some emails about the secondary shadow box etc
16:51:01 <pbrobinson> it's going OK, I'm awaiting a firmware update from IBM to upgrade the power 8 boxes
16:51:15 <pbrobinson> once that's arrived we'll go to 7.2 and then F-23 for the builders
16:51:25 <dgilmore> cool
16:51:27 <pbrobinson> should have the firmware this week
16:51:45 <pbrobinson> my primary focus is to get all the bits aligned for the EPEL builds of ppc64le
16:51:57 <pbrobinson> I need to send all the details to the epel list for that
16:52:08 <dgilmore> cool
16:52:15 <pbrobinson> I'm just trying to test and clarify a few details before I send the email
16:52:24 <dgilmore> I know people are wanting it
16:52:26 <pbrobinson> need to quiz nirik about a few bits there
16:52:39 <pbrobinson> dgilmore: not as much as I know! ;-)
16:52:45 <dgilmore> pbrobinson: do we have any plans around docker base images in f24?
16:52:58 <pbrobinson> dgilmore: yes, but post EPEL
16:53:01 <dgilmore> or other new deloiverables?
16:53:38 <pbrobinson> basically my order for PPC is 1) EPEL 2) rebuild all the infra to new 100% ansible 3) dockah dockah dockah 4) nightly qemu and docker images
16:54:04 <pbrobinson> for PPC on F-24 stuff we've started the conversation
16:54:09 <pbrobinson> mostly it's docker
16:54:33 <pbrobinson> and there's a bunch of new packages (mostly rebased on primary) we're working to assist other teams/people to get done
16:54:44 <pbrobinson> like nodejs 4.x and other joyous bits
16:55:31 <pbrobinson> probably move go support from gcc-go to golang, again coordinating with the maintainers for how that gets done
16:55:42 <dgilmore> okay
16:55:52 <pbrobinson> and general cleanups so we don't repeat the mess of F-23 in the late cycle :-D
16:56:17 <dgilmore> I am working on trying to tansition how we make rawhide. they will effect all secondaries and will make many things easier
16:56:37 <pbrobinson> affect even ;-)
16:56:59 <pbrobinson> by that I presume you mean pungi4 and friends?
16:57:18 <dgilmore> pbrobinson: :P and yes
16:57:21 <maxamillion> has there been any movement on upstream docker officially supporting other architectures?
16:57:44 <maxamillion> I've not really followed that like I probably should
16:58:34 <pbrobinson> maxamillion: I need to follow that up, I've got docker across ppc/aarch64/armv7 on this cycle so I'll review it when I get closer to achieving that
16:58:34 <sharkcz> AFAIK they are accepting needed changes for non-x86
16:58:43 <maxamillion> pbrobinson: +1
16:59:09 * kushal is here now
16:59:11 <pbrobinson> sharkcz: mostly needed around the docker registry rather than the docker binaries themselves
16:59:59 <sharkcz> pbrobinson: yep
17:02:26 <dgilmore> #topic Secondary Architectures update - s390
17:02:31 <dgilmore> sharkcz: how goes s390?
17:03:05 <sharkcz> f24 is still behind primary (now due mariadb test-suite failures)
17:03:15 <sharkcz> need to talk to the maintainers
17:03:29 <sharkcz> but I see issues with the performance of the hub
17:04:17 <sharkcz> sometimes newRepo takes the 2 minutes it should, sometimes it's 5 or even 10 minutes, reducing the throughput dramatically
17:04:29 <nirik> odd.
17:04:47 <sharkcz> dunno if it's the storage, maybe we should just try to reboot the hub & db
17:05:07 <sharkcz> but restarting httpd didn't help
17:05:26 <dgilmore> sharkcz: I am reworking the createrepo_C patches to make it acceptable to mikem
17:05:42 <dgilmore> he wants it to be a config option
17:05:42 <sharkcz> dgilmore: yep, I saw the pull request, thx
17:05:52 <nirik> FWIW I was going to mention in open floor:
17:06:06 <nirik> we are planning mass update/reboots for tuesday/wed. (get everything on 7.2)
17:06:38 <sharkcz> ok, then we will see after the reboots if there is a change to normal
17:08:04 <sharkcz> dgilmore: can we patch the releng koji to use createrepo_c?
17:08:13 <dgilmore> sharkcz: I am planning on it
17:08:14 <sharkcz> the releng packages
17:08:16 <sharkcz> ok
17:08:33 <dgilmore> sharkcz: I need to do some testing and make sure that mergerepo_c --koji is working as expected
17:09:20 <sharkcz> dgilmore: understood
17:09:38 <dgilmore> he seemed to be confused as to the expected behaviour
17:09:44 <dgilmore> #topic Secondary Architectures update - arm
17:09:50 <dgilmore> pbrobinson: how is arm>
17:09:51 <dgilmore> ?
17:09:54 <pbrobinson> OK
17:10:09 <pbrobinson> I fixed the python3.5 stuff today so builds should catch up
17:10:32 <pbrobinson> we put a new firmware on the builders last week in the hope to fix the stability issues (fingers crossed)
17:10:45 <dgilmore> :) cool
17:10:49 <pbrobinson> next up will be to move them all f23
17:11:23 <maxamillion> mmmmm python 3.5
17:11:26 <pbrobinson> plans for f23 are images with the live-media-creator feature, so dgilmore happy to test some koji patches when ever you have them
17:11:39 <pbrobinson> as well as docker and probably cloud images
17:11:48 <dgilmore> cool
17:12:20 <pbrobinson> as well as hopefully some cheapo board support too.... will depend on my time really
17:12:37 <pbrobinson> and moonshot support
17:12:39 <dgilmore> I wonder if we shouldnt reach out to the folks at scaleway and see about having them run our cloud images
17:13:12 <maxamillion> dgilmore: I thought they needed a custom kernel or something
17:13:16 <pbrobinson> dgilmore: maybe, we need to get them working first, main issue is a redistributable uEFI firmware
17:14:38 <pbrobinson> maxamillion: yes, they do, although it should just be a DT blob
17:14:55 <maxamillion> pbrobinson: ah ok
17:15:11 <pbrobinson> maxamillion: I need to go and check them out again
17:15:24 * pbrobinson needs more non weekend days in the week :(
17:15:36 <maxamillion> +1
17:15:44 <nirik> no no, more weekend days. ;)
17:16:05 <pbrobinson> nirik: agreed :-D
17:16:20 <dgilmore> anyway :)  lots to work on for f24
17:16:37 <pbrobinson> dgilmore: yes, for both power and arm
17:16:50 <dgilmore> pbrobinson: and primary
17:16:58 <sharkcz> and s390 as well :-)
17:17:07 <dgilmore> #action dgilmore to file a ticket with all the f24 work
17:17:14 <pbrobinson> so no change there..... a thousand miles an hour forwards
17:17:25 <pbrobinson> dgilmore: tickets?
17:17:42 <dgilmore> when I file it I would appreciate everyone filling in the things that they plan to work on for f24
17:17:51 <dgilmore> pbrobinson: a ticket
17:18:08 <pbrobinson> dgilmore: wouldn't we be better tracking each item in a separate ticket?
17:18:08 <dgilmore> just to track things
17:18:26 <dgilmore> pbrobinson: I epxect that each thing will be tracked individulaly yes
17:18:42 <dgilmore> I am talking about a overall picture thing
17:20:20 <dgilmore> #topic Open Floor
17:20:30 <dgilmore> does anyone have anything they want to bring up?
17:20:48 * tyll has something
17:20:59 <dgilmore> tyll: go ahead :)
17:21:17 <tyll> I noticed that there might be some confusion about how to process unretired pkgs
17:21:51 <dgilmore> tyll: okay
17:21:54 <tyll> This is currently a manual process, even if someone requests unretirement in pkdbd, one has to decide what to do manually to fulfill it
17:22:53 <tyll> changing the state in the admin request does not do anything else but to to notify the requester, that the request was processed, but it does not process it
17:23:10 * maxamillion has two quick things for after tyll
17:23:21 <dgilmore> tyll: not sure what you mean there
17:23:39 <dgilmore> "changing the state in the admin request does not do anything else but to to notify the requester,"
17:23:47 <tyll> to actually do the unretirement one needs to e.g. run "pkgdb-cli unorphan --poc rfenkhuber mockito epel7" and then "koji unblock-pkg epel7 mockito"
17:24:05 <tyll> dgilmore: there are admin requests listed at https://admin.fedoraproject.org/pkgdb/admin/actions/
17:24:41 <dgilmore> tyll: that I get, but when you process it, everything but the unblocking is suypposed to happen automatically by pkgdb
17:24:48 <tyll> dgilmore: if you update the status at URLs like https://admin.fedoraproject.org/pkgdb/admin/action/1479/status it will not unretire the package but only notify in this case irina, that the pkgs was unretired
17:25:05 <dgilmore> tyll: that is not correct
17:25:13 <dgilmore> and if it is we need to fix pkgdb
17:25:19 <tyll> dgilmore: you cannot process the unretirement process right now
17:25:36 <tyll> dgilmore: yes, it needs to be changed, I just want to make sure everyone knows
17:25:48 <dgilmore> tyll: where is the bug about this>
17:25:49 <dgilmore> ?
17:26:18 <dgilmore> nirik: were you aware of this?
17:26:34 <nirik> no
17:26:45 * dgilmore was not either
17:27:03 <nirik> so does this mean we need to redo some packages that should be retired, but aren't?
17:27:40 <dgilmore> if I read what tyll is saying there could be packages that are supposed to be unretired but are not
17:28:13 <tyll> dgilmore: here are some related bugs: https://github.com/fedora-infra/pkgdb2/issues/278 https://github.com/fedora-infra/pkgdb2/issues/277 - but I did not yet file a bug about improving pkgdb-admin to process the requests, since I was looking into creating a patch for it
17:29:03 <tyll> yes, there might be package that should be unretired but are not - but I guess package maintainers should actually notify it pretty fast if the package was not properly unretired
17:29:40 <dgilmore> reminds me of one thing I wanted to bring up
17:29:58 <dgilmore> #action all, always file issues when you hit bugs in anything
17:30:13 <dgilmore> we have done a pretty poor job of filing bugs
17:30:29 <nirik> yeah, bugs are good...
17:30:32 <dgilmore> it was kinda obvious with trying to get sigul fixes done
17:30:44 * nirik needs to update the sigul one
17:30:46 <dgilmore> but I know there is a bunch of iccues with Bodhi also
17:30:51 <dgilmore> issues
17:31:08 <dgilmore> we need to be sure that we always take the time and file the issues
17:31:21 <dgilmore> then we have a paper trail to point people at
17:31:40 <dgilmore> tyll: thanks for teh update, that is very surpissing news :(
17:31:48 <dgilmore> gahh I can not spell
17:31:59 <nirik> yeah... thanks for finding it.
17:32:07 <dgilmore> maxamillion: you have some issues?
17:32:15 <nirik> tyll++ thanks for working on tracking down unretirement issues in pkgdb
17:32:15 <zodbot> nirik: Karma for till changed to 1 (for the f23 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:32:32 <dgilmore> tyll++
17:32:32 <zodbot> dgilmore: Karma for till changed to 2 (for the f23 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:32:47 <maxamillion> dgilmore: this can be merged now if you're good with it ... there was a conflict but it's been fixed https://pagure.io/releng/pull-request/42
17:33:04 <dgilmore> maxamillion: it can not be merged still
17:33:16 <dgilmore> maxamillion:  "Could not determine if the PR can be merged "
17:33:18 <tyll> thanks
17:33:28 <maxamillion> also, kushal pointed out in #fedora-cloud that the cloud builds are failing http://koji.fedoraproject.org/koji/tasks?start=0&state=all&view=tree&method=image&order=-id
17:33:35 <dgilmore> maxamillion: you have to sign off on all patches
17:33:43 <maxamillion> dgilmore: bah! it said it could be earlier
17:33:49 <maxamillion> dgilmore: sign off?
17:33:54 <dgilmore> maxamillion: yes
17:34:02 <dgilmore> add -s when commiting
17:34:50 <maxamillion> alrright
17:35:19 <maxamillion> dgilmore: I'll add that to the doc then
17:36:44 <dgilmore> maxamillion: okay
17:36:52 <dgilmore> I am guessing that is the issue
17:37:07 <dgilmore> maxamillion: anything else? or was that it?
17:37:07 <maxamillion> dgilmore: nope :(
17:37:18 <maxamillion> dgilmore: yeah, that was it ... did you see my mention of the cloud images being busted?
17:37:36 <dgilmore> maxamillion: yeah I looked at the oz log
17:37:44 <dgilmore> Received SIGCHLD from PID 716 (dhclient).
17:37:45 <dgilmore> Child 716 (dhclient) died (code=exited, status=2/INVALIDARGUMENT)
17:37:56 <dgilmore> maxamillion: apparently dhclient changed behaviour
17:38:02 <maxamillion> *fun*
17:38:15 <maxamillion> dgilmore: anything needed there or is it being worked on?
17:38:25 <dgilmore> maxamillion: no idea if it is being worked on
17:38:32 <maxamillion> alright, I'll follow up on it in a bit then
17:38:56 <dgilmore> it would need a dhclient bug filed, or bugs for everything using dhclient to change their behaviour
17:39:16 <dgilmore> I have not looked into it any more than to see in the log why the image build failed
17:39:19 <maxamillion> rgr
17:39:24 <maxamillion> alright, that's all from me
17:39:25 <dgilmore> it will take more investigation
17:39:34 <dgilmore> anyone have anything else?
17:39:44 <maxamillion> nirik: did you mention earlier in the meeting that you had something or did I imagine that?
17:40:15 <nirik> I just wanted to note that we have some outages this week.
17:40:18 <nirik> tomorrow and wed...
17:40:27 <nirik> announcements went out a bit ago
17:40:41 <maxamillion> oh right
17:40:45 <maxamillion> :)
17:41:41 * dgilmore will note that all US based people will be off on Thursday and Friday
17:42:40 <maxamillion> +1
17:42:49 <kparal> is friday also a holiday?
17:42:59 <nirik> yeah.
17:43:14 <dgilmore> kparal: yes
17:43:25 <kparal> ok, my calendar shows only thursday as thanksgiving day
17:43:35 <kparal> good to know
17:43:44 <maxamillion> kparal: yeah, I had to double check with the holiday listings in Mojo ... the calendar is incorrect :/
17:43:59 <dgilmore> kparal: it is, but Red HAt and many employers give the employees friday off as well
17:44:16 <kparal> I see
17:44:20 <dgilmore> I used to work at a place that made me work on the Friday :(
17:45:02 <dgilmore> if there is nothing more I will end the meeting
17:45:03 <pbrobinson> most product week of the year for everyone else ;-)
17:45:03 <maxamillion> I need to step afk ... thanks dgilmore for running the meeting
17:45:42 <dgilmore> #endmeeting