18:00:13 <nirik> #startmeeting EPEL
18:00:13 <zodbot> Meeting started Fri Sep  4 18:00:13 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:14 <nirik> #meetingname EPEL
18:00:14 <zodbot> The meeting name has been set to 'epel'
18:00:14 <nirik> #chair smooge avij bstinson dgilmore nirik Evolution
18:00:14 <nirik> #topic Init process / agenda gathering
18:00:14 <zodbot> Current chairs: Evolution avij bstinson dgilmore nirik smooge
18:00:31 <avij> hello
18:00:36 <bstinson> hi all
18:01:35 <nirik> morning everyone
18:01:57 <avij> good 9pm everyone
18:01:57 <nirik> There's folks interested in talking other arches... any other topics for today aside from that?
18:02:14 <smooge> hello
18:02:55 <nirik> dgilmore: you around?
18:03:00 <smooge> I have an item
18:03:10 <smooge> RHEL-7.2 beta (and then final)
18:03:26 <bryanpkc> hi
18:04:01 <nirik> sure.
18:05:08 <smooge> nirik, want me to run the meeting since I am here now?
18:05:17 <nirik> if you like. ;)
18:05:24 <smooge> I think dgilmore is out getting a cheque or something so won't be around
18:05:45 <smooge> Evolution, you here for this meeting?
18:05:49 <Evolution> I am
18:05:52 <Evolution> for once.
18:06:01 <smooge> ok so we have 4 out of 5 so have super quorum
18:06:16 <smooge> penultimate quorum?
18:06:25 <smooge> anyway first topic since it is short
18:06:39 <smooge> #topic RHEL-6.7 and RHEL-7.2 beta
18:07:12 <smooge> RHEL-6.7 was released a while ago.. a weeding of packages that were moved in and out of the distro probably needs to be done.
18:07:43 <Evolution> to throw a monkey wrench in that slightly....
18:07:49 <nirik> sure, just needs someone to go over the list and file bugs/ticket.
18:07:56 <smooge> Evolution, ?
18:07:59 <Evolution> do we care about server vs workstation package differences?
18:08:05 <Evolution> I seem to recall a bug or two about that.
18:08:24 <smooge> yeah there is a problem in that we build against server but there are packages in workstation
18:08:33 <nirik> yes. we don't use/care about workstation. ;)
18:08:53 <avij> I filed a number of bugs for the packages to be retired; http://fpaste.org/263713/
18:09:33 <nirik> avij: cool. ;) perhaps we should use a tracking bug also and then we can go force retire if maintainers don't respond
18:10:21 <bryanpkc> just curious: if something is dropped from the distribution, would EPEL pick it up again?
18:10:34 <smooge> bryanpkc, if someone wants to take it
18:10:42 <avij> I'm not sure if those cover all the packages to be retired, but those were the ones that I filed
18:11:05 <smooge> bryanpkc, we don't pick up packages.. we just give people room to support those packages using our hardware :). [Or something like that.]
18:11:09 <nirik> they are usually good about listing new ones in release notes.
18:11:44 <smooge> Evolution, does that cover what you wanted to monkey with?
18:12:03 <avij> in any case, I think I checked that there's a bug for each package to be retired, either filed by me or someone else
18:12:11 <Evolution> yup.
18:12:26 <nirik> I've not looked recently, but in the past when we looked at workstation vs server there were things like them shipping different versions of the same packages, which would make our lives more crazy
18:12:28 <bryanpkc> smooge, thanks for the explanation
18:12:31 <smooge> #info RHEL-6.7 was released a while ago.. a weeding of packages is commencing thanks to avij
18:12:45 <smooge> #info RHEL-7.2 beta was released this week.
18:12:58 * nirik grumbles about the pidgin situation too.
18:13:13 <orionp> There were broken deps due to ImageMagick soname bump in 6.7, not sure if that's all be sorted
18:13:24 <smooge> RHEL-7.2 is a HUGE rebase of a lot of the graphics stack and some other items. I have noticed we have some packages which are moved in and some packages which are moved out (and 1-2 packages we have already in)
18:13:29 <nirik> orionp: I rebuilt all the ones I could see and submitted updates on them.
18:13:41 <Evolution> have those updates been pushed past testing?
18:13:43 <orionp> nirik - great!
18:13:54 <smooge> Evolution, if anyone tested them
18:13:55 <nirik> Evolution: don't think so yet, need karma or more time
18:14:15 <nirik> smooge: of course we can't really act on it since it's not final yet, but good preview of fun ahead.
18:14:43 <Evolution> smooge: perhaps a ticket of anticipated breakage/changes?
18:15:18 <nirik> https://admin.fedoraproject.org/updates/drawtiming-0.7.1-2.el6 https://admin.fedoraproject.org/updates/psiconv-0.9.8-6.el6 https://admin.fedoraproject.org/updates/perl-Image-SubImageFind-0.03-2.el6 https://admin.fedoraproject.org/updates/spectrum-1.4.8-6.el6
18:15:33 <smooge> I am trying to get an idea of all the src.rpms which have changed majorly so we can see. Since some of this is in workstation and some of this is in server and they may not overlap
18:15:42 <avij> yes, thanks nirik, looks like all the packages depending on ImageMagick have now been taken care of
18:16:16 <smooge> if all goes well maybe next meeting?
18:16:25 <Evolution> sounds like a plan.
18:17:01 <smooge> anything else anyone wants to talk about RHEL releases? [Maybe RHEL-5 EOL sometime in 2029? :)]
18:17:37 <orionp> 2017
18:17:38 <Evolution> in *theory* we should have a 32bit build for 7 done soon.
18:18:06 <smooge> orionp, sure sure.. I will believe it when I see it. [It is still 35+% of EPEL load :)]
18:18:18 <Evolution> we need to spin and test install media, and that's it.
18:18:38 <smooge> Evolution, cool. At that point we can look at what is possible...
18:18:49 <smooge> ok next topic then
18:19:00 <smooge> #topic Alternative Archictectures
18:19:12 <smooge> which would be i386 and s390
18:19:35 <smooge> and ppc64le and mips :)
18:19:45 * Evolution shoots mips in the face
18:19:46 <nirik> and armv7 and aarch64
18:20:09 <nirik> anyhow, we talked about i386/i686 at flock.
18:20:16 <Evolution> aarch64 is gold and released already, but dgilmore wanted to begin with x86
18:20:16 <smooge> Evolution, if we do mips32 we could be on every router :)
18:20:30 <nirik> dgilmore said he thought we could get things to work building that against centos and still building x86_64 against rhel...
18:20:49 <nirik> I didn't think koji would let us, but if he says it could work I trust him. ;)
18:20:59 <bryanpkc> aarch64 is not 2ndary arch?
18:21:25 <bryanpkc> or am I misunderstanding evolution
18:21:50 <smooge> bryanpkc, I believe it is gold in CentOS
18:21:59 <bryanpkc> oh i see
18:22:04 <Evolution> bryanpkc: different terminology.
18:22:10 <Evolution> alternate vs secondary
18:22:25 <Evolution> some folks said 'secondary' made it feel like they mattered less.
18:22:48 <smooge> I am not using secondary on purpose because our build system has conotations for secondary which seemed to get confusion occuring
18:22:51 <nirik> there's talk of changing what that means in fedora as well.. (although nothing has been officially proposed)
18:23:10 <Evolution> smooge: yeah. I don't care one way or the other, just pointing it out.
18:23:25 <smooge> secondary usually has a different koji, different rules for releases and such
18:23:33 <nirik> anyhow, I think 32bit intel is the thing we would/should try first and move on from there.
18:23:36 <Evolution> x86 is built but not assembled. aarch64 is out the door with a shiny gold star.
18:23:41 <smooge> In that case, EPEL is not set up to deal with "secondary" in that form.
18:23:56 <nirik> the logistics of other arches could be difficult, since we may not have builders for them... we would have to figure out what we want to do then.
18:24:14 <bryanpkc> we have several options s390 builders... but they'll be remote
18:24:28 <bstinson> re x86: sounds like a job for a set of targets in koji.stg :)
18:25:20 <nirik> bryanpkc: yeah. We have some remote ones that fedora s390 secondary arch uses, could use those, there was talk about some at syracuse, and some other sites...
18:25:28 <nirik> bstinson: yeah
18:26:45 <bryanpkc> Syracuse is not officially GA until November but we can start getting you some VMs now
18:27:18 <bryanpkc> if physical access is a must... someone has to got pay for the box :-)
18:28:31 <Brian_TS> I think I'm late to the party
18:28:50 <avij> we were just talking about s390 builders
18:28:50 <nirik> hey Brian_TS
18:29:05 <Brian_TS> Sorry about my lateness....got pulled into a security meeting.
18:29:15 <nirik> bryanpkc: we should likely talk to the s390 fedora secondary arch maintainer... I don't know much about those builders.
18:30:30 <Brian_TS> Anyhoo....I've got access to s390x/zvm hardware and was wanting to compile RPMs but I understand that I'm an outsider (not in a bad way) and there's a trust relationship that would be hard to overcome because the fedora project doesn't have access to Z architecture to ensure that nastiness doensn't get injected into the RPMs.  I completely understand.
18:30:38 <nirik> perhaps before we do anything official folks who do have access to the hw might do a rebuild and put it up somewhere so we can see issues?
18:31:10 <nirik> Brian_TS: there's that, also there's a matter of control... if we depend on someone elses infra we are down if they are down, etc.
18:31:22 <Brian_TS> nirik: agreed.
18:31:26 <bryanpkc> hi Brian_TS
18:31:44 <bryanpkc> nirik, we have done rebuilds for a bunch of EPEL SRPMs
18:31:46 <nirik> which may paritially be mitigated with control over vms (or whatever s390x calls them) :)
18:31:52 <nirik> cool.
18:32:14 <bryanpkc> last time i checked, 75% of SRPMs could be built out of the box
18:32:32 <bryanpkc> we just used a stupid shell script that iterated over the SRPMs
18:32:33 <smooge> The builder issue is my major worry mainly because in the past that was the bigger complaining "We can't build F-nn because EPEL is down (whether true or not)"
18:32:42 <Brian_TS> bryanpkc: I've had higher numbers than that in my own non-scientific testing.
18:32:57 <bryanpkc> Brian_TS: EPEL6 or EPEL7
18:32:58 <bryanpkc> ?
18:33:08 <Brian_TS> bryanpkc: both...but mostly EPEL6 at this point.
18:33:29 <Brian_TS> bryanpkc: However, I've only built a small subset of things so far.  Perhaps 200 RPMs from EPEL.
18:33:40 <bryanpkc> oh
18:33:53 <bryanpkc> Brian_TS: we were trying 4400 SRPMs
18:33:57 * nirik runs to get more coffee, back in a few.
18:34:06 <bryanpkc> Brian_TS: all of EPEL7
18:34:23 <Brian_TS> bryanpkc: The only stuff I've had issue with is things that require dmidecode and other intel-ish stuff.
18:35:12 <Brian_TS> however 50% of what I compile ends up being noarch RPMs.
18:35:24 <bryanpkc> I think we had issues with ghc-related stuff, some python-related stuff
18:35:41 <nirik> ghc is often a problem. ;)
18:35:50 <bryanpkc> I didn't look too deeply at the issues. I have zero knowledge about ghc
18:36:16 <bryanpkc> i have a list somewhere... need to find where it is
18:36:30 <Brian_TS> I have next to zero knowledge about rpmbuilding.  You guys do a great job of making the specs quite portable.
18:36:43 <bryanpkc> anyway, in IBM we could build the packages and try to report on the issues
18:36:50 <bryanpkc> but our hands are tied around publishing binaries
18:37:23 <bryanpkc> and i'm not certain that our method (a shell script that iterates on the SRPMs) is correct
18:38:05 <bryanpkc> i use mock to do the builds, but i didn't set up koji... not sure if that's ok
18:38:43 <smooge> mock should cover most of the issues
18:39:03 <smooge> bryanpkc, you invented copr-0.9
18:39:41 <bryanpkc> smooge, lol
18:40:54 <smooge> so for me, if we can get some sort of way to build the RPMS via the fedora s390 2ndary builder without breaking Fedora then I don't see why not. However those are a couple of big ifs
18:42:01 <smooge> until those get answered... we are in a hold pattern
18:42:04 <bryanpkc> why would EPEL build failures break Fedora?
18:42:16 <smooge> it is more about builders not being available
18:42:29 <nirik> well, you mean other arch epel builds?
18:43:23 <smooge> nirik, yeah. I know in the far past when EPEL ppc builders went down, koji would sometimes stop processing Fedora builds
18:43:39 <smooge> until someone went in and did some magic
18:43:42 <nirik> that must have been back when ppc was a fedora primary arch or something.
18:44:37 <bryanpkc> build failures could cause builders to go down?
18:44:42 <smooge> I can't remember.. I thought it was after ppc was primary and the only builders were for EPEL. We have had such good uptimes in the last 4 years that I don't know if it still matters
18:45:19 <bryanpkc> what if we add net new s390 builders for EPEL
18:45:25 <nirik> anyhow, I think next steps here are to work on i686 first... and also talk with fedora s390 secondary arch maintainer(s) to see how they handle builders and if epel could do something similar.
18:45:27 <bryanpkc> and stay away from the fedora 2ndary builders
18:45:56 <bstinson> bryanpkc: i think the issue is more with remote builders that go down for $REASONS (unrelated to the builds themselves)
18:46:10 <nirik> right.
18:47:39 <bryanpkc> new builders should still help right?
18:47:41 <smooge> The way it was explained to me by a koji developer several FUDcons ago was koji etc are built on the assumption that the builders are local. that it works when they aren't.. is not a planned feature :)
18:47:54 <bryanpkc> if these new builders go down, they would affect EPEL but not Fedora
18:48:13 <nirik> bryanpkc: sure... but blocking epel builds is still anoying and bad.
18:48:43 <bryanpkc> nirik, agreed... and that's the other concern... how can we experiment with a new arch without blocking other arches
18:49:18 <nirik> if koji had the ability to not care about failures on some arches that would be handy for that.
18:49:40 <nirik> but I don't think it does... perhaps thats on the roadmap for 2.0
18:49:54 <bstinson> bryanpkc: if you are already building bits of EPEL internally, and are prepared to report bugs/patches (even if you don't post any results) that would be helpful
18:50:10 <nirik> failing that there's setting up a new koji with just that arch and building the builds as they come on primary
18:50:48 <nirik> perhaps something could be setup to listen on fedmsg for epel builds and fire those off on s390
18:50:55 <bryanpkc> bstinson: yeah, we can do that. would i report issues on bugzilla.redhat.com? or somewhere else?
18:50:58 <bryanpkc> mailing list?
18:51:45 <nirik> bugzilla should be good.
18:52:58 <bryanpkc> nirik, setting up a new koji sounds like a solution
18:53:19 <bryanpkc> it could host all the new arches as issues are ironed out
18:54:41 <nirik> yeah, not sure how easy it would be to shadow the main one... it would be different than the fedora secondary arch ones...
18:55:34 <Evolution> coming up on the hour soonish. are we good to keep going on this, or does anyone have anything else burning to address?
18:55:56 <smooge> I don't have anything more to address on this.
18:56:31 <bryanpkc> nirik: you will talk to the fedora s390 secondary arch maintainer(s)?
18:56:37 <nirik> I can do so yeah.
18:57:07 <bryanpkc> ok i'll take a look at our build results from a month or so ago (haven't done re-builds recently)
18:57:15 <bryanpkc> and start combing through the failues
18:59:01 <smooge> #action nirik will contact s390 arch maintainer about EPEL
18:59:07 <smooge> or is that item...
18:59:24 <smooge> anyway.. going to close out the meeting in case another group has it in 1 minute
18:59:30 <smooge> #endmeeting