18:00:05 <smooge> #startmeeting EPEL (2019-08-28)
18:00:05 <zodbot> Meeting started Wed Aug 28 18:00:05 2019 UTC.
18:00:05 <zodbot> This meeting is logged and archived in a public location.
18:00:05 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:05 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:05 <zodbot> The meeting name has been set to 'epel_(2019-08-28)'
18:00:06 <smooge> #meetingname epel
18:00:06 <zodbot> The meeting name has been set to 'epel'
18:00:06 <smooge> #topic Chair and Introductions
18:00:06 <smooge> #chair bstinson Evolution nirik smooge pgreco tdawson
18:00:06 <zodbot> Current chairs: Evolution bstinson nirik pgreco smooge tdawson
18:00:28 * bstinson waves
18:00:32 <nirik> morning
18:00:40 <pgreco> hello
18:00:57 <smooge> #topic Agenda
18:00:57 <smooge> #info EPEL-7 aarch64 problems
18:02:00 <tdawson> hello
18:03:13 <smooge> ok looks like 4 people
18:04:22 <smooge> bstinson, you around?
18:04:33 <bstinson> yep, i'm here
18:04:47 <smooge> ok and I see Evolution in channel so I will assume he is present and accounted for
18:04:58 <smooge> #topic EPEL-7 aarch64 problems
18:05:32 <smooge> alright we got reports that EPEL-7 builds were having issues because the RPM in 7.7 and 7.6 are different.
18:05:52 <pgreco> do you have an example handy to follow?
18:06:00 <nirik> the root package is one...
18:06:26 <nirik> due to the rpm in 7.7 subpackages appear different and the builds fail.
18:07:05 * jsmith lurks
18:08:23 <nirik> more and more will hit it.
18:08:28 <tdawson> I don't see any failed root builds.  Did they get it fixed?  Or was it all scratch builds?
18:08:41 <pgreco> I was going to say the same thing
18:08:53 <pgreco> I"m looking if I can find the failed task
18:09:12 <tdawson> Is there a bug or something we can look at?
18:09:53 <pgreco> tdawson https://koji.fedoraproject.org/koji/taskinfo?taskID=37241968
18:10:39 <tdawson> Ouch ... so it's different on different architectures.
18:10:52 <pgreco> yeap, that is the base problem
18:11:16 <pgreco> 7.6 was the last release for aarch64
18:11:18 <nirik> nordugrid-arc is another one
18:11:20 <smooge> https://pagure.io/releng/issue/8676
18:12:25 <smooge> so since 7.6 has one version of rpm and other items and 7.7 have other .so updates.. there will be build differences
18:13:23 <nirik> I guess my only question is if we might migrate it to live next to armv7 in centos land?
18:13:33 <nirik> or would that work?
18:14:19 <pgreco> currently, epel for armv7 is built in a rather precarious old system (not even koji)
18:14:36 <nirik> is centos going to do a 7.7 for aarch64?
18:14:44 <pgreco> but if we could create a second koji instance (in fedora or centos, same with me)
18:15:00 <pgreco> to build those arches together, from centos instead of from rhel
18:15:00 <nirik> we actually could switch to using that if it existed...
18:15:42 <smooge> i think we have 2 conversations going. let us see if we can get some answers
18:15:50 <pgreco> nirik, old plague system...., in process of being removed
18:16:07 <nirik> ah, plague... I remember it well.
18:16:33 <smooge> bstinson, pgreco are there plans to do aarch64 for EL7? I am guessing you would need to do just pure upstream kernel for that
18:16:33 * nirik gets coffee, back in a min
18:17:11 <pgreco> smooge, the current working theory is using same userspace as other arches and using kernel from rhel8 (4.18.x)
18:18:07 <pgreco> which would translate to "just a kernel upgrade from 4.14 to 4.18"
18:18:25 <smooge> two would be setting up an koji mbox for epel-alt be feasible?
18:18:32 <bstinson> i think we should consider if EPEL proper wants to maintain aarch64 packages for 7 and let CentOS decide what we want to do (and account for the cross collaboration)
18:18:48 <bstinson> smooge: yes (that's where i was going), but not immediately
18:19:47 <tdawson> If we were to continue aarch64 for EPEL7, it looks like we would at least need a new rpm.  Do we know if we would need any other packages?
18:22:38 <smooge> tdawson, I don't know..
18:22:39 <pgreco> ok, the other question is "what will epel7/aarch64 look like?"
18:22:58 <pgreco> I mean, like other arches (and centos), or like rhel?
18:26:31 <smooge> tdawson, in looking at the packages rolling out.. there were 2500 packages updated between 7.6 and 7.7 and 1100 have some sort of version bump
18:27:07 <tdawson> Ouch
18:27:20 <tdawson> That sounds like a complete rebuild ... almost.
18:28:17 <smooge> well that was me looking at finished packages and not srpms
18:29:50 <smooge> but qt5 got a bump other things got a bump
18:30:08 <smooge> I don't know how many of those are going to change dependencies
18:30:19 <smooge> wheee
18:30:31 <nirik> we would have to use centos 7.7
18:30:43 <smooge> ok I was hoping pbrobinson would be able to come in with some tips
18:31:01 <pgreco> and the inclusion of python3 is the biggest change affecting us, I think
18:31:54 <smooge> nirik, yeah. I was not sure how mixing the streams would do
18:32:15 <nirik> well, it could break, but it's more likely to work than what we have now. ;)
18:32:48 <nirik> which is basically known not to.
18:33:35 <pgreco> nirik, separate target but using the same koji, could it work?
18:33:49 <nirik> that would be a lot of mess.
18:34:14 <pgreco> is there any chance of less mess?? :D
18:34:23 <nirik> IMHO we should try just replacing the external rhel7 aarch64 repos with centos7.7
18:34:25 <nirik> yes
18:34:42 <pgreco> what about noarch signature?
18:35:39 <nirik> signature? you mean comparing noarch subpackages?
18:36:09 <pgreco> afaik, koji doesn't like when it gets the same noarch pkg with different signatures in different repos
18:36:28 <tdawson> OK, so tracked it down somewhat.  This is not really a matter of rpm being updated (although it might be for perl packages) it's a matter of other packages being added/updated.
18:36:42 <pgreco> which would happen if we pull aarch64 repos from centos, and the other arches from rhel
18:37:18 <tdawson> So, my initial thought of "just add the packages that need to be added" is not a course of action.
18:37:20 <nirik> well, koji at build time checks all the noarch subpackages and makes sure they are the same, but thats build time of the epel package, it's not checking the repos.
18:37:43 <nirik> koji itself doesn't check any signatures... it assumes everything in it's store is fine.
18:37:52 <pgreco> nirik, I'm saying before that
18:38:17 <pgreco> it needs all the noarch to be the same in all repos
18:38:24 <nirik> it checks also on import time, but we never import rhel either... we use external repos...
18:38:52 <pgreco> ok, then I love the idea
18:39:29 <nirik> I mean isn't rhel7.6 alt already different from 7.6? I know kernel is... but are there other packages that are different?
18:39:56 <smooge> kernel/boot and I think glibc was a bit newer
18:40:55 <smooge> ok I am ok with this plan of action
18:41:01 <pgreco> minor fixes shouldn't generate problems, rebases (like qt) or additions (like python3) are likely to break things
18:41:52 <pgreco> nirik rhel7.6alt and rhel7.6 only differ in kernel (3.10 vs 4.14)
18:41:53 <smooge> we will just expect packages in aarch64 to need a rebuild or break builds overall for the N time it takes for CentOS-7.x to get out
18:42:14 <nirik> pgreco: are you sure? I see a number of .el7a packages...
18:42:44 <pgreco> maybe arch specific (additions)
18:42:49 <pgreco> which one do you have in mind?
18:43:34 <nirik> trying to find a epel7 package that builds against a .el7a package so I can see if it's doing what I expect...
18:43:41 <nirik> but will take more digging
18:44:17 <smooge> looks like anaconda
18:44:27 <pgreco> that was for 7.5
18:44:42 <pgreco> they fixed anaconda after the release of 7.5 and used el7a
18:44:44 <smooge> I just have a large bucket
18:44:52 <pgreco> then it got unified in 7.6
18:45:25 <pgreco> and it was done specially for power9, didn't apply to aarch64
18:45:35 <pgreco> iirc :D
18:46:08 <smooge> so I don't know when things like rdma or binutils were replaced..
18:46:22 <smooge> ok in that case.. I think we have a plan of action
18:46:36 <nirik> well, or as best of one as we can come up with...
18:47:08 <smooge> I will start syncing c7 trees over and get them merged. I will need a list of them
18:47:38 <nirik> well, 7.7 doesn't exist yet?
18:47:53 <smooge> oh yeah
18:48:12 <pgreco> well, cr is in the works, which should be enough for what we need
18:48:32 <nirik> we could also tell folks to exclude it for now, readd once we have 7.7 in place?
18:48:48 <nirik> but some of them may forget
18:49:00 <pgreco> I'd rather leave it broken for a few days
18:49:10 <pgreco> if we can come up with a PoC fast
18:49:44 <smooge> ok is there naything else on this?
18:49:56 <smooge> we are nearing the end of this meeting
18:49:57 <nirik> sure
18:52:08 <smooge> #topic Open Floor
18:52:28 <smooge> assuming nirik meant sure that we can come up with a PoC versus sure he had something else to say
18:52:55 <nirik> sure. :)
18:53:05 <nirik> (yes, you were correct, I have nothing to say)
18:54:31 <tdawson> Nothing from me
18:54:31 <smooge> ok and with that.. I am going to call this a meeting. I will try to write up what we have come up with as a possible plan and see how it flies in the wind.
18:54:42 <smooge> thank you all for coming and see you next week
18:54:45 <smooge> #endmeeting