14:31:59 <dgilmore> #startmeeting RELENG (2014-11-17)
14:32:00 <zodbot> Meeting started Mon Nov 17 14:31:59 2014 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:32:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:32:14 <dgilmore> #meetingname releng
14:32:14 <zodbot> The meeting name has been set to 'releng'
14:32:14 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson
14:32:14 <zodbot> Current chairs: bochecha dgilmore masta nirik pbrobinson sharkcz tyll
14:32:15 <dgilmore> #topic init process
14:32:37 * nirik is here, but also trying to catch up on morning email and first cup of coffee.
14:32:48 <bochecha> hi everyone
14:33:28 * dgilmore nearly fell asleep
14:33:34 * danofsatx-work is present for informational purposes only
14:34:13 * mizdebsk is there to answer questions related to koschei and srpm repo generation
14:35:37 * drag00n drag00n is here
14:36:30 <dgilmore> lets get going
14:36:34 <dgilmore> #topic Secondary Architectures updates
14:36:36 <dgilmore> #topic Secondary Architectures update - ppc
14:37:04 * pbrobinson is here
14:37:34 <pbrobinson> dgilmore: what am I up for?
14:38:57 <dgilmore> pbrobinson: look at the tpoic
14:39:00 <dgilmore> toppic
14:39:03 <dgilmore> topic
14:39:05 <dgilmore> ppc status
14:39:48 <pbrobinson> Beta RC2 is out, testing in motion
14:39:58 <pbrobinson> hoping this one will go GA
14:40:10 <dgilmore> cool
14:40:24 <pbrobinson> a few blockers found for mainline GA mostly around multi path so hoping to get that closed out RSN
14:41:22 <dgilmore> anything else?
14:41:26 <pbrobinson> nope
14:41:42 <dgilmore> #topic Secondary Architectures update - s390
14:42:00 <dgilmore> sharkcz is not here today as its a public holiday for him
14:42:10 <dgilmore> so not sure we will get an update
14:42:31 <dgilmore> #topic Secondary Architectures update - arm
14:42:41 <dgilmore> pbrobinson: hows arm
14:43:08 <pbrobinson> Beta RC1 is out, it's looking pretty good, nothing particularly blocking so hoping we can sign that off as Beta today
14:43:17 <pbrobinson> builds are looking pretty reasonable
14:43:33 <dgilmore> cool
14:43:41 <pbrobinson> fixed a few packages that were blocking other packages and overall for F-21 we're looking pretty good as well
14:44:04 <pbrobinson> a LUKS issue for both ppc/aarch64 was fixed in 3.17.3
14:44:15 <pbrobinson> so with luck we should be good with that too
14:45:31 <pbrobinson> and that's about it
14:45:59 <dgilmore> cool
14:46:15 <dgilmore> #topic #5967 Enable source repo generation
14:46:16 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5967
14:46:45 <dgilmore> mizdebsk: so we have some major concerns over enabling srpm repo for koji repos
14:47:00 <dgilmore> namely the extra disk and resources needed
14:47:07 <dgilmore> for no apparent benefit
14:47:54 <mizdebsk> before starting to work on koschei i've been told that we can get extra hardware for koji, if needed
14:48:18 <pbrobinson> what is koschei?
14:48:41 <dgilmore> mizdebsk: well we would need to look at netapp disk mostly
14:48:43 <mizdebsk> so i am fine with condition that srpm repo will be enabled once extra hardware is delivered
14:48:45 <nirik> mizdebsk: yeah, disk. ;)
14:48:57 <dgilmore> mizdebsk: and as i understand it thats not possible to extend right now
14:49:00 <mizdebsk> pbrobinson: https://fedoraproject.org/wiki/Koschei
14:49:20 <nirik> we are looking at next years budget and really looking like we might need another shelf to keep up for a while.
14:49:26 <dgilmore> mizdebsk: i do not see any benefit for what is a massive cost
14:49:38 <dgilmore> mizdebsk: so you need to convince us why
14:49:52 <mizdebsk> srpm repo has to be generated somewrere, for koschei
14:49:59 <dgilmore> mizdebsk: why?
14:50:00 <mizdebsk> currently koschei generates it itself
14:50:14 <dgilmore> mizdebsk: I really do not see why it needs to be
14:50:22 <mizdebsk> koschei has to resolve build dependencies of all packages it tracks
14:50:46 <mizdebsk> it can't use repo from compose as delay would be too long - that defeats purpose of CI
14:51:05 <dgilmore> mizdebsk: why can it not use the previous days rawhide and the newly done build?
14:51:17 <mizdebsk> as i said, too long delay
14:51:25 <dgilmore> mizdebsk: why?
14:51:41 <dgilmore> mizdebsk: deps do not change that often
14:52:10 <nirik> perhaps you could give an example?
14:52:46 <dgilmore> mizdebsk: I really would like to understand the problem.
14:52:51 <mizdebsk> koschei has to catch all kind of bugs as soon as they appear, 24 h is too long delay IMO, compared to 20 min (if repo is generated)
14:52:57 <dgilmore> mizdebsk: but I have yet to see it
14:53:38 <mizdebsk> if you see the bug you caused after 30 min or 1h you can untag broken package and submit new build
14:53:48 <mizdebsk> after compose it may not be so easy
14:53:51 <dgilmore> mizdebsk: that does not help make your case. the deps really do not change much. you can easily evaluate the build in question to see if its BuildRequires change
14:53:55 <nirik> dgilmore: I guess the case is 'perl-Foo' is rebuild, you want to know that 'perl-Bar' and 'perl-Baz' fail to build with the new one. You could wait a day for the deps, but you could fix it right after.
14:54:05 <dgilmore> mizdebsk: in fact the repodata can be useless
14:54:24 <dgilmore> mizdebsk: because the buildRequires are embeded at srpm generation time
14:54:50 <mizdebsk> dgilmore: it's definitely not useless
14:55:35 <dgilmore> mizdebsk: say a package on arm doesnt "BuildRequires: foo"  because foo is not built on arm we happen to include the arm built srpm that BuildRequres willnot exist in the repo
14:55:41 <dgilmore> mizdebsk: it really is
14:56:08 <dgilmore> mizdebsk: as you need to evaluate the BuildRequires for teh srpm built on all arches
14:56:30 <dgilmore> to properlly evaluate all possible BuildRequires
14:56:58 <dgilmore> mizdebsk: the BuildRequires can change based on the arch that the srpm is built on
14:57:03 <mizdebsk> possibly, but i don't see any other way to reliably obtain package BR
14:57:32 <mizdebsk> msimacek: any comment on this?
14:57:32 <nirik> if we enable this it's just one repo? or one per arch?
14:58:12 <mizdebsk> nirik: koschei currently runs only on primary rawhide, so only one repo
14:58:13 <dgilmore> mizdebsk: the only possibly way is to rebuild teh srpm for every arch
14:58:43 <dgilmore> mizdebsk: we can not enable source repo creation on only one tag
14:58:48 <nirik> yeah, but then we run into the thing dgilmore is talking about with different contents depending on where it was built.
14:59:29 <dgilmore> mizdebsk: I am having a really hard time seeing the benefit of enabling it
14:59:30 * nirik wishes src.rpms were arch independent... but no such luck.
14:59:34 <dgilmore> when it has a high cost
14:59:50 <dgilmore> and it doesnt really solve the problem its trying to
15:00:21 <dgilmore> mizdebsk: if the cost was low I would gladly enable it
15:00:42 <mizdebsk> dgilmore: i see, but the only alternative we have is to keep generating the repo ourselves
15:00:46 <bochecha> nirik, I've always wished we'd eventually pass a policy for that, and treat arch-dependent srpms as bugs
15:01:12 <dgilmore> bochecha: but they are not bugs
15:01:12 <msimacek> dgilmore, it's better than using a day old repo from compose. The deps don't change much, but when they do, we'll get incorrect data
15:02:09 <msimacek> dgilmore, the ones I already saw were mostly copypaste errors
15:02:14 <dgilmore> msimacek: I think you would get much better and reliable results evaluating the srom in isolation
15:02:33 <dgilmore> by rebuilding the srpm for all arches
15:02:45 <dgilmore> and using a db for all build requires
15:03:01 <dgilmore> you can easily detect when BuildRequires change
15:03:59 <dgilmore> msimacek: the Requires in the srpm repo have to be assumed to be wrong
15:04:33 <nirik> dgilmore: is there a way to scratch build just srpm on particular arches?
15:04:45 <dgilmore> msimacek: so please help me to help you. I still do not really grasp why you think you need it. please can you provide an example?
15:04:55 <dgilmore> nirik: there is no way
15:05:18 <nirik> bummer. that might be handy actually.
15:05:21 <dgilmore> nirik: well you could do a scratch build for each arch
15:05:45 <nirik> with --arch-override?
15:05:46 <dgilmore> since you can override the arches for scratch builds
15:05:49 <dgilmore> yep
15:06:03 <nirik> but the srpm still gets built on $random?
15:06:15 <dgilmore> but that will involve a scratch  build for each arch
15:06:27 <dgilmore> nirik: well the srpm task yes
15:06:34 <dgilmore> but the resulting srpm no
15:06:37 <nirik> ah right, indeed.
15:08:22 <mizdebsk> (koschei doesn't run scratch builds from scm, but from srpms that are results of official builds)
15:08:49 <nirik> yeah, but those src.rpms can be made on any arch...
15:09:00 <nirik> most of the time thats fine...
15:09:06 <dgilmore> mizdebsk: okay, and what exactly does it do?
15:09:22 <mizdebsk> dgilmore: what do you mean?
15:10:11 <dgilmore> mizdebsk: well how does it determine what to build?
15:11:12 <mizdebsk> it is determined by priority, which amond other things, depends on number of build dependencies changed since last build
15:11:27 <mizdebsk> it is explained in detail at https://fedoraproject.org/wiki/Koschei
15:11:28 <dgilmore> mizdebsk: that is very vague
15:11:39 <dgilmore> I have not had time to look at it at all
15:12:58 <nirik> it uses a scoring system right? so it adds points to things that's BR's have changed, and builds the highest scoring ones first...
15:13:13 <dgilmore> mizdebsk: I won't have time to read that until after f21 is out
15:13:19 <nirik> dgilmore: there's also some info with questions and answers on the infra list.
15:13:43 <msimacek> dgilmore, after repo done event, it resolves all srpm deps in the new repo and raises the priority score of packages whose deps changed
15:13:49 <mizdebsk> nirik: something like this
15:14:13 <dgilmore> mizdebsk: so i think you are best of doing this
15:14:29 <dgilmore> keep in a db all deps for all arches of every srpm
15:14:41 <dgilmore> so rebuild the srpm for all arches
15:15:10 <dgilmore> and for every build done in koji fetch the srpm, rebuild it for all arches to get correct BuildRequires
15:15:35 <dgilmore> and compare the BuildRequires to your internal database
15:15:48 <dgilmore> which should be a inexpensive query
15:15:58 <dgilmore> you can then update if deps change
15:16:26 <dgilmore> it seems to be the only way to do it
15:16:27 <mizdebsk> dgilmore: you suggest to rebuild all packages on all arches after every new repo? and that requires less resources than our proposal?
15:16:39 <dgilmore> mizdebsk: no
15:16:56 <dgilmore> mizdebsk: I suggest you locally rebuild the srpm for all arches
15:17:18 <dgilmore> and locally evaluate the srpm after being built for each arch
15:17:40 <dgilmore> mizdebsk: so an example.
15:17:45 <mizdebsk> still that requires a lot of resources
15:17:49 <dgilmore> it is a little vague
15:17:50 <dgilmore> but
15:18:15 <dgilmore> many packages historically have not had valgrind support on %arm arches
15:18:22 <dgilmore> as valgrind used to be missing
15:19:13 <dgilmore> so if you get a srpm that was built on the armv7hl task it could well be misisng valgrind as a BuildRequires
15:19:38 <dgilmore> when the srpm on the next build comes from x86_64 it gains the valgrind dep
15:20:01 <mizdebsk> that's not a big problem for koschei
15:20:27 <mizdebsk> worst case it will miss dep update of valgrint and not trigger scratch build
15:20:27 <dgilmore> mizdebsk: it doesnt really, it requires you build the srpm from spec multiple times but thats a local task that doesnt take long to run
15:21:04 <dgilmore> mizdebsk: if thats not a big deal then neither is using yesterdays rawhide and considering the build in question
15:21:24 <mizdebsk> we don'n require BR to be 100% accurate, 99% cases is good enough approximation for our use case (koschei)
15:21:51 <mizdebsk> this is totally different
15:22:05 <dgilmore> mizdebsk: then using rawhides repodata and grabbing the srpm and evaluating it in isolation is nota  big deal either
15:22:43 <dgilmore> mizdebsk: what exactly does koschei use the srpm metadata for?
15:22:47 <mizdebsk> we want to catch bugs introduced by changing BR, we have to use latest srpms to catch bugs asap
15:22:48 <dgilmore> mizdebsk: its really not
15:22:55 <dgilmore> you need to explain why it is
15:23:07 <dgilmore> though at this point the metting is over time
15:23:28 <mizdebsk> dgilmore: i can explain in the ticket
15:24:02 <dgilmore> mizdebsk: can you please update the ticket and explain clearly what koschei needs the metadata for. perhaps we can come up with a more efficent way to solve the problem
15:24:17 <mizdebsk> dgilmore: sure, i'll do that
15:24:22 <mizdebsk> thanks
15:25:04 <dgilmore> mizdebsk: I am not trying to be difficult, just trying to understand why and what its doing. I would like to find a less expansive way to solve the problem
15:25:18 <mizdebsk> understood
15:29:55 <dgilmore> mizdebsk: just as an example of teh cost in disk, in the last 24 hours which is mostly a weekend so rawhide churn is quiet. there was 81 new reawhide repos
15:30:04 <dgilmore> 20Mb data per repo
15:30:14 <dgilmore> 1620M
15:30:27 <dgilmore> rawhide always gets 2 repos
15:30:40 <dgilmore> due to some internal things in how we have things setup
15:30:52 <mizdebsk> i need srpm repo only in one repo, f22-build
15:30:58 <dgilmore> so 3240M a day
15:30:59 <mizdebsk> not in "rawhide"
15:31:12 <dgilmore> mizdebsk:same thing
15:31:23 <dgilmore> mizdebsk: and we have to turn it on for all repos
15:31:38 <dgilmore> so right now thats epel5 6 and 7 and fedora 19 20 21 22
15:31:51 <dgilmore> any side tag means extar newRepos
15:32:01 <mizdebsk> i thought that it was possible to enable it per-repo
15:32:15 <dgilmore> mizdebsk: it is not
15:32:49 <dgilmore> we are looking at using a lot of disk for srpm repos
15:33:27 <mizdebsk> if it is not possible to enable it just for one repo then i can prepare patch for koji to implement this
15:33:56 <dgilmore> mizdebsk: https://git.fedorahosted.org/cgit/koji/tree/util/kojira.conf its a global setting
15:34:40 <dgilmore> mizdebsk: we would then need to update configs everytime we branch
15:35:00 <dgilmore> it then becomes an ongoing maintainence issue
15:36:35 <mizdebsk> dgilmore: i see, thx, i start to understand better why you don't like the idea
15:37:23 <dgilmore> mizdebsk: if we can do it for one repo with the number of repos we have for f22-build right now we would use an extra 10760M of disk which is not end of the world
15:38:17 <dgilmore> 47820M for all current repos
15:38:30 <dgilmore> which goes back a week
15:38:50 <dgilmore> anyway mizdebsk I look forward to seeing your ticket update
15:38:58 <dgilmore> and lets end the meeting
15:40:30 <dgilmore> mizdebsk: once f21 is done there will be more cycles to look at things also
15:40:50 <dgilmore> #endmeeting