14:31:59 <dgilmore> #startmeeting RELENG (2014-11-17) 14:32:00 <zodbot> Meeting started Mon Nov 17 14:31:59 2014 UTC. The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:32:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:32:14 <dgilmore> #meetingname releng 14:32:14 <zodbot> The meeting name has been set to 'releng' 14:32:14 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson 14:32:14 <zodbot> Current chairs: bochecha dgilmore masta nirik pbrobinson sharkcz tyll 14:32:15 <dgilmore> #topic init process 14:32:37 * nirik is here, but also trying to catch up on morning email and first cup of coffee. 14:32:48 <bochecha> hi everyone 14:33:28 * dgilmore nearly fell asleep 14:33:34 * danofsatx-work is present for informational purposes only 14:34:13 * mizdebsk is there to answer questions related to koschei and srpm repo generation 14:35:37 * drag00n drag00n is here 14:36:30 <dgilmore> lets get going 14:36:34 <dgilmore> #topic Secondary Architectures updates 14:36:36 <dgilmore> #topic Secondary Architectures update - ppc 14:37:04 * pbrobinson is here 14:37:34 <pbrobinson> dgilmore: what am I up for? 14:38:57 <dgilmore> pbrobinson: look at the tpoic 14:39:00 <dgilmore> toppic 14:39:03 <dgilmore> topic 14:39:05 <dgilmore> ppc status 14:39:48 <pbrobinson> Beta RC2 is out, testing in motion 14:39:58 <pbrobinson> hoping this one will go GA 14:40:10 <dgilmore> cool 14:40:24 <pbrobinson> a few blockers found for mainline GA mostly around multi path so hoping to get that closed out RSN 14:41:22 <dgilmore> anything else? 14:41:26 <pbrobinson> nope 14:41:42 <dgilmore> #topic Secondary Architectures update - s390 14:42:00 <dgilmore> sharkcz is not here today as its a public holiday for him 14:42:10 <dgilmore> so not sure we will get an update 14:42:31 <dgilmore> #topic Secondary Architectures update - arm 14:42:41 <dgilmore> pbrobinson: hows arm 14:43:08 <pbrobinson> Beta RC1 is out, it's looking pretty good, nothing particularly blocking so hoping we can sign that off as Beta today 14:43:17 <pbrobinson> builds are looking pretty reasonable 14:43:33 <dgilmore> cool 14:43:41 <pbrobinson> fixed a few packages that were blocking other packages and overall for F-21 we're looking pretty good as well 14:44:04 <pbrobinson> a LUKS issue for both ppc/aarch64 was fixed in 3.17.3 14:44:15 <pbrobinson> so with luck we should be good with that too 14:45:31 <pbrobinson> and that's about it 14:45:59 <dgilmore> cool 14:46:15 <dgilmore> #topic #5967 Enable source repo generation 14:46:16 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5967 14:46:45 <dgilmore> mizdebsk: so we have some major concerns over enabling srpm repo for koji repos 14:47:00 <dgilmore> namely the extra disk and resources needed 14:47:07 <dgilmore> for no apparent benefit 14:47:54 <mizdebsk> before starting to work on koschei i've been told that we can get extra hardware for koji, if needed 14:48:18 <pbrobinson> what is koschei? 14:48:41 <dgilmore> mizdebsk: well we would need to look at netapp disk mostly 14:48:43 <mizdebsk> so i am fine with condition that srpm repo will be enabled once extra hardware is delivered 14:48:45 <nirik> mizdebsk: yeah, disk. ;) 14:48:57 <dgilmore> mizdebsk: and as i understand it thats not possible to extend right now 14:49:00 <mizdebsk> pbrobinson: https://fedoraproject.org/wiki/Koschei 14:49:20 <nirik> we are looking at next years budget and really looking like we might need another shelf to keep up for a while. 14:49:26 <dgilmore> mizdebsk: i do not see any benefit for what is a massive cost 14:49:38 <dgilmore> mizdebsk: so you need to convince us why 14:49:52 <mizdebsk> srpm repo has to be generated somewrere, for koschei 14:49:59 <dgilmore> mizdebsk: why? 14:50:00 <mizdebsk> currently koschei generates it itself 14:50:14 <dgilmore> mizdebsk: I really do not see why it needs to be 14:50:22 <mizdebsk> koschei has to resolve build dependencies of all packages it tracks 14:50:46 <mizdebsk> it can't use repo from compose as delay would be too long - that defeats purpose of CI 14:51:05 <dgilmore> mizdebsk: why can it not use the previous days rawhide and the newly done build? 14:51:17 <mizdebsk> as i said, too long delay 14:51:25 <dgilmore> mizdebsk: why? 14:51:41 <dgilmore> mizdebsk: deps do not change that often 14:52:10 <nirik> perhaps you could give an example? 14:52:46 <dgilmore> mizdebsk: I really would like to understand the problem. 14:52:51 <mizdebsk> koschei has to catch all kind of bugs as soon as they appear, 24 h is too long delay IMO, compared to 20 min (if repo is generated) 14:52:57 <dgilmore> mizdebsk: but I have yet to see it 14:53:38 <mizdebsk> if you see the bug you caused after 30 min or 1h you can untag broken package and submit new build 14:53:48 <mizdebsk> after compose it may not be so easy 14:53:51 <dgilmore> mizdebsk: that does not help make your case. the deps really do not change much. you can easily evaluate the build in question to see if its BuildRequires change 14:53:55 <nirik> dgilmore: I guess the case is 'perl-Foo' is rebuild, you want to know that 'perl-Bar' and 'perl-Baz' fail to build with the new one. You could wait a day for the deps, but you could fix it right after. 14:54:05 <dgilmore> mizdebsk: in fact the repodata can be useless 14:54:24 <dgilmore> mizdebsk: because the buildRequires are embeded at srpm generation time 14:54:50 <mizdebsk> dgilmore: it's definitely not useless 14:55:35 <dgilmore> mizdebsk: say a package on arm doesnt "BuildRequires: foo" because foo is not built on arm we happen to include the arm built srpm that BuildRequres willnot exist in the repo 14:55:41 <dgilmore> mizdebsk: it really is 14:56:08 <dgilmore> mizdebsk: as you need to evaluate the BuildRequires for teh srpm built on all arches 14:56:30 <dgilmore> to properlly evaluate all possible BuildRequires 14:56:58 <dgilmore> mizdebsk: the BuildRequires can change based on the arch that the srpm is built on 14:57:03 <mizdebsk> possibly, but i don't see any other way to reliably obtain package BR 14:57:32 <mizdebsk> msimacek: any comment on this? 14:57:32 <nirik> if we enable this it's just one repo? or one per arch? 14:58:12 <mizdebsk> nirik: koschei currently runs only on primary rawhide, so only one repo 14:58:13 <dgilmore> mizdebsk: the only possibly way is to rebuild teh srpm for every arch 14:58:43 <dgilmore> mizdebsk: we can not enable source repo creation on only one tag 14:58:48 <nirik> yeah, but then we run into the thing dgilmore is talking about with different contents depending on where it was built. 14:59:29 <dgilmore> mizdebsk: I am having a really hard time seeing the benefit of enabling it 14:59:30 * nirik wishes src.rpms were arch independent... but no such luck. 14:59:34 <dgilmore> when it has a high cost 14:59:50 <dgilmore> and it doesnt really solve the problem its trying to 15:00:21 <dgilmore> mizdebsk: if the cost was low I would gladly enable it 15:00:42 <mizdebsk> dgilmore: i see, but the only alternative we have is to keep generating the repo ourselves 15:00:46 <bochecha> nirik, I've always wished we'd eventually pass a policy for that, and treat arch-dependent srpms as bugs 15:01:12 <dgilmore> bochecha: but they are not bugs 15:01:12 <msimacek> dgilmore, it's better than using a day old repo from compose. The deps don't change much, but when they do, we'll get incorrect data 15:02:09 <msimacek> dgilmore, the ones I already saw were mostly copypaste errors 15:02:14 <dgilmore> msimacek: I think you would get much better and reliable results evaluating the srom in isolation 15:02:33 <dgilmore> by rebuilding the srpm for all arches 15:02:45 <dgilmore> and using a db for all build requires 15:03:01 <dgilmore> you can easily detect when BuildRequires change 15:03:59 <dgilmore> msimacek: the Requires in the srpm repo have to be assumed to be wrong 15:04:33 <nirik> dgilmore: is there a way to scratch build just srpm on particular arches? 15:04:45 <dgilmore> msimacek: so please help me to help you. I still do not really grasp why you think you need it. please can you provide an example? 15:04:55 <dgilmore> nirik: there is no way 15:05:18 <nirik> bummer. that might be handy actually. 15:05:21 <dgilmore> nirik: well you could do a scratch build for each arch 15:05:45 <nirik> with --arch-override? 15:05:46 <dgilmore> since you can override the arches for scratch builds 15:05:49 <dgilmore> yep 15:06:03 <nirik> but the srpm still gets built on $random? 15:06:15 <dgilmore> but that will involve a scratch build for each arch 15:06:27 <dgilmore> nirik: well the srpm task yes 15:06:34 <dgilmore> but the resulting srpm no 15:06:37 <nirik> ah right, indeed. 15:08:22 <mizdebsk> (koschei doesn't run scratch builds from scm, but from srpms that are results of official builds) 15:08:49 <nirik> yeah, but those src.rpms can be made on any arch... 15:09:00 <nirik> most of the time thats fine... 15:09:06 <dgilmore> mizdebsk: okay, and what exactly does it do? 15:09:22 <mizdebsk> dgilmore: what do you mean? 15:10:11 <dgilmore> mizdebsk: well how does it determine what to build? 15:11:12 <mizdebsk> it is determined by priority, which amond other things, depends on number of build dependencies changed since last build 15:11:27 <mizdebsk> it is explained in detail at https://fedoraproject.org/wiki/Koschei 15:11:28 <dgilmore> mizdebsk: that is very vague 15:11:39 <dgilmore> I have not had time to look at it at all 15:12:58 <nirik> it uses a scoring system right? so it adds points to things that's BR's have changed, and builds the highest scoring ones first... 15:13:13 <dgilmore> mizdebsk: I won't have time to read that until after f21 is out 15:13:19 <nirik> dgilmore: there's also some info with questions and answers on the infra list. 15:13:43 <msimacek> dgilmore, after repo done event, it resolves all srpm deps in the new repo and raises the priority score of packages whose deps changed 15:13:49 <mizdebsk> nirik: something like this 15:14:13 <dgilmore> mizdebsk: so i think you are best of doing this 15:14:29 <dgilmore> keep in a db all deps for all arches of every srpm 15:14:41 <dgilmore> so rebuild the srpm for all arches 15:15:10 <dgilmore> and for every build done in koji fetch the srpm, rebuild it for all arches to get correct BuildRequires 15:15:35 <dgilmore> and compare the BuildRequires to your internal database 15:15:48 <dgilmore> which should be a inexpensive query 15:15:58 <dgilmore> you can then update if deps change 15:16:26 <dgilmore> it seems to be the only way to do it 15:16:27 <mizdebsk> dgilmore: you suggest to rebuild all packages on all arches after every new repo? and that requires less resources than our proposal? 15:16:39 <dgilmore> mizdebsk: no 15:16:56 <dgilmore> mizdebsk: I suggest you locally rebuild the srpm for all arches 15:17:18 <dgilmore> and locally evaluate the srpm after being built for each arch 15:17:40 <dgilmore> mizdebsk: so an example. 15:17:45 <mizdebsk> still that requires a lot of resources 15:17:49 <dgilmore> it is a little vague 15:17:50 <dgilmore> but 15:18:15 <dgilmore> many packages historically have not had valgrind support on %arm arches 15:18:22 <dgilmore> as valgrind used to be missing 15:19:13 <dgilmore> so if you get a srpm that was built on the armv7hl task it could well be misisng valgrind as a BuildRequires 15:19:38 <dgilmore> when the srpm on the next build comes from x86_64 it gains the valgrind dep 15:20:01 <mizdebsk> that's not a big problem for koschei 15:20:27 <mizdebsk> worst case it will miss dep update of valgrint and not trigger scratch build 15:20:27 <dgilmore> mizdebsk: it doesnt really, it requires you build the srpm from spec multiple times but thats a local task that doesnt take long to run 15:21:04 <dgilmore> mizdebsk: if thats not a big deal then neither is using yesterdays rawhide and considering the build in question 15:21:24 <mizdebsk> we don'n require BR to be 100% accurate, 99% cases is good enough approximation for our use case (koschei) 15:21:51 <mizdebsk> this is totally different 15:22:05 <dgilmore> mizdebsk: then using rawhides repodata and grabbing the srpm and evaluating it in isolation is nota big deal either 15:22:43 <dgilmore> mizdebsk: what exactly does koschei use the srpm metadata for? 15:22:47 <mizdebsk> we want to catch bugs introduced by changing BR, we have to use latest srpms to catch bugs asap 15:22:48 <dgilmore> mizdebsk: its really not 15:22:55 <dgilmore> you need to explain why it is 15:23:07 <dgilmore> though at this point the metting is over time 15:23:28 <mizdebsk> dgilmore: i can explain in the ticket 15:24:02 <dgilmore> mizdebsk: can you please update the ticket and explain clearly what koschei needs the metadata for. perhaps we can come up with a more efficent way to solve the problem 15:24:17 <mizdebsk> dgilmore: sure, i'll do that 15:24:22 <mizdebsk> thanks 15:25:04 <dgilmore> mizdebsk: I am not trying to be difficult, just trying to understand why and what its doing. I would like to find a less expansive way to solve the problem 15:25:18 <mizdebsk> understood 15:29:55 <dgilmore> mizdebsk: just as an example of teh cost in disk, in the last 24 hours which is mostly a weekend so rawhide churn is quiet. there was 81 new reawhide repos 15:30:04 <dgilmore> 20Mb data per repo 15:30:14 <dgilmore> 1620M 15:30:27 <dgilmore> rawhide always gets 2 repos 15:30:40 <dgilmore> due to some internal things in how we have things setup 15:30:52 <mizdebsk> i need srpm repo only in one repo, f22-build 15:30:58 <dgilmore> so 3240M a day 15:30:59 <mizdebsk> not in "rawhide" 15:31:12 <dgilmore> mizdebsk:same thing 15:31:23 <dgilmore> mizdebsk: and we have to turn it on for all repos 15:31:38 <dgilmore> so right now thats epel5 6 and 7 and fedora 19 20 21 22 15:31:51 <dgilmore> any side tag means extar newRepos 15:32:01 <mizdebsk> i thought that it was possible to enable it per-repo 15:32:15 <dgilmore> mizdebsk: it is not 15:32:49 <dgilmore> we are looking at using a lot of disk for srpm repos 15:33:27 <mizdebsk> if it is not possible to enable it just for one repo then i can prepare patch for koji to implement this 15:33:56 <dgilmore> mizdebsk: https://git.fedorahosted.org/cgit/koji/tree/util/kojira.conf its a global setting 15:34:40 <dgilmore> mizdebsk: we would then need to update configs everytime we branch 15:35:00 <dgilmore> it then becomes an ongoing maintainence issue 15:36:35 <mizdebsk> dgilmore: i see, thx, i start to understand better why you don't like the idea 15:37:23 <dgilmore> mizdebsk: if we can do it for one repo with the number of repos we have for f22-build right now we would use an extra 10760M of disk which is not end of the world 15:38:17 <dgilmore> 47820M for all current repos 15:38:30 <dgilmore> which goes back a week 15:38:50 <dgilmore> anyway mizdebsk I look forward to seeing your ticket update 15:38:58 <dgilmore> and lets end the meeting 15:40:30 <dgilmore> mizdebsk: once f21 is done there will be more cycles to look at things also 15:40:50 <dgilmore> #endmeeting