16:33:50 <dgilmore> #startmeeting RELENG (2015-03-09)
16:33:50 <zodbot> Meeting started Mon Mar  9 16:33:50 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:33:50 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:33:56 <dgilmore> #meetingname releng
16:33:56 <zodbot> The meeting name has been set to 'releng'
16:33:56 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou
16:33:56 <zodbot> Current chairs: bochecha dgilmore masta nirik pbrobinson pingou sharkcz tyll
16:33:58 <dgilmore> #topic init process
16:34:23 <nirik> morning
16:34:40 <tyll> Hi
16:36:40 <dgilmore> lets ping people in #fedora-releng
16:37:54 * jreznik is here
16:38:32 <dgilmore> #topic #6016 Use fedpkg-minimal in Fedora buildroots
16:38:37 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6016
16:38:53 <dgilmore> #action dgilmore to make this change everywhere this week
16:39:04 <dgilmore> pbabinca: has built it everywhere
16:39:18 <dgilmore> #topic #5963 Orphaned vulnerable packages in EPEL
16:39:23 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5963
16:39:38 <dgilmore> tyll: I will let you know when I have made the srpm buildroot change
16:39:44 <tyll> dgilmore: ok
16:40:06 <dgilmore> #topic #5775 switch to partitioned image for EC2; remove editing of menu.lst from image processing
16:40:10 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5775
16:40:16 <dgilmore> I am going to close this
16:40:24 * pbrobinson is here
16:40:28 <dgilmore> afaik it is what we have done with the upload service
16:40:31 * threebean is here
16:42:38 <dgilmore> #topic #5870 rawhide signing
16:42:44 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5870
16:43:18 <dgilmore> I really am not sure how much we should pursue this until we get sigul sorted
16:43:21 <nirik> I think we are stalled until sigul is sored
16:43:22 * roshi is here as well
16:43:23 <nirik> sorted
16:43:36 <dgilmore> nirik: :)
16:43:46 <tyll> I can start the autosigner again and we can see how good it works nowadays
16:43:48 <dgilmore> sigul is just a bit too fragile
16:44:16 <nirik> I did a run yesterday and this morning and just started another one
16:44:17 <dgilmore> tyll: realistically its only going to be reliable with a batch size of 1
16:44:20 <nirik> tyll: it does not.
16:44:33 <nirik> I tried various things yesterday and it always locked unless you use a batch size of 1
16:44:47 <dgilmore> batch size of 1 has never locked up
16:44:53 <tyll> I see, I can also start it with a batch size of 1 and we can see if it ever catches up
16:44:54 <dgilmore> but uses a different code path in sigul
16:44:58 <nirik> right, I thik thats a different path
16:44:59 <nirik> ha.
16:45:07 * nirik is really on the same page with dgilmore today. :0
16:45:13 <dgilmore> nirik: seems so :)
16:45:40 <tyll> it uses a receiver and sender thread for a batch size > 1
16:45:53 <threebean> longer term, should we look at patching sigul to get larger batch sizes working? (forgive me if this has already been discussed)
16:46:00 <tyll> btw. there are some GSoC applicants that might help fixing sigul
16:46:12 <dgilmore> threebean: it works its just not stable
16:46:21 <dgilmore> there seems to be some race conditions
16:46:21 <threebean> s/working/working all the time/
16:46:36 <nirik> yeah, fixing it for batches would be lovely.
16:46:42 <nirik> it can go much much faster with them
16:46:51 <dgilmore> tyll: hopefully one of them gets accepted and can get it working
16:47:04 <threebean> tyll: are you managing that gsoc application process?  or.. who is?
16:47:46 <tyll> Miloslav was not so happy about having to review patches btw because of missing time, I hope we can support him much there
16:48:19 <tyll> threebean: dgilmore and me are currently the mentors/contacts on the wiki page, so we get most mails
16:48:57 <tyll> threebean: afaik they will have to apply via the google app and then every (possible) mentor will review applicants together - this is how I understood the process
16:49:05 * threebean nods
16:49:11 <tyll> threebean: would you like to co-mentor?
16:49:25 <threebean> yeah, typically the people who work with the applicants ahead of time will have the best idea about whether or not someone could really pull it off.
16:49:57 <threebean> tyll: but yeah, sign me up.  I'd be glad to help review patches too, but after a quick search around the sigul trac I'm not sure where to do that.
16:50:32 <tyll> threebean: the patches will still have to be written, afaik since rel-eng is the only sigul consumer, there were not many bug reports
16:51:24 <nirik> sigul is pretty complex code... hopefully someone can find the race condition.
16:52:35 <dgilmore> threebean: sigul is complex and not actively developed
16:53:06 <threebean> I understand. we should establish a place for development discussion and code review to happen, then.
16:53:15 <threebean> perhaps the releng list?
16:54:13 <dgilmore> threebean: yeah that would be fine
16:55:47 <tyll> The GSoC project idea description is here btw: https://fedoraproject.org/wiki/Summer_coding_ideas_for_2015#Improve_Sigul_Signing_Server
16:57:13 <dgilmore> #topic Secondary Architectures updates
16:57:14 <dgilmore> #topic Secondary Architectures update - ppc
16:57:24 <pbrobinson> preparing for a RC1
16:57:26 <dgilmore> pbrobinson: how is ppc this week?
16:57:31 <dgilmore> cool
16:57:50 <pbrobinson> so just dealing with all the various bits and pieces surrounding that
16:57:52 <dgilmore> #topic Secondary Architectures update - s390
16:57:58 <dgilmore> Dan is on holidays afaik
16:58:04 <jreznik> yep, he is
16:58:06 <pbrobinson> yup
16:58:08 <dgilmore> #topic Secondary Architectures update - arm
16:58:15 <dgilmore> how is aarch64?
16:58:21 <pbrobinson> preparing for a RC1, basically the same as ppc
16:58:39 <dgilmore> awesome
16:58:55 <dgilmore> #topic Open Floor
16:59:01 <dgilmore> I have a quick item
16:59:14 <pbrobinson> initial for both aarch64/ppc will be similar to F-21, and we'll be adding extra bits probably post alpha
16:59:17 <nirik> I had a few too. ;)
16:59:22 <dgilmore> with DST in the US starting yesterday do we want to follow DST or stay at 16:40UTC
16:59:24 <moezroy> 2 mass rebuilds
16:59:51 <dgilmore> 16:30UTC that is
16:59:54 <nirik> dgilmore: I am fine with either time.
17:00:06 <dgilmore> okay. lets stick with UTC for now
17:00:23 <dgilmore> nirik: what is your items?
17:00:38 <tyll> In Europe DST starts on the last Sunday this month
17:00:50 <dgilmore> tyll: right
17:00:55 <nirik> 1. I commented on and closed a bunch of tickets. Please feel free to reopen or add comments to anything anyone disagrees with or has more info on.
17:01:20 <nirik> 2. I had a few tickets I was unsure of. do we want to quickly go over them? or next week?
17:01:37 <dgilmore> nirik: we can quickly go over them
17:01:54 <nirik> 3. pbrobinson: can you let me know when you are done with secondary sigul for rc's? I hope to find time to reinstall secondary sigul, but don't want to interfere with secondary releases.
17:02:10 <walters> was https://fedorahosted.org/rel-eng/ticket/6119 not in the queue?
17:02:10 <nirik> ok.
17:02:17 <nirik> https://fedorahosted.org/rel-eng/ticket/6008 <- is this done?
17:02:38 <walters> maybe it needs a meeting keyword?  /me adds
17:02:45 <pbrobinson> nirik: sure, how long does it take for reinstall?
17:02:45 <nirik> walters: yeah, meeting keyword...
17:02:54 <dgilmore> walters: It was not but I do need to follow up on it
17:03:00 <nirik> pbrobinson: I will be moving them to ansible/rhel7, so probibly a few hours?
17:03:13 <dgilmore> nirik: it is not done, I need to build a pungi update first
17:03:22 <nirik> ok, cool.
17:03:25 <dgilmore> I have a couple of other pungi patches I need to get in
17:03:39 <nirik> https://fedorahosted.org/rel-eng/ticket/5866 <- anything left to do here?
17:04:12 <dgilmore> not really, I was waiting on feedback that never came
17:04:12 <nirik> oh, and that reminds me, we are getting 10ssd's for armv7 builders. I was going to stick 8 or 9 in builders and perhaps releng00?
17:04:32 <dgilmore> nirik: sure. though I am not sure they will help much
17:04:48 <dgilmore> but it would let us put more in the channel for eclipse
17:04:50 <nirik> finally there is: https://fedorahosted.org/rel-eng/ticket/5665 I can do this as I update those images. Does anyone object to me doing so?
17:05:07 <nirik> dgilmore: I was hoping we could put a few more things in there... gcc in particular.
17:05:34 <dgilmore> nirik: I am still not really sure what it is that they want there
17:06:02 <nirik> dgilmore: they want a single signed file with all the checksums from http://boot.fedoraproject.org/download
17:06:17 <nirik> right now that just in the web content
17:07:14 <nirik> thats all I had off hand.
17:07:25 <moezroy> would it be possible for 2 mass rebuilds for rawhide? One right away, and another after a month or so when gcc5 becomes more stable?
17:07:25 <dgilmore> okay. I do not read the request as that
17:07:31 <dgilmore> moezroy: Please wait
17:07:35 <moezroy> sorry
17:08:00 <dgilmore> moezroy: nirik put his hand up first you are next
17:08:21 <dgilmore> nirik: I would like a clearer request on that one
17:08:22 <nirik> dgilmore: I can try and ask for more info. the link in the request is not at all right.
17:08:49 <dgilmore> nirik: we would want to update the process that makes the composes to creat the file they need
17:08:58 <dgilmore> so we need more info
17:09:51 <nirik> dgilmore: the process is me. I build those when requested or once a cycle.
17:09:57 <nirik> we don't have any formal build setup for it.
17:10:01 <nirik> we should.
17:10:16 <dgilmore> moezroy: 2 mass rebuilds not likely. we have been waiting on jakub to say that gcc is ready for the massrebuild. we need to come up with a new process as mass rebuilds are not ordered and because of the c++ abi change they need to be ordered
17:10:41 <dgilmore> nirik: okay. we should be making the content as part of a compose
17:10:56 <nirik> dgilmore: sure, should I file a new ticket on that?
17:11:08 * nirik isn't sure how it would work out, but we could.
17:12:10 <dgilmore> nirik: yeah
17:12:18 <moezroy> dgilmore, hmm ok
17:12:27 <dgilmore> moezroy: why do you want 2 mass rebuilds?
17:13:00 <moezroy> there are some PIE patches landing right now in gcc5
17:13:07 <moezroy> which improves their performance
17:13:26 <dgilmore> okay
17:13:36 <pbrobinson> moezroy: it's quite a bit of effort so the plan was to rebuild once gcc5 was stable
17:13:36 <moezroy> if a rebuild is done now, we would know the scope of the hardening change
17:13:38 <dgilmore> not really a reason to do two builds
17:14:19 <dgilmore> the gcc developers will ask for a mass rebuild when gcc is ready for it
17:14:46 <moezroy> I am thinking many of the hardening problems may get fixed automagically after everything is rebuild ;)
17:15:33 <pbrobinson> IMO the terms "think" and "automagically" are mutually exclusive, I tend to prefer to deal in facts, not theory
17:15:33 <nirik> which hardening problems? I'm aware of a few, but those packages just disabled it while it was being looked at.
17:15:50 <moezroy> pbrobinson, I assumed it was a just a script which interfaces with koji for the mass rebuild?
17:16:06 <pbrobinson> moezroy: it's but one component
17:17:10 <tyll> disabling is a workaround until everything can be fixed, the earlier the rebuild the better it is known how many/which packages can be fixed as part of a GSoC project
17:17:33 <dgilmore> moezroy: its a script that interfaces with git and koji, but the cost of a mass rebuild is high
17:17:39 <dgilmore> it takes time, has fall out
17:17:41 <pbrobinson> tyll: is there a tracking bug to track the packages that have been disabled?
17:17:45 <dgilmore> its not something done lightly
17:17:52 <dgilmore> it creates churn on the mirrors
17:18:09 <pbrobinson> it creates churn on secondary arches too
17:18:10 <moezroy> pbrobinson, https://bugzilla.redhat.com/show_bug.cgi?id=1199775
17:19:01 <pbrobinson> moezroy: that BZ appears to be missing some eg all of xorg
17:19:13 <nirik> the script doesn't update the buildroot it does it in a side tag...
17:19:24 <nirik> pbrobinson: I think it's just server. but I could be mistaken
17:19:28 <dgilmore> nirik: we are going to have to change that for the ABI bump
17:19:40 <pbrobinson> nirik: they went through and disabled it on all the drivers too
17:19:59 <tyll> pbrobinson: yes, there is tooling planned to automatically check all pkgs for disabled hardening flags
17:20:26 <tyll> pbrobinson: the hardening should only affect binaries/tools, so disabling on the drivers should not matter
17:20:54 <nirik> pbrobinson: ok.
17:20:57 <moezroy> nirik, why do you do the rebuild on a sidetag? instead of the real rawhide -- i.e. in real time?
17:21:12 <dgilmore> moezroy: to avoid buildroot churn
17:21:30 <moezroy> what is buildroot churn?
17:21:33 <dgilmore> moezroy: as it is expensive and slows the process down
17:21:55 <dgilmore> moezroy: when you do a build the contents that is available to populate the buildroot changes
17:22:02 <nirik> moezroy: because there's no script that can figure out the exact build order. And there's circular deps too.
17:22:13 <dgilmore> moezroy: when you rebuild 17k packages you would get massive churn in the packages
17:22:33 <dgilmore> each time a build completes we would get 2 newRepo tasks
17:22:45 <dgilmore> though ieach one would include multiple builds
17:22:53 <tyll> FYI: for the hardening change, we only need to rebuild arch packages that contain programs and not just libs
17:22:55 <dgilmore> it consumes a lot of resources needlessly
17:23:17 <dgilmore> tyll: the cflags will be changed on the libs also
17:23:23 <nirik> well, it would take a lot longer to do a mass rebuild if we tried to update the buildroot for each set of packages.
17:24:05 <moezroy> dgilmore, nirik,  ok
17:24:15 <tyll> dgilmore: afaik only the binaries need to be compiled differently, not the libs, as they are position independent already
17:24:19 <nirik> if we can make a script that can figure this out it would be handy in this case (abi change in gcc)... but it's not going to be easy at all.
17:24:20 <dgilmore> nirik: we would need to do it only for the c++ builds
17:24:29 <dgilmore> nirik: since there is an ABI change
17:24:42 <dgilmore> its not going to be easy or simple
17:24:52 <nirik> dgilmore: sure. Ideally it would be nice to have a script that rebuild (with new buildroots) everything in order that it needs.
17:24:59 <nirik> but circular deps...
17:25:09 <pbrobinson> there's probably reasons in gcc5 to rebuild standard C stuff too
17:25:11 <nirik> circular buildrequires rather
17:25:13 <moezroy> tyll, the libs are getting z now?
17:26:09 <nirik> I mean we could just use mockchain. ;)
17:26:11 * nirik runs
17:26:15 <dgilmore> nirik: :P
17:26:54 <dgilmore> moezroy: so the answer to your initial question is we will not do 2 mass rebuilds. but as soon as the gcc devs say gcc is ready there will be one
17:26:56 <nirik> in fact... I wonder...
17:27:19 <nirik> if we could beef up copr-dev on the new cloud and try a mass rebuild of rawhide.
17:27:30 <nirik> probibly take it a good while, but might be useful
17:28:14 <tyll> moezroy: but afaik this is not needed at least for ASLR
17:29:23 <dgilmore> nirik: maybe do a matt domsch style FTBFS check
17:29:51 <nirik> dgilmore: yeah, the nice thing about copr is that it does use mockchain... so it would rebuild and rebuild until the same set of things failed.
17:30:54 <dgilmore> anyone have anything else for open floor?
17:32:12 <dgilmore> #endmeeting