16:34:36 <masta> #startmeeting releng
16:34:37 <zodbot> Meeting started Mon Feb 17 16:34:36 2014 UTC.  The chair is masta. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:34:37 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:34:37 <masta> #meetingname releng
16:34:37 <masta> #topic === Releng Roll Call ===
16:34:37 <masta> #chair dgilmore dwa nirik
16:34:37 <zodbot> The meeting name has been set to 'releng'
16:34:37 <zodbot> Current chairs: dgilmore dwa masta nirik
16:34:48 <masta> Howdy folks
16:35:13 <sharkcz> hi
16:35:35 <masta> #chair sharkcz
16:35:35 <zodbot> Current chairs: dgilmore dwa masta nirik sharkcz
16:36:00 <janeznemanic> hi
16:36:02 <dgilmore> hey
16:37:03 <dgilmore> hi janeznemanic
16:37:03 <masta> anybody seen dwa today?
16:37:23 <tyll> Hi
16:37:28 <dgilmore> nope and hes not online internally
16:37:32 <dgilmore> hey tyll
16:37:59 <dgilmore> #chair tyll
16:37:59 <zodbot> Current chairs: dgilmore dwa masta nirik sharkcz tyll
16:38:05 * nirik is here
16:39:34 <dgilmore> #topic tickets
16:39:49 <dgilmore> https://fedorahosted.org/rel-eng/report/10
16:39:58 <dgilmore> we have two tickets today
16:40:22 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5854
16:40:27 <dgilmore> first ticket
16:41:08 <dgilmore> not really sure why its a meeting item
16:41:21 <tyll> iirc nirik asked for this
16:41:33 <nirik> I asked him to file this when he complained that he's been trying to get it applied for many years.
16:41:33 <dgilmore> but I don't know we want to apply the patch, its not really clear what problem its trying to solve
16:42:02 <masta> I read through that patcch, and did not see any glaring problems but since my python is weak I did not apply.
16:42:26 <tyll> it is explained in the mailing list: https://lists.fedoraproject.org/pipermail/devel/2014-February/195426.html
16:42:29 <dgilmore> autoqa is supposed to check upgade paths for things in updates and updates-testing
16:42:44 <nirik> it's to fix some false positives...
16:43:01 <nirik> yeah, typically we use this for branched ?
16:43:27 <dgilmore> the issue is there is many many corner cases here
16:43:34 <dgilmore> a lot of it to do with timimg
16:43:57 <nirik> there was some talk of running this more regularly, but many thought autoqa covered it.
16:43:58 <dgilmore> we dont run the script because of the many issues
16:44:18 <dgilmore> really the only place to run it is f20-updates to rawhide
16:44:31 <dgilmore> perharps f20-updates-testing to rawhide
16:44:53 <dgilmore> autoqa is supposed to cover everything but the to rawhide case
16:45:10 <dgilmore> autoqa is a big part of why we stopped running it
16:45:12 <nirik> right, although there are cases where people do things out of order and forget.
16:45:52 <dgilmore> nirik: we dont typically use this script at all
16:45:58 <masta> and autoqa is being replaced by taskotron, right?
16:46:03 <nirik> masta: right.
16:46:10 <nirik> someday.
16:46:12 <dgilmore> yeah
16:47:05 <dgilmore> so the only place id be comfortable in running this today regulary is f20-updates to rawhide
16:48:19 <nirik> ok. proposal: apply patch, run test run and gather feedback if qa/packagers want us to run it weekly or not?
16:48:36 <masta> dgilmore: does that mean we are unlikely to apply this patch, or the patch needs to be enhanced before acceptance ?
16:49:50 <dgilmore> nirik: in what capacity would we run it?
16:50:31 <nirik> what do you mean? what branches? just f20-updates -> rawhide as you suggested... to nag/remind maintainers about upgrade path
16:50:45 <dgilmore> nirik: okay, no need to apply the patch then
16:50:53 <dgilmore> i dont think it would hurt
16:51:04 <nirik> or we could do f20-updates-testing too... not sure.
16:51:41 <dgilmore> ideally anything in f20-updates-testing would have a higher e:vr in rawhide
16:52:05 <nirik> yep. so, sure, f20-updates-testing -> f20-updates -> rawhide?
16:52:34 <masta> do not disagree with the ideal world stuff... but why not also f19 ?
16:52:41 <dgilmore> afaik we don't need to do anything about the stable release upgrade paths as checking is done as part of the update process
16:53:01 <dgilmore> masta: because its checked elsewhere
16:53:13 <nirik> masta: autoqa should be naging about that (except corner cases)
16:53:18 <masta> ah, ok then...
16:53:21 <nirik> perhaps we could take another tack here...
16:53:29 <nirik> instead of spamming maintainers...
16:53:40 <nirik> weekly just send report to devel / test lists.
16:54:05 <nirik> for all of them. This would help see packages that need fixing by not the normal maintainers...
16:54:44 <masta> hrm... that sounds good
16:55:43 <nirik> so, it would be addative to autoqa.
16:55:54 <dgilmore> so who wants to work on a patch for this?
16:56:06 <masta> so what is the plan? what action items do we need setup, and for who ?
16:56:30 <nirik> it would need tweaking to not nag maintainers and make a nice report.
16:56:41 <dgilmore> someone needs to write a patch to change the output.
16:57:05 <masta> dgilmore: I'll take the ticket and actions under the condition that off the meeting I get help to fulyl understand the problem statement, and "solving-for" statement...
16:58:13 <dgilmore> masta: the problem is simple, the epoch:versrion-release of rawhide packages needs to be higher than that of updates of teh previous release
16:59:20 <masta> #action masta to take ownership of ticket #5854
16:59:45 <nirik> thanks masta
16:59:49 <dgilmore> cheers
17:00:17 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5362 is the next ticket
17:01:33 * nirik doesn't know of any progress on this. we should see if we can find some owners for the action items.
17:02:01 <dgilmore> owners will help
17:02:12 <dgilmore> I guess I should probably own them
17:02:28 <masta> so I looked over the code a bit, and startedd thinking how to implement... but I thought somebody else already had a patch?
17:02:46 <dgilmore> masta: ive never seen a patch
17:03:08 <dgilmore> as upstream of both tools I would hope somoene would send me a patch
17:03:16 * imcleod perks up - createrepo change?  Make xz the default?
17:03:17 <masta> maybe tyll or bochecha spoke about it... but never provided
17:03:27 <masta> not sure who
17:03:35 <masta> so hate to thrown names around even
17:03:42 <nirik> imcleod: yeah, been on the radar for a long time, but keeps never getting done. ;)
17:03:53 <nirik> would make repodata smaller hopefully (less to download)
17:03:54 <dgilmore> #action dgilmore to take ownership of implementing
17:03:56 <tyll> I did not write  patch
17:04:15 <dgilmore> imcleod: createrepo refused to change the default
17:04:20 <imcleod> bah
17:04:28 <dgilmore> so we need to change it in everything using createrepo
17:04:51 <dgilmore> moving on
17:04:52 <masta> also.... the sqlite data uses bzip =(
17:05:14 <masta> that might be part of the "change everything" Dennis just wrote
17:05:24 <dgilmore> #topic s390 status
17:05:30 <dgilmore> sharkcz: hows things?
17:05:59 <dgilmore> is he here?
17:06:05 <sharkcz> I'm
17:06:42 <sharkcz> all works as usual, I've asked perl-Net-SSLeay mainmtainer to look on the problem in rawhide
17:07:05 <dgilmore> okay.
17:07:20 <sharkcz> and fedmsg implemented for koji
17:07:26 <dgilmore> :) cool
17:09:06 <dgilmore> anything else?
17:09:38 <sharkcz> no
17:09:55 <dgilmore> #topic ppc status
17:10:03 <dgilmore> no dwa
17:10:19 <dgilmore> but ppc also setup fedmsg on the hub
17:10:36 <dgilmore> nothing else I know of to bring up
17:10:45 <dgilmore> #topic aarch64 status
17:11:23 <dgilmore> we have daily rawhide aarch64 composes. the broken deps are getting smaller daily as its getting fully brought up
17:11:37 <dgilmore> going to be looking at making images soon
17:11:54 <dgilmore> masta: anything aarch64 you want to mention?
17:12:29 <masta> dgilmore: not really, just that I probably need to spend more time with it
17:12:47 <dgilmore> okay
17:12:56 <dgilmore> #topic open floor
17:13:28 <dgilmore> so imcleod came into #fedora-releng the other day to offer help
17:13:39 <nirik> welcome imcleod. ;)
17:13:40 <dgilmore> and he has looked at createrepo some
17:13:46 <masta> howdy imcleod
17:13:52 <dgilmore> imcleod: want to share your findings?
17:13:57 <tyll> Hey imcleod
17:14:00 <imcleod> Hey all.
17:14:16 <imcleod> So, primary finding is that deltarpm creation is done serially
17:14:32 <imcleod> Even if createrepo is using multiple workers for the primary repo creation tasks
17:14:46 <imcleod> And for the test cases I looked at, the runtime is dominated by the delta creation.
17:14:53 <dgilmore> yep, that was a design decison for a few reasons
17:15:04 <imcleod> I've written a patch to allow this to be done in parallel as well, and it seems to help, but does require a fair amount of memory
17:15:23 <dgilmore> main reason why its done serially
17:15:39 <nirik> so, related to this... why are we keeping so many drpms for rawhide? ;(
17:15:49 <imcleod> dgilmore: Next step I was going to suggest is that I'll enhance the patch to make the number of delta workers a distinct command line and config option, and also allow a limit on the total size of the "in flight" delta rpm creations.
17:15:55 <dgilmore> there is apparently a flag to makedeltarpm to limit ram, at a cost of some delta size efficiency
17:16:11 <imcleod> dgilmore, yeah, that exists as well.  I can fiddle with that to see just how much of a difference it makes.
17:16:24 <imcleod> So I could add that as another configuration option.
17:16:43 <dgilmore> any change we make needs to not break secondary arches or other consumers of createrepo
17:17:06 <nirik> we have currently about 38k drpms per arch in rawhide... so ~100+k or so...
17:17:25 <imcleod> dgilmore: K.  If default behavior is preserved for anyone not using the new command line or config object options, will that be acceptable?
17:17:28 <dgilmore> so while we can throw more ram etc at a problem not everyone can and we cant hardcode for it
17:17:32 <imcleod> Right
17:17:41 <dgilmore> imcleod: thats fine.
17:17:49 <imcleod> But if more ram makes the primary build system 4x faster, I'd argue we should do it ;-)
17:17:56 <dgilmore> imcleod: note we dont ever call createrepo directly
17:18:16 <dgilmore> we would need to patch mash
17:18:19 <nirik> I'm not sure what the s390/ppc composers are like, but I think they have a fair bit of ram available too?
17:18:24 <dgilmore> and make it config options in there
17:18:30 <dgilmore> nirik: they dont
17:18:37 <imcleod> dgilmore: Understood.  I had a brief look at how mash calls it.  I think I can make that work.
17:18:38 <dwa> sorry I suck today, life conspired against me :(
17:18:41 <dgilmore> ppc has some, s390 is the hub box
17:18:46 <masta> we should assume a 4 GiB ceiling
17:19:07 <sharkcz> nirik: s390 hub has 14 GB
17:19:09 <dgilmore> dwa: it happens
17:19:11 <dwa> ppc composer has 8g, we're getting more hardware up imminently so we can make a composer as big as needed
17:19:25 <masta> with a 3/1 vm split
17:19:29 <nirik> sharkcz: wow. ok, we were looking at possibly replacing that, or if not, we can surely add more mem
17:20:02 <dgilmore> nirik: i think that was the max it took
17:20:04 <imcleod> Further note on this.  I don't think delta RPM creation actually needs to live in createrepo.  As best I can tell from the code, the deltas do not in any way change the repo metadata.  (But perhaps I have this wrong.)
17:20:12 <nirik> dgilmore: old box. ;(
17:20:14 <dgilmore> or maybe it was just too expensive ta the time to add more
17:20:20 <dgilmore> nirik: its old
17:20:23 <sharkcz> nirik: that might help, I've already asked for new hw thru our engineering mgmt chain
17:20:46 <nirik> sharkcz: I can't recall if we asked for it too or not. ;)
17:20:47 <dgilmore> imcleod: it needs to, its where it fits into the processes
17:21:24 <dgilmore> imcleod: there is a field added to repomd.xml pointing at the delta metadata
17:21:25 <sharkcz> can't we just disable deltas for secodnary arches? don't think there are (m)any consumers
17:21:42 <imcleod> dgilmore: K.
17:21:48 <dgilmore> sharkcz: maybe
17:22:01 <nirik> sharkcz: +1
17:22:06 <dgilmore> sharkcz: aarch64 i think they will be generally useful
17:22:15 <masta> sharkcz: we always need to make hosting the secondary bits attractive to the mirrors
17:22:19 <dgilmore> but maybe we should just disable for ppc and s390
17:22:27 <sharkcz> dgilmore: +1
17:22:42 <dgilmore> masta: delta is more about users
17:22:49 <nirik> IMHO we should also only keep a short term window for rawhide.
17:22:55 <imcleod> I can have the default mash behavior be to limit the in-flight parallel delta size based on available or total RAM.
17:23:11 <nirik> I mean if you are running rawhide and haven't updated in a week... and given the amount of updates...
17:23:17 <masta> nirik: +1 short term for rawhide
17:23:33 <dgilmore> nirik: delta rpms are only ever against the last build
17:23:51 <nirik> dgilmore: yes, but we are keeping old crap.
17:23:52 <dgilmore> there is only ever one delta unline updates
17:23:56 <nirik> there's 100,000+ drpms
17:24:06 <dgilmore> hrrm
17:24:09 <nirik> there are drpms from nov 2013
17:24:39 <dgilmore> something that can be looked at
17:24:43 <nirik> so, it gets generated when the package updates, but stays around until the next time the package is updated I guess.
17:24:55 <dgilmore> yep
17:25:00 <nirik> I really think something like a week max would be sane
17:25:11 <nirik> if you waited more than that, too bad.
17:25:24 <sharkcz> nirik: aren't some counted twice? there is maybe 70k rpms
17:25:30 <dgilmore> updates repos there can be two delta's per update
17:25:42 <dgilmore> a delta from GA and a delta fromprevious update
17:25:50 <nirik> I was looking at: http://kojipkgs.fedoraproject.org/mash/rawhide-20140217/rawhide/x86_64/os/drpms/
17:25:56 <nirik> if you wget that and grep:
17:26:02 <nirik> grep drpm index.html| wc -l
17:26:03 <nirik> 37966
17:26:16 <nirik> and it's the same for i386/arm too
17:26:24 <dgilmore> rawhide armhfp is 70k rpms, i386 is 72k rpms and x86_64 is 84k rpms
17:26:25 <nirik> or similar anyhow
17:26:53 <dgilmore> very little is actually multilibbed
17:27:42 <nirik> f20-updates x86_64 has 1479814798
17:27:43 <nirik> 14798
17:28:20 <dgilmore> some things we don't make deltas for
17:28:39 <nirik> 21024 in f19-updates x86_64
17:28:55 <nirik> anyhow, I think we need to drastically prune rawhide's
17:29:14 <dgilmore> im okay with it if we can work out a sane way to do so
17:29:50 <nirik> I think mash could just look if it's rawhide and when copying the drpms, only copy ones less than N days
17:30:06 <dgilmore> perhaps
17:30:23 <dgilmore> would need to be an option set in the mash config
17:30:36 <dgilmore> branched is probably the same
17:31:44 <masta> only copy less than N days OR ...
17:31:47 <dgilmore> where we only need to keep a smaller window
17:32:01 <dgilmore> all
17:32:08 <nirik> sure.
17:32:16 <masta> is there any chance an update might be, say.. a month old BUT be the latest delta update?
17:32:42 <nirik> sure, on stable updates we probibly want to keep them around... for people installing fresh and updating.
17:32:49 <dgilmore> masta: if you do a install without updates 6 months after release, then update, you want a delta for the package released a week after the distro went ga if its the only update
17:33:16 <masta> dgilmore: yeah!
17:33:32 <dgilmore> masta: we only keep deltas to the latest update
17:34:05 <masta> dgilmore: ok, I ws under the impression we were keeping all prior updates
17:34:24 <dgilmore> masta: no, RHEL does that, but doesnt make deltas
17:34:38 <dgilmore> fedora updates is only the latest update in the updates repo
17:34:58 <masta> dgilmore: looks like mash is going to be worked on a lot in the coming days/weeks...
17:35:17 <nirik> centos folks were also keeping all deltas for everything, but not long back dropped a bunch because the disk space was too much for mirrors (I think)
17:36:14 <dgilmore> nirik: yeah i can imagine it would be, if we made every update available the mirror space would be massive
17:37:04 <nirik> I'd like to see more logging in mash too...
17:37:20 <nirik> like when it starts deltas and finishes them, and perhaps a unified log with times?
17:38:07 <dgilmore> seems like I have some mash work to do :)
17:38:49 <nirik> so, I bumped releng02 way up on memory, should I move it back down? or does it matter/
17:39:06 <nirik> (releng02 is the rawhide compose machine)
17:39:11 <dgilmore> nirik: probably should drop it down
17:39:40 <dgilmore> I think if we go to do parrallel delta creation we should limit the ram used per delta
17:39:54 <dgilmore> though if its hurting nothing
17:40:00 <nirik> ok, what should I change it to? 8G? 12?
17:40:06 <dgilmore> 16G
17:40:10 <nirik> ok
17:40:58 <nirik> sharkcz: let me know if you dont get a replacement machine, and we could at least give you one of the ones we are replacing... it would have a lot more memory at least.
17:41:16 * nirik checked and we didn't put in for replacing it. I thought we did, but might have missed it.
17:41:51 <dgilmore> nirik: i know is a X series something
17:42:17 <sharkcz> nirik: ok, IIRC the current hub is a loan from IBM
17:42:19 <dgilmore> we could see if it would take more but I doubt it i think we maxed it out when we got it
17:42:24 <nirik> sharkcz: ah, ok.
17:42:32 <dgilmore> sharkcz: i thought denise brought it
17:42:45 <sharkcz> nirik: but I'll check the state
17:42:52 <dgilmore> maybe im wrong and she got them to give it to us to use
17:43:07 <nirik> well, our outgoing virthosts have like 40-50gb mem in them...
17:43:13 <sharkcz> dgilmore: this would be my recollection
17:43:19 <nirik> so, that would be a step up from 14.
17:43:36 <dgilmore> nirik: it would be
17:43:46 <nirik> and 16cpus
17:44:13 <dgilmore> i think we have 4 now?
17:44:29 <sharkcz> 8
17:45:12 <sharkcz> 2x quad core, no hyper threading
17:45:36 <nirik> anyhow, let me know.
17:45:46 <sharkcz> sure
17:46:11 <dgilmore> sharkcz: even having a second box to run releng things from if we didnt move the hub would help
17:46:22 <dgilmore> nirik: ^if its an option
17:46:35 <nirik> sure, possible I guess.
17:46:51 <dgilmore> moving would be better
17:46:57 <dgilmore> anyways
17:47:04 <sharkcz> dgilmore: I don't think the hub is a bottleneck, more the network to the builders and storage
17:47:09 <dgilmore> imcleod: thanks for taking a look
17:47:34 <imcleod> dgilmore: np - will follow up on the config changes and forward another patch
17:47:42 <sharkcz> dgilmore: I'm more worrying about the condition of the hw itself
17:47:52 <dgilmore> sharkcz: true
17:48:55 <dgilmore> anyone have any other open floor item to bring up?
17:50:14 <dgilmore> if not ill wrap up in 30
17:50:57 <dgilmore> #endmeeting