15:31:04 <dgilmore> #startmeeting RELENG (2015-04-27)
15:31:04 <zodbot> Meeting started Mon Apr 27 15:31:04 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:31:04 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:31:15 <dgilmore> #meetingname releng
15:31:15 <zodbot> The meeting name has been set to 'releng'
15:31:15 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
15:31:15 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
15:31:18 <dgilmore> #topic init process
15:31:20 <dgilmore> who all is here
15:31:24 * maxamillion is here
15:31:25 * pbrobinson waves
15:31:40 <nirik> I'm partly here only. dealing with fires. Ping me if I am specifically needed.
15:33:56 <dgilmore> lets get moving
15:34:12 * threebean is here
15:34:12 <maxamillion> +1
15:34:14 <dgilmore> #topic #6163 Request for review of mash patch
15:34:20 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6163
15:34:31 <dgilmore> so I do not want to spend time in the meeting reviewing patched
15:34:34 <dgilmore> spatches
15:34:49 <threebean> sure.  can we commit someone to review it and resolve it out of band then?
15:34:58 <dgilmore> I will remove the meeting keyword and it can be discussed on list or in #fedora-releng
15:35:18 <dgilmore> #topic #6158 Request to discuss Rel-Eng Project Planning Proposal
15:35:22 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6158
15:35:36 <dgilmore> I was way to busy last week to look at this
15:36:01 <dgilmore> I would rather at this point we follow up on the list and discuss next week if appropriate
15:36:41 <pbrobinson> +1
15:36:51 <maxamillion> +1
15:36:54 <dgilmore> Does anyone else have any other opinions?
15:37:21 <pbrobinson> I had a week from hell last week and just never got the time to properly read the threads and followup RFC
15:37:31 <maxamillion> I'd definitely prefer that be discussed at length in the mailing list so everyone has time to think it over before chewing time in the irc meeting
15:37:43 <dgilmore> #topic #6159 Request to discuss Git workflow for rel-eng tooling
15:37:45 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6159
15:37:51 <dgilmore> this one is kinda the same
15:37:55 <maxamillion> +1
15:38:10 <dgilmore> however I oretty strongly think we should move the git repos to pagure
15:38:17 <dgilmore> pretty strongly
15:38:17 <maxamillion> neither are urgent, just things I was thinking about ... I'll nag appropriately if I feel like it starts to fall through the cracks ;)
15:38:21 <maxamillion> dgilmore: +1
15:38:34 <maxamillion> I've been pretty happy with pagure so far, it's really good
15:38:39 <dgilmore> and we do need to setup a process for review etc
15:39:38 <maxamillion> +1
15:40:28 <dgilmore> #topic #5886 need method for distributing urgent fixes... urgently
15:40:34 <dgilmore> https://fedorahosted.org/rel-eng/ticket/5886
15:40:43 <dgilmore> nirik: did QA provide any feedback?
15:41:30 <nirik> no
15:41:41 * masta is here now
15:41:47 <masta> howdy folks - sry late
15:41:58 <tyll_> Hi
15:41:59 <pbrobinson> masta: you always are... ;-)
15:43:13 <dgilmore> nirik: fun, I will try get adamw to provide some feedback
15:43:26 <adamw> what'd i do
15:43:29 <adamw> i thought we did?
15:43:35 <adamw> well, maybe only casually.
15:43:58 <adamw> i mean, we could deal with the testing, it still seems like an awful lot of complication to me, but mostly on your side not ours.
15:44:38 <dgilmore> adamw: it kinda is a lot of complication. but we do need some way to get updates out faster
15:44:55 <adamw> is it not plausible to just work on making the current process more efficient?
15:45:12 <dgilmore> adamw: no.
15:45:25 <dgilmore> not unless we turn off delta rpms
15:45:35 <tyll> IMHO it would be good to get something for urgend updates now and then improve it once it is running
15:45:54 <maxamillion> it was discussed to just turn off deltarpms for the urgent updates, yes? was that rejected as an option?
15:46:00 <dgilmore> tyll: there is a lot of things that need to be in place for it to be effective
15:46:08 <adamw> tyll: the thing is, once you build a big chunk of infra, it tends to stick around.
15:46:24 <masta> adamw: the problem here as I was told, is that when a critical CVE comes down while a huge push is in progress, well we cannot do any work until that push has completed.
15:46:25 <dgilmore> maxamillion: it turns them off for everything and will remove all existing delta rpms
15:46:38 <maxamillion> dgilmore: oh
15:46:45 <dgilmore> which would result in there being no deltas for some period of time and updates
15:46:49 <masta> so it would be well to have a way to push security fix on the side quickly.
15:47:10 <adamw> masta: i understand the theory, it just seems like a big amount of work which ultimately saves us, like, a day.
15:47:12 <dgilmore> masta: thats not the problem
15:47:19 <adamw> when we still have CVEs from 2014. soo, y'know.
15:47:22 <dgilmore> masta: it is a piece of the problem
15:47:39 <maxamillion> dgilmore: gotchya, and what does that? I'd like to dig through the code to see if I could add an option to trim that down to a selected list of pkgs
15:47:43 <nirik> I think we can make the current process better... but not sure how much.
15:47:57 <nirik> bodhi2 would help a lot if it ever actually happens.
15:48:31 <pbrobinson> I was wondering what the state of bodhi2 was
15:48:53 <pbrobinson> and if it doesn't happen where do we go from there?
15:49:25 <dgilmore> nirik: will it really help a lot though?
15:49:31 <masta> I would imagine the current process could improve, but not to the point of being significantly faster. So lets just move forward with both improving the current stuff, and having a rapid process for important security things.
15:49:52 * dgilmore is really doubtful that bodhi2 will bring benefits
15:50:28 <dgilmore> but I welcome being proved wrong
15:50:28 <nirik> dgilmore: well, there's a lot more parallelism in it
15:50:40 <nirik> I think for example a f22/21/20 push would fire off 6 mashes...
15:50:45 <nirik> instead of 1 1 1 1 1 1
15:50:53 <dgilmore> nirik: where exactly is it? I have not heard a status update since the fad
15:51:11 <nirik> beats me. We should invite lmacken next week to tell us.
15:51:21 <pbrobinson> well having bodhi2 would be a good start as it allows us to evolve the updates platform because at the moment all the bugs are "It's known, and it'll be fixed in bodhi2". We've been stagnant for years
15:51:29 <dgilmore> nirik: some communication would be nice
15:51:31 <nirik> sure.
15:51:39 <nirik> last I heard we were doing packaging
15:51:45 <nirik> not sure if thats done yet or not.
15:51:53 <nirik> then we were going to do a staging instance.
15:52:11 <masta> parallel mashes would be neat.
15:52:30 <pbrobinson> last I heard it was due to go GA post F-21 GA but we're nearly to F-22 part of the process and I've not seen any movement this year
15:53:11 <nirik> how about we invite lmacken next week and go over it? or I can try and get more details.
15:53:32 <pbrobinson> +1
15:53:42 <masta> Would rather not wait for the rapid security work flow to be a bodhi2 thing, would be nice to hack into existing bodhi now.
15:54:07 <pbrobinson> masta: no point doing lots of dev if bodhi2 can deal with it and is around the corner
15:54:56 <masta> pbrobinson: yeah. I guess we need to establish the facts of bohdi2
15:54:56 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6164 filed asking lmacken to attend next week and give us a status update
15:55:22 <dgilmore> lets move on
15:55:32 <dgilmore> #topic Secondary Architectures updates
15:55:33 <dgilmore> #topic Secondary Architectures update - ppc
15:55:38 <dgilmore> pbrobinson: how is ppc?
15:56:30 <pbrobinson> we have a bug with xz for producing the product images, just doing some testing but we'll need to do a RC3
15:56:55 <pbrobinson> looks like the easy fix is going to be to disable 40 cores from the compose box :-D
15:57:14 <pbrobinson> other than that the Beta is pretty much good to go
15:57:48 <dgilmore> pbrobinson: perhaps the one that we workout on primary with "XZ_DEFAULTS=--memlimit-compress=3700MiB"
15:57:53 <pbrobinson> basically it screws up the creation of the product-image stuff
15:57:59 <dgilmore> https://git.fedorahosted.org/cgit/releng/tree/scripts/run-pungi#n115 as an example
15:58:19 <pbrobinson> it's to do with the thread count
15:58:22 <masta> pbrobinson: what is the synopsis of that bug? seems to be implied it cannot do threaded?
15:58:33 <dgilmore> pbrobinson: the memlimit causes xz to use less cores
15:58:53 <masta> pbrobinson: maybe do an 'xz --threads=1 ...'
15:59:04 <pbrobinson> masta: it can do threads, it's just the number, and it might even be the amount of data to be processed by the threads
15:59:05 <dgilmore> pbrobinson: i suspect that adding a XZ_DEFAULTS=--memlimit-compress=3700MiB in front of pungi will fix it
15:59:26 <pbrobinson> I can work around it for the compose and then there will be a bug filed with the xz maintainers
15:59:34 <dgilmore> pbrobinson: we had to do that because it would try to use more than 4GiB ram on 32 bit
15:59:43 <dgilmore> pbrobinson: there is alreadya  bug
15:59:55 <pbrobinson> dgilmore: it's all 64 bit here, no 32 bit
16:00:03 <dgilmore> pbrobinson: same issue
16:00:19 <dgilmore> pbrobinson: xz just assumes it can alloc as much memory as it wants
16:00:32 <masta> dgilmore: hehe I was also going to suggest the XZ_OPTS= infront of whatever does the work =)
16:01:02 <dgilmore> pbrobinson: https://bugzilla.redhat.com/show_bug.cgi?id=1196786
16:01:21 <dgilmore> https://bugzilla.redhat.com/show_bug.cgi?id=1196481
16:01:29 <dgilmore> pbrobinson: the two bugs from F21
16:01:30 <pbrobinson> well given the data it's compressing is in the tens of Kb I'm unsure the memory usage is the issue but I have work around already and I don't need to be told how to suck eggs!
16:01:44 <dgilmore> pbrobinson: xz does some insane stuff
16:02:01 <pbrobinson> and it's a side problem to the compose, thanks for the BZ, I'll review but I have the issue in hand
16:02:03 <masta> pbrobinson: lol
16:02:09 <dgilmore> one would thing taht compressing a few files would not try and allocate more than 4GiB on 32 bit x86
16:02:53 <dgilmore> pbrobinson: the first bug i linked has the details of adamw talking with the xz guys
16:03:15 <dgilmore> I suspect that due to 40 cores its trying to allocate more memory than is available
16:03:19 <pbrobinson> dgilmore: this "xz -T8 -9 needs amost 10 GiB of memory." is amusing
16:03:31 <pbrobinson> 48 cores actually ;-)
16:03:38 <dgilmore> pbrobinson: indeed, as i said xz is not really sane
16:04:25 <dgilmore> pbrobinson: so its probably trying to allocate something like 60GiB of ram
16:04:32 <dgilmore> even though the contents is tiny
16:04:56 <masta> wow - that's nuts
16:04:57 <pbrobinson> I can get it to work with the same commands if I use it to tar up say spins-kickstarts, I think there might even be some data starvation in my use case
16:05:00 <dgilmore> pbrobinson: with the export on primary xz prints a message it is limiting to two cores due to memory
16:05:20 <dgilmore> okay
16:05:24 <pbrobinson> but anyway, who cares, enough time spent on xz issues! This is after all a rel-eng meeting ;-)
16:05:35 <dgilmore> pbrobinson: well I guess it is worth a test to see if it works okay
16:05:47 <dgilmore> https://bugzilla.redhat.com/show_bug.cgi?id=1196481
16:05:48 <pbrobinson> dgilmore: I'll play some more later
16:05:51 <dgilmore> gahh
16:05:57 <dgilmore> #topic Secondary Architectures update - s390
16:06:05 <dgilmore> no sharckcz
16:06:14 <pbrobinson> he was about earlier
16:06:18 <dgilmore> pbrobinson: any idea on how s390 went last week?
16:06:26 <pbrobinson> it's OK
16:06:29 <dgilmore> okay
16:06:32 <dgilmore> #topic Secondary Architectures update - arm
16:06:39 <dgilmore> how is arm?
16:06:44 <pbrobinson> he has some issues where the lorax team didn't add product support for s390
16:07:14 <dgilmore> pbrobinson: sounds fun
16:07:15 <pbrobinson> arm is trundling along OK, the beta should be signed off RSN and sent to mirrors
16:07:22 <dgilmore> awesome
16:07:29 <masta> cool
16:07:49 <pbrobinson> I fixed up a bunch of build issues on Friday too and most of them have had enough karma make it to stable
16:07:52 <pbrobinson> just one
16:07:56 <dgilmore> #info Beta for arm should be out soon
16:08:06 <pbrobinson> the xorg drivers package thingy
16:08:41 <dgilmore> that was droping the qxl driver from being pulled in until it is fixed right?
16:08:51 * pbrobinson awaits bodhi
16:08:57 <pbrobinson> tup
16:08:59 <pbrobinson> yup
16:09:13 <pbrobinson> xorg-x11-drivers-7.7-13.fc22
16:09:16 <nirik> pbrobinson: is there a time this week I could redo secondary sigul? that wouldn't hit you and sharkcz?
16:09:40 <pbrobinson> nirik: what window will you need?
16:09:56 <nirik> should just be a few hours.
16:10:02 <nirik> or less
16:10:10 <pbrobinson> nirik: can you do it today?
16:10:26 <nirik> yeah, I might be able to later today... as long as the fires stay low. ;)
16:10:27 <pbrobinson> or first up tomorrow?
16:11:15 <nirik> ok, how about I try for today, failing that first thing tomorrow.
16:11:19 <nirik> when today would be ok? anytime?
16:11:20 <pbrobinson> nirik: I think the first half of this week should mostly be OK, just ping me before you do it
16:11:35 <nirik> ok, can do
16:11:36 <maxamillion> nirik: we still sync'ing up on the staging koji today or should be punt that to tomorrow?
16:11:52 <pbrobinson> nirik: yep, today and anytime tomorrow should be OK, just ping me before you pull the trigger
16:12:08 <nirik> maxamillion: well, we can see. ;) Having fires this morning... but perhaps
16:12:28 <nirik> maxamillion: would like to get this sigul thing done, but after that could look at koji01.stg?
16:12:47 <nirik> or perhaps dgilmore would have time to poke at koji01.stg with you...
16:12:51 <maxamillion> nirik: no worries, just let me know ... I'm running through https://fedoraproject.org/wiki/Koji/ServerHowTo right now for personal practice since I'm not very seasoned in the ways of koji administration
16:14:47 <dgilmore> maxamillion: I can probably spend some time with you
16:14:59 <dgilmore> okay
16:15:04 <dgilmore> lets move on
16:15:15 <dgilmore> #topic Open Floor
16:15:24 <dgilmore> does anyone have anything for open floor?
16:15:57 <dgilmore> I will take taht as a no
16:16:01 <dgilmore> thanks all
16:16:05 <dgilmore> #endmeeting