15:33:12 <dgilmore> #startmeeting RELENG (2016-04-11)
15:33:12 <zodbot> Meeting started Mon Apr 11 15:33:12 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:33:12 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:33:12 <zodbot> The meeting name has been set to 'releng_(2016-04-11)'
15:33:12 <dgilmore> #meetingname releng
15:33:12 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
15:33:12 <zodbot> The meeting name has been set to 'releng'
15:33:12 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
15:33:15 <dgilmore> #topic init process
15:33:18 <nirik> morning
15:33:22 * sharkcz is here
15:34:28 * pbrobinson waves
15:34:36 <dgilmore> lets get started
15:34:41 <dgilmore> there is no tickets
15:34:52 <dgilmore> #topic Fedora 24 status
15:35:05 <dgilmore> Just wanted to go over where things are
15:35:26 <dgilmore> I worked with lsedlar friday
15:35:40 <dgilmore> pungi is now making ostree repos as runroot tasks in koji
15:35:46 <pbrobinson> nice!
15:35:57 <dgilmore> currently only enabled for rawhide, need to do f24 todat
15:35:59 <dgilmore> today
15:36:21 <maxamillion> .hello maxamillion
15:36:22 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
15:36:25 <maxamillion> sorry, was getting coffee
15:36:31 <dgilmore> I merged in what should be the last fix to getting the ostree installer dvd working
15:36:44 <dgilmore> and I need to get the atomic cloud images enabled
15:36:46 <nirik> cool
15:37:15 <dgilmore> we need some pungi aand koji work to polish it and make it more functional
15:37:33 <dgilmore> I also want to try make the nightly two week compose using pungi
15:37:57 <dgilmore> it will remove some of teh fragility of the existing way
15:38:04 <pbrobinson> that would be nice
15:38:19 <maxamillion> +1
15:38:23 <dgilmore> #info ostree deliverables for a first pass in pungi nearly complete, polish to come
15:38:47 <dgilmore> #info 32 bit arm docker base image enabled
15:39:10 <dgilmore> I also enabled on Friday building 32 bit arm docker base images
15:39:37 <dgilmore> need to work with maxamillion and poke docker upstream on how to ship it
15:39:57 <pbrobinson> dgilmore can you work with me too, I have that problem to solve on all other arches
15:40:05 <dgilmore> pbrobinson: sure
15:40:27 <dgilmore> hopefullyi the same process for armhfp works for all the other arches also
15:40:31 <maxamillion> dgilmore: +1
15:40:44 <maxamillion> I'd definitely like to add that to the list of stuff we're pushing
15:40:55 <dgilmore> need to also make sure that when we put the x86_64 image in the fedora docker registry we put the arm one there also
15:41:05 <dgilmore> possibly also the secondary arches
15:41:06 <pbrobinson> dgilmore: yep, but I want to know all the details of said process....
15:41:13 <maxamillion> would enable some interesting things like kubernetes/openshift on arm .... (pending we can get some of the weird segfaults sorted)
15:41:34 <dgilmore> pbrobinson: sure. right now we have no clue, maxamillion asked a few weeks back and got no answer
15:41:34 <pbrobinson> maxamillion: weird segfaults on arm or in general?
15:41:41 <maxamillion> pbrobinson: on arm, for openshift
15:41:50 <pbrobinson> maxamillion: cool
15:41:57 <dgilmore> pbrobinson: i build openshift, it died because it could not figure out the cpu speed
15:42:14 <dgilmore> built even
15:42:33 <maxamillion> jim perrin from centos did as well on some arm hardware, it also blew up in spectacular fashion but differently than what dgilmore was seeing
15:42:40 <pbrobinson> I tried building kubernetes on aarch64 the other day and it died in some weird way so I moved on to other golang horrors instead
15:42:48 <maxamillion> hopefully we can get all of it upstream and sort that stuff out
15:43:24 <dgilmore> #info intresting problems to sort around docker registries and non wx86_64 support
15:44:02 <dgilmore> I think the one big remaining problem for f24 is s390
15:44:16 <dgilmore> but I doubt we can solve that here in 5 minutes
15:45:44 <dgilmore> #info we also need to sort out building and signing windows binaries for live usb creator
15:45:45 <pbrobinson> maxamillion: I've fixed up a bunch of dependencies on some of the lower packages, I've got a feeling I know what some of the issues might be (sadly, yes!)
15:46:30 <maxamillion> pbrobinson: huzzah!
15:47:31 <dgilmore> does anyone have any f24 thinsg they want to bring up?
15:48:17 <pbrobinson> we're just over a week out of freeze right?
15:48:42 <nirik> The storage upgrade that was going to be at alpha release time that got postponed, is now likely to be next tuesday (beta freeze start)
15:48:44 <dgilmore> https://fedoraproject.org/wiki/Releases/24/Schedule says April 19th
15:48:59 <dgilmore> so yeah, next Tuesday
15:49:24 <dgilmore> nirik: that should be okay
15:49:44 <nirik> I am not sure it's set yet, they said it was likely then... we will see.
15:49:53 <dgilmore> okay
15:50:11 <dgilmore> maxamillion: so layered images
15:50:17 <pbrobinson> nirik: that should be non distruptive or total outage?
15:50:22 <maxamillion> dgilmore: yo
15:50:24 <dgilmore> maxamillion: will it be in prod by then?
15:50:39 <maxamillion> dgilmore: I sure hope so
15:51:11 <maxamillion> dgilmore: if I can figure out what's broken with my docker registry auth behind the proxy, I think I could get it all done in stage properly, but I'm still a bit at a loss as to what's going on here
15:51:26 <nirik> pbrobinson: well, they need to throttle us, so that might make things slower, we are looking at locking master mirror rsync to tier1 mirrors for that time... but the actual event is supposed to be non disruptive. (no outage)
15:51:28 <dgilmore> maxamillion: okay, I will try look this arvi
15:51:30 <dgilmore> arvo
15:51:36 <maxamillion> dgilmore: arvo?
15:51:43 <dgilmore> afternoon
15:51:57 <pbrobinson> nirik: ACK, I figured as much, thanks
15:52:10 <maxamillion> dgilmore: ah ok, thanks
15:52:26 <maxamillion> dgilmore: yeah, I have a meeting right after this one so I likely won't get back around to it until after lunch
15:52:43 <dgilmore> maxamillion: okay, lets try look at it then
15:52:47 <maxamillion> dgilmore: sounds good
15:53:16 <dgilmore> #topic Secondary Architectures updates
15:53:19 <dgilmore> #topic Secondary Architectures update - ppc
15:53:25 <dgilmore> lets jump into ppc
15:53:30 <pbrobinson> statistics: {'older': 158, 'local_only': 13, 'remote_only': 366, 'same': 17552, 'newer': 3, 'total_missing_builds': 357}
15:53:30 <dgilmore> pbrobinson: how is it?
15:53:38 <dgilmore> pretty close
15:53:38 <pbrobinson> we're looking pretty good
15:54:04 <pbrobinson> builds mostly there, blocking on a a couple of core packages, both now filed as FE to get some actual traction on them
15:54:20 <pbrobinson> the two I filed affect all secondary platforms
15:54:28 <dgilmore> :) okay
15:54:48 <dgilmore> not good that we have bugs only on all secondary arches
15:54:59 <dgilmore> likely means someone did something silly
15:55:01 <pbrobinson> I suspect it's all gcc6 fall out
15:55:29 <dgilmore> okay
15:55:32 <sharkcz> nope, their code knows only x86 and arm, eg. 386-ds-base
15:55:36 <pbrobinson> amuses me that v6 was suppose to be a minor bump and fully compatible with 5 as it's been one of the worst bumps for a while
15:55:48 <pbrobinson> sharkcz: yes, on that one I agree
15:55:49 <dgilmore> always fun
15:56:00 <pbrobinson> but they've often been x86ist
15:56:29 <dgilmore> probably as it is due to visibility
15:56:32 <pbrobinson> we've got some golang bits to close out (and a few for aarch64 too) around container stuff but other than that I think ppc is pretty feature complete
15:56:47 <dgilmore> we never really did a great job of letting people know when things fail on secondaries
15:56:48 <pbrobinson> and not looking too bad
15:57:08 <dgilmore> #info ppc is looking good right now
15:57:10 <sharkcz> ack
15:57:11 <dgilmore> #topic Secondary Architectures update - s390
15:57:11 <pbrobinson> dgilmore: one of the many reasons I'd like the koji convergence :)
15:57:19 <dgilmore> sharkcz: so how goes s390?
15:57:39 <dgilmore> we do need to figure out how to make the runroot tasks work somehow
15:57:39 <pbrobinson> we're ready to go for pungi stuff, blocking on the storage team poking the netapp over exports
15:57:45 <sharkcz> after pbrobinson installed the f23 based x86 builder for newRepos we are getting closer again
15:58:07 <pbrobinson> dgilmore: once I can get pungi working (netapp export issue) I'll be looking closer at the runroot side of things
15:58:29 <sharkcz> mean I'll close 1300669 as an infra issue
15:58:29 <dgilmore> pbrobinson: okay
15:58:48 <sharkcz> pbrobinson: I have one builder ready to be used as runroot one
15:59:12 <pbrobinson> sharkcz: OK, I'll sync with you offline on that
15:59:24 <sharkcz> pbrobinson: ok
15:59:29 <dgilmore> sharkcz: on that we have set install_weak_deps=0 everywhere
15:59:47 <nirik> pbrobinson: whats the status on the ppc8 rebuilds?
15:59:47 <dgilmore> something you were kinda getting for free
16:00:24 <sharkcz> dgilmore: yep, but we needed to get over the older builder that required weak deps installed
16:00:44 <pbrobinson> nirik: getting there, they dropped back a bit while I dealt with other more pressing things
16:01:11 <nirik> ok. right now ansible is a bit confused. ;) Since all the ones in inventory are .ppc.fedoraproject.org and it can't reach them at that.
16:01:25 <nirik> so they don't appear and their guests don't appear as reachable
16:02:07 <pbrobinson> nirik: I'll try and get to that again shortly
16:02:18 <nirik> np, just wanted to mention it.
16:02:24 <nirik> let me know if I can assist any
16:02:31 <pbrobinson> nirik: yep, am aware
16:06:08 * nirik has a few small items for open floor.
16:06:22 <dgilmore> #topic Secondary Architectures update - arm
16:06:28 <dgilmore> pbrobinson: how is arm
16:06:41 <pbrobinson> statistics: {'older': 62, 'local_only': 1, 'remote_only': 315, 'same': 17702, 'newer': 0, 'total_missing_builds': 134}
16:06:52 <dgilmore> aarch64 is closer than ppc
16:06:59 <pbrobinson> stats wise looking good, same packages issues as ppc
16:07:22 <dgilmore> #info same issues as ppc but slightly closer build wise
16:07:22 <pbrobinson> spent some time over the weekend fixing up other stuff
16:07:48 <pbrobinson> should hopefully have cloud/docker images enables for tomorrow's nightlies
16:08:07 <dgilmore> #info docker and cloud images to be enabled soon
16:08:39 <pbrobinson> and also "disk images" hacked in too as we need to use lmc which doesn't have koji integration yet (appliance builder doesn't work there)
16:08:55 <pbrobinson> this week for all of them one way or the other!
16:09:22 <pbrobinson> nothing much else to report there
16:09:41 <dgilmore> pbrobinson: cloud images are just a different type of disk image, maybe they can just be done with imagefactory
16:09:53 <pbrobinson> dgilmore: possibly
16:09:58 <dgilmore> just using a different kickstart
16:10:05 <pbrobinson> I did think about that as an option
16:10:14 <dgilmore> just enable raw-xz only
16:10:31 <dgilmore> may be simpler
16:10:53 * linuxmodder pops in to  brush up on arm
16:11:03 <dgilmore> anything else?
16:11:20 <pbrobinson> not from me
16:11:31 <dgilmore> #topic Open Floor
16:11:39 <dgilmore> nirik: you have some things?
16:11:48 <nirik> yeah, just a few quick ones. ;)
16:12:10 <nirik> 1. we have some outages this week... tomorrow for build stuff, wed for the rest... updates/reboot cycle
16:12:20 <nirik> I was thinking I'd update the builders later this week/weekend...
16:12:34 <dgilmore> okay
16:12:50 <dgilmore> nirik: there is a manual schema update needed on koji
16:12:50 <pbrobinson> nirik: ignore ppc/arm and I'll do them, the former will be done as part of the rebuilds anyway
16:12:56 <nirik> and per our discussion the other day, I am going to add some more buildvm's to stg with nested virt enabled. ;)
16:12:58 <pbrobinson> builders that is
16:13:18 <dgilmore> nirik: primary koji has had it, but teh other koji's have not
16:13:19 <nirik> dgilmore: ok, you want to do that part? or I can if you tell me what.
16:13:26 <nirik> pbrobinson: ok.
16:13:31 <nirik> or I guess pbrobinson can do it. ;)
16:13:36 <dgilmore> nirik: either way, just need to be aware of it
16:14:05 <dgilmore> it is not in a schema file
16:14:27 <pbrobinson> dgilmore: so where is it then?
16:14:41 <pbrobinson> a pull req somewhere?
16:14:44 <nirik> pbrobinson: you want me to do hubs tomorrow? or you want to do them when doing builders later? or ?
16:14:51 <dgilmore> alter table repo add signed BOOLEAN DEFAULT 'false';
16:15:04 <dgilmore> pbrobinson: it is part of jays signed repo changes
16:15:16 <pbrobinson> dgilmore: I'd figured a much
16:15:38 <dgilmore> it is in the schema file, but there is no update schema file
16:15:53 <pbrobinson> nirik: I'll upgrade the koji + db changes on the hubs, can you do the patch/reboots on them?
16:16:16 <nirik> sure. Sounds good.
16:16:26 <nirik> outage tomorrow starts at 21UTC...
16:17:04 <pbrobinson> nirik: I'll do the secondary hubs tonight or tomorrow morning so I'm out of your way :)
16:17:13 <nirik> Oh, last thing... jberkus is looking at our koji db issues. I gathered a bunch of info for him and he's going to look and get back to us later today.
16:17:34 <nirik> hopefully we will find out why primary is at least getting so hammered
16:17:58 * nirik doesn't know if secondary has seen slowdowns
16:18:27 <dgilmore> nirik: awesome, thanks for getting the data to him
16:19:14 <pbrobinson> nirik: they're mostly OK, did you apply the DB optimise bits to them?
16:19:22 <nirik> yep.
16:19:26 <pbrobinson> they likely get hit a lot less than primary
16:19:32 <nirik> yeah.
16:19:36 <sharkcz> nirik: seems s390 hub is running fine, which slowdowns primary sees?
16:19:51 <nirik> sharkcz: the db gets heavily loaded and things like kojira can't work
16:20:11 <nirik> https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=db-koji01.phx2.fedoraproject.org&plugin=load&timespan=604800&action=show_selection&ok_button=OK
16:20:34 <pbrobinson> nirik: is the primary DB on shared storage or local?
16:21:01 <nirik> it's on iscsi...
16:21:40 <nirik> but it was fine until about january...
16:21:43 <nirik> https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=db-koji01.phx2.fedoraproject.org&plugin=load&timespan=31622400&action=show_selection&ok_button=OK
16:22:52 <pbrobinson> nirik: yes, but you've got a lot more up in the air, 1gb links, switch ports, network queues etc etc
16:22:54 <nirik> anyhow, we should know more later today.
16:23:06 <dgilmore> nirik: I suspect it was when we updated to support content generators
16:23:06 <nirik> it's actually 10G. ;)
16:23:41 <dgilmore> and the db schema is not optimised
16:23:47 <dgilmore> or the queries are horrible
16:23:57 <nirik> yeah.
16:24:25 <nirik> I know there's some gigantic queries... might be people doing dumb things with the web interface, or crawling things
16:24:36 <dgilmore> and that
16:25:54 * nirik had nothing else.
16:26:09 <jkurik> Hi, I have a question
16:26:37 <jkurik> a rumour has reached me that releng wants to drop alpha releases starting from F26 - any info about this plans ? Or is it just a false info ?
16:29:13 <dgilmore> jkurik: we have said it a lot we plan to do it
16:29:24 <dgilmore> jkurik: we have talked to adamw about it
16:29:44 * nirik has heard about it, but I haven't seen any formal proposal
16:29:55 <dgilmore> jkurik: until we are sure that the nightly testing is sufficent to keep things always at an alpha level there will be no changes
16:30:02 <nirik> we would need to figure out what happens to the milestones around that
16:30:07 <dgilmore> so far it is something we would like to do
16:30:33 <dgilmore> but we can not propose it until we are sure that we can deliver at a high quality without it
16:30:54 <dgilmore> jkurik: so for right now it is just an idea
16:30:59 <dgilmore> not set in stone
16:31:04 <jkurik> dgilmore: ok, thanks for the explanation; please keep in in the loop if there are any changes regarding milestones
16:31:18 <nirik> It does mean less press...
16:31:26 <nirik> but not sure how big a deal that is
16:31:41 <dgilmore> jkurik: as soon as we have something concrete we will be sure to let you know
16:31:44 * adamw is all for it.
16:32:08 <jkurik> great, that is all I had
16:32:10 <dgilmore> it boils down to can we do it and not comprimise the release quality
16:33:17 <dgilmore> one quick thing from herr
16:33:18 <dgilmore> here
16:33:40 <dgilmore> https://careers-redhat.icims.com/jobs/51819/software-engineer/job
16:33:42 <dgilmore> https://careers-redhat.icims.com/jobs/51274/software-engineer/job
16:34:04 <dgilmore> two jobs in Release Engineering at red hat
16:34:25 <dgilmore> the first is in Westford, the second is in India
16:34:32 <nirik> cool. ;)
16:34:34 <maxamillion> +1
16:34:40 <dgilmore> they will have osme level of interaction in fedora
16:34:48 <dgilmore> the first will have quite a bit
16:35:02 <dgilmore> there is alos another one coming that will be based in Brno
16:35:40 <dgilmore> that is all I have
16:35:49 <jkurik> looks like releng wants to implement the "follow the sun" availability
16:35:51 <dgilmore> if nothing else I will wrap up
16:38:46 <dgilmore> #endmeeting