14:32:22 <dgilmore> #startmeeting RELENG (2016-09-12)
14:32:23 <zodbot> Meeting started Mon Sep 12 14:32:22 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:32:23 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:32:23 <zodbot> The meeting name has been set to 'releng_(2016-09-12)'
14:32:23 <dgilmore> #meetingname releng
14:32:23 <zodbot> The meeting name has been set to 'releng'
14:32:23 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion mboddu
14:32:23 <zodbot> Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll
14:32:26 <dgilmore> #topic init process
14:32:36 * pbrobinson o/
14:32:38 * sharkcz is here
14:33:28 <nirik> morning
14:34:03 <dgilmore> lets wait for a few
14:35:25 <bowlofeggs> .hello bowlofeggs
14:35:26 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
14:36:16 <dgilmore> lets get started
14:36:29 <dgilmore> #topic aarch64 migration
14:36:47 <dgilmore> So I think it went fairly smoothly
14:36:50 <pbrobinson> so the main packages have been merged and we're building
14:37:03 <pbrobinson> I'm checking things over a it looks reasonable
14:37:21 <dgilmore> eclipse is in a weird state due to a quick in how it is built
14:37:42 <pbrobinson> yes, but the maintainer is investigating
14:37:48 <nirik> cool. ;) congrats.
14:37:54 <dgilmore> https://bugzilla.redhat.com/show_bug.cgi?id=1374938
14:37:59 <nirik> we still need to adjust rawhide compose right?
14:38:02 <pbrobinson> I'm just finishing up bits around compose stuff and should send a PR for pungi-fedora
14:38:09 <dgilmore> we do need to adjust rawhide
14:38:23 <pbrobinson> dgilmore: nirik: I should have a PR later today
14:38:34 <nirik> cool. :)
14:38:48 <dgilmore> #info pbrobinson to send in a pull request today to make changes to the compose process
14:39:49 <dgilmore> we missed adding aarch64 builders to the eclipse channel
14:40:32 <dgilmore> it was much less work than when we brought armhfp online
14:40:39 <pbrobinson> yes, I fixed that this morning, with luck that channel will go away shortly
14:41:00 <dgilmore> pbrobinson: I believe that the plan is to bring power online in 4-6 weeks?
14:41:13 <dgilmore> or maybe a tad longer, but well before mass rebuild?
14:41:15 <pbrobinson> correct
14:41:42 <pbrobinson> I would like to see it merged by late Oct (ie before the focus shifts from F-25 -> 26)
14:41:45 * limburgher lurking
14:42:25 <nirik> sounds good
14:42:39 <nirik> how far are we from armv7 builder vms?
14:42:39 <dgilmore> awesome
14:43:02 <dgilmore> hi linuxmodder
14:43:05 <dgilmore> hi limburgher
14:43:09 <pbrobinson> we've got a few issues to fix up there, but they're actually issues are seen on x86 too
14:43:11 <dgilmore> sorry linuxmodder
14:43:36 <pbrobinson> some of them have gone with the systemd limits change for kojid
14:43:51 <dgilmore> nirik: did you send in a PR to koji?
14:43:57 <pbrobinson> but nothing major really
14:44:13 <nirik> dgilmore: for what?
14:44:14 <dgilmore> nirik: or would you rather I do it?
14:44:23 <dgilmore> nirik: for changing the systemd limits
14:44:39 <nirik> oh, right. No, I haven't yet, if you would like to that would be great
14:44:51 <nirik> I think setting it unlimited is probibly the best bet
14:44:56 <dgilmore> sure
14:45:20 <nirik> TasksMax=infinity
14:45:25 <dgilmore> does anyone have any questions about the changes that have been made?
14:45:59 * nirik is looking forward to replacing the calxeda builders. ;)
14:46:19 <dgilmore> :)
14:46:26 <dgilmore> the io on them is slow
14:46:29 <pbrobinson> nirik: I think we're almost there :)
14:46:55 <dgilmore> #topic #6467 Consider splitting the ancient portions of the archive
14:46:59 <nirik> yeah, it will be interesting to see what ends up being slowest... probibly still them, but much faster than we have now.
14:47:10 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6467
14:47:36 <dgilmore> nirik: at least for some lings, like java where the optimisation is not good
14:47:59 <pbrobinson> yes, without doubt, I'd still bet on Power8 being the fastest ;-)
14:48:15 <dgilmore> so spliting up  the archive module
14:48:52 * nirik doesn't recall where we were here. going to gather info?
14:48:57 <dgilmore> http://download.opensuse.org/distribution/
14:49:01 <dgilmore> yeah
14:49:16 <dgilmore> so suse does not seem to have all of their old releases online
14:50:13 <pbrobinson> do we need to mirror them? could we just put them on a dedicated non mirrored thing in the corner? Any idea what the traffic for them is?
14:50:31 <nirik> well, recently eoled versions get a lot of traffic.
14:51:10 <dgilmore> nirik: maybe we keep 2 eol versions in /pub/fedora etc
14:51:12 <pbrobinson> so maybe recent-history and ancient-history are needed :)
14:51:27 <dgilmore> and make archive no rsyncable
14:51:39 <dgilmore> not rsyncable
14:51:54 <nirik> I was thinking we just change when we remove eol releases from /pub/fedora... just make that longer until traffic dies out?
14:52:22 <nirik> why not? if people want to mirror archive, why not let them?
14:52:22 <pbrobinson> rotate them out a cycle later and not advertise them on mirror manager which actively then breaks the repos and they need to manually change the baseurl
14:52:44 <nirik> we have not done that ever, would be a pretty big change
14:53:06 <dgilmore> nirik: yeah. just keep the content in /pub/fedora longer
14:53:11 <dgilmore> it would make it bigger
14:53:48 <dgilmore> there is 4 mirrors for fedora 20 x86_64
14:53:59 <dgilmore> 3 for fedora 21
14:54:22 <dgilmore> fedora 22 has not moved to archive yet
14:54:27 * nirik goes to get more coffee
14:55:43 <dgilmore> sorry did not have county=global
14:56:34 <dgilmore> there is 7 mirrors globally for f21 x86_64
14:57:26 <dgilmore> f22 has not yet been archived
14:57:36 <dgilmore> and soon f23 will be eol
14:58:05 <dgilmore> #proposal we move f22 to archive when f26 is released
14:58:27 <pbrobinson> +1
14:58:56 <nirik> sounds reasonable
14:59:15 <sharkcz> +1
14:59:33 <dgilmore> that should leave it in place long enough for it to not be overly popular
14:59:43 <dgilmore> #agreed  we move f22 to archive when f26 is released
14:59:59 <dgilmore> which will mean f23 is moved when f27 is released
15:00:24 <dgilmore> we need to have some discussions with mirrors on ostree, and moving content around
15:00:42 <nirik> yep and pimp quick-mirror some more.
15:00:45 <dgilmore> I did send an email today asking for more mirrors to pick up /pub/fedora-secondary
15:01:19 <dgilmore> I got one email asking how to do so
15:01:38 <pbrobinson> have we got a solution to the billions of tiny files problem for ostree at all?
15:01:43 <nirik> a reply! cool. ;)
15:02:07 <dgilmore> pbrobinson: kinda
15:02:23 <dgilmore> using ostree itself to do the mirroring and not rsync seems best
15:02:59 <dgilmore> but yeah it needs some work there
15:03:04 <nirik> I wonder if it could pack up the files into an archive... but then we have to pack/unpack
15:04:05 <dgilmore> nirik: possibly
15:05:01 * nirik shrugs. dunno. ;(
15:05:03 <dgilmore> okay I will update the ticket
15:05:37 <dgilmore> we probably need a good document covering when we move releases to archive
15:05:43 <pbrobinson> dgilmore: can you ref the ticket for for notes
15:06:20 <bowlofeggs> i'm hoping to help out with the ostree mirroring problem once i get docker planning finished
15:06:28 <bowlofeggs> so i'd love to be involved there
15:06:32 <dgilmore> pbrobinson: which ticket?
15:07:10 <pbrobinson> dgilmore: you mentioned updating a ticket on a subject (not sure which one) above
15:07:18 <dgilmore> pbrobinson: the one in the topic
15:07:28 <pbrobinson> ah :)
15:07:31 <dgilmore> that is already referenced in the meeting notes
15:07:39 * pbrobinson goes back to sleep
15:08:02 <dgilmore> #topic #6420 Hypothetical maximum speed mirroring
15:08:09 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6420
15:08:18 <dgilmore> so I think that this is all done
15:08:27 <dgilmore> and we just need to tout the solution
15:08:49 <dgilmore> try get more mirrors using quick-fedora-mirror
15:09:29 <dgilmore> #topic Secondary Architectures updates
15:09:40 <dgilmore> this section probably needs to go away
15:09:52 <dgilmore> or at the least be refactored in purpose
15:09:56 <nirik> s/Secondary/Alternative/
15:10:06 <dgilmore> well that
15:10:28 <dgilmore> but build and compose issues will be more generic and readily visible
15:10:40 <nirik> yeah
15:10:55 <dgilmore> for instance I am not sure we need to discuss aarch64 anymore
15:11:05 <pbrobinson> well not for rawhide
15:11:11 <pbrobinson> for f25 we're going fine
15:11:11 <dgilmore> sure
15:11:25 <dgilmore> over time it will go away entirely
15:11:37 <pbrobinson> no issues to report, mostly boring just like I like my architectures :-P
15:11:56 <dgilmore> sharkcz: anything to report for s390x?
15:12:16 <dgilmore> I think for now I will condense it to one Alternative Archiecture topic
15:12:27 <pbrobinson> I will be working on compose things for s390x this week
15:12:30 <sharkcz> no problems build-wise, but need pbrobinson to set the composes for f25 (and fix rawhide repo job)
15:12:45 <pbrobinson> sharkcz: yes, on my todo list for today/tomorrow
15:12:51 <sharkcz> pbrobinson: thx :-)
15:12:53 * nirik still needs to file a ticket on the firewalling/networking stuff. it's on my list
15:13:11 <nirik> I'm not very happy about that situation, but I'm sure we will sort it as best we can.
15:14:24 <dgilmore> #info s390 issues being worked on
15:14:40 <dgilmore> nirik: about having the builders rw /mnt/koji
15:15:00 <nirik> hum?
15:15:11 <dgilmore> nirik: well what are you not happy about?
15:15:57 <nirik> oh sorry... well, that the builders are remote and need to only be accessable by RH employees.
15:16:14 <nirik> thats ok for secondary, but it's not great for when it's in the primary koji
15:16:37 <dgilmore> oh yeah, that is not exactly avoidable unless we got some bubble in the network, which I am not sure I see RH IT doing for us
15:17:13 <pbrobinson> still working on the IBM get us a Z-series option too but I'm not hopeful
15:17:19 <nirik> I'm not sure how the only rh employees will work if they are reachable for ansible...
15:17:33 <dgilmore> ideally IBM gives us a small z series machine to put in phx2 so it is all the same as everything else
15:17:36 <nirik> but we might also be able to get some kind of firewalling
15:17:48 <nirik> yeah, that would be ideal.
15:18:05 <pbrobinson> nirik: fact is that the area of the network the z-series is in is already accessible by partners so I'm kind of shocked that it's fully open to the rest of the network
15:18:42 <nirik> yeah, seems bad. ;( anyhow, we will keep working on it and hopefully come up wth a solution thats acceptable
15:18:51 <dgilmore> indeed
15:18:58 <dgilmore> okay lets move on
15:19:00 <dgilmore> #topic Open Floor
15:19:10 <dgilmore> bowlofeggs: you had something
15:19:12 <dgilmore> ?
15:19:27 * nirik had a few items to ask about/note.
15:20:04 <dgilmore> nirik: why don't you go
15:20:09 <nirik> ok.
15:20:19 <nirik> dgilmore: did you get the f26 key reissued and ready to go?
15:20:24 <dgilmore> nirik: yes
15:20:37 <nirik> cool.
15:20:44 <dgilmore> I am waiting on a resolution of the type= issue to send in the PR
15:20:53 * nirik tries to recall the other items.
15:21:03 <dgilmore> everyone has access to the f26 key, and needs to change thier passwords
15:21:05 <nirik> oh, we have lives that are hanging in both branched and rawhide now
15:21:14 <pbrobinson> dgilmore: that's just primary key right?
15:21:20 <dgilmore> pbrobinson: right
15:21:33 <nirik> https://bugzilla.redhat.com/show_bug.cgi?id=1374809
15:21:41 <dgilmore> the primary key ended up with fedora-25 in the uid
15:21:46 <nirik> if anyone has time to try and figure it out
15:21:55 <mboddu> dgilmore: cool, I will change my password later
15:21:59 <nirik> oh, one last one:
15:22:37 <nirik> updates/updates-testing repodata is still not right. It's adding weak deps for new updates that are added in, but it's not seeing them on updates that were already in repo before the change.
15:22:48 * dgilmore suspects that the live is a packaging change causing different ordering of the packages being installed
15:22:48 <nirik> I nuked the repocache dirs and that didn't seem to do it.
15:22:54 <dgilmore> and causing something to hang
15:23:04 <dgilmore> nirik: that will be expected
15:23:04 <nirik> is there any other caching anyone knows of?
15:23:13 <nirik> why?
15:23:22 <dgilmore> nirik: yes, mash uses the previous repo
15:23:31 <nirik> ah poop. :(
15:23:45 <dgilmore> though would have to look at what exactly bodhi is doing
15:23:48 <nirik> so how do we get around this? manually update the repodata?
15:24:03 <dgilmore> depends on what bodhi is doing
15:24:17 <dgilmore> I honestly an not 100% sure how bodhi calls mash
15:24:18 <nirik> bodhi seems to make a repocache dir...
15:24:26 <nirik> but yeah, I don't know either
15:25:00 <dgilmore> It will take someone looking into what bodhi is doing
15:25:04 <nirik> I can ask around and see if we can sort it. If anyone can, feel free. ;)
15:25:12 <nirik> thats all I had
15:25:17 <dgilmore> okay
15:25:17 <puiterwijk> I have two things
15:25:24 <dgilmore> bowlofeggs: seems to have vanished
15:25:34 <dgilmore> puiterwijk: go for it
15:25:50 <bowlofeggs> oh!
15:26:01 <puiterwijk> So, first of all: we now have autosigning in staging, and planning to enable that for production tonight. We can then start to enable tags one by one
15:26:04 <bowlofeggs> there's a new bodhi 2.1.9 beta deployed to stg - please test!
15:26:16 <dgilmore> bowlofeggs: uno momento por favor
15:26:36 <dgilmore> puiterwijk: nice :)
15:26:47 <dgilmore> puiterwijk: we should start with f26
15:26:56 <pbrobinson> \o/
15:26:57 <nirik> puiterwijk: so to clarify: this uses autosign02 and listens for fedmsgs on builds?
15:27:07 <puiterwijk> nirik: yes.
15:27:11 <nirik> or is there any docs on it all? (if not we should make one :)
15:27:29 <dgilmore> we need docs and debugging instructions
15:27:39 <puiterwijk> nirik: I'm also adding a cron job that will make it go through its assigned tags to forcefully sign everything that it didn't pick up (due to fedmsg issues or whatever)
15:27:40 <nirik> and it moves tags after signing? or ?
15:27:42 <puiterwijk> dgilmore: yep, will do.
15:27:51 <puiterwijk> nirik: yep. It moves them to a configured target tag
15:28:10 <puiterwijk> So you configure per tag with which key to sign and to which tag to move afterwards.
15:28:17 <nirik> ok, if so then we should perhaps adjust rawhide to go to a pending-signing tag and have it move after they are signed.
15:28:22 <nirik> yeah. cool. ;)
15:28:24 <puiterwijk> Yep, that was the plan
15:28:26 <dgilmore> we will need to adjust build targets
15:28:28 <nirik> rawhide signing here we come!
15:28:41 <nirik> a long dream.
15:28:47 <puiterwijk> dgilmore: yep. So we'll sync up about that :)
15:28:55 <dgilmore> nirik: dream/nightmare
15:29:16 <nirik> oh, did we want to repave autosign02 before moving to prod? and adjust it so only a few people have access?
15:29:25 <nirik> and perhaps we can nuke autosign01?
15:29:26 <puiterwijk> Yep, was going to do that
15:29:33 <dgilmore> nirik: /me would say yes
15:29:56 <puiterwijk> I have all in my ansible working directory now, and after that's pushed I'll execute it all
15:30:23 <puiterwijk> PSA for signers: part of that will requires updating sigul, so I'll sync up when it's not being used so I can do that
15:30:25 <dgilmore> cool
15:30:40 <nirik> dgilmore: is it ok for puiterwijk to have access to help setup/debug?
15:30:55 <dgilmore> nirik: it should be okay
15:31:01 <nirik> cool.
15:31:40 <puiterwijk> Also, the new version has a more strict username vs certificate CN check. Theoretically, if the certificates are issued correctly, nobody should notice anything. But if any certs are misissued, it will start screaming Authorization Denied. If it does, we have a way to exempt certain certificates from this check.
15:31:46 <puiterwijk> (new version of sigul)
15:32:04 <nirik> and you wanted to do a new CA right? so we all need new certs
15:32:26 <dgilmore> ?
15:32:39 <puiterwijk> nirik: we could, but that would be something else. So we can do it at the same time, or just do that later
15:32:54 <nirik> ok, later is fine.
15:32:58 <dgilmore> that is news to me
15:33:00 <nirik> or more discussion on it then later.
15:33:05 <dgilmore> puiterwijk: what and why?
15:33:18 <puiterwijk> dgilmore: currently the private key for the CA that issues the client certs is "online" on fas01. I suggested we might make a separate CA for this and keep that one offline.
15:33:35 <puiterwijk> (sigul client certs*)
15:33:51 <dgilmore> puiterwijk: sure, there is other protections in place as well
15:34:02 <puiterwijk> Yes, I am aware. Was just a suggestion :)
15:34:15 <puiterwijk> dgilmore: just to verify: your question mark was about the new CA, and not the CN vs username check, right?
15:34:25 <dgilmore> at this point I would rather wait.
15:34:31 <puiterwijk> sure
15:34:41 <dgilmore> there still needs to be a tie in to koji, and the same ca
15:34:54 <dgilmore> and we have other changes coming in auth land
15:34:56 <puiterwijk> doesn't need to be the same CA
15:35:16 <puiterwijk> but yeah, more changes are upcoming, so we can discuss this further
15:35:26 <nirik> sure, sorry for sidetracking. ;)
15:35:52 <puiterwijk> dgilmore: but you understood the CN vs username check, and how that can make it shout if we have misissued certs?
15:36:07 <dgilmore> puiterwijk: sure
15:36:17 <puiterwijk> great. Wanted to make sure that came across
15:36:50 * dgilmore would rather any discussion about changes to the auth to be an all encompasing plan rather than disjoint changes
15:37:12 <puiterwijk> yep, agreed
15:37:35 <dgilmore> does not mean all auth has to be the same
15:37:43 <dgilmore> just that its part of a single plan
15:37:58 <dgilmore> is that all puiterwijk ?
15:38:10 <puiterwijk> dgilmore: yep, that's all from me for now
15:38:13 <puiterwijk> thanks for bearing with me :)
15:38:13 <dgilmore> #info puiterwijk has autosigning working, plans to deploy to production today
15:38:23 <dgilmore> bowlofeggs: okay the floor is yours now
15:39:04 <dgilmore> 15:26 < bowlofeggs> there's a new bodhi 2.1.9 beta deployed to stg - please test!
15:39:11 <dgilmore> he has vanished agian
15:39:14 <dgilmore> again
15:39:23 <dgilmore> but there is a new bodhi in stg
15:39:33 <dgilmore> and he would like it to be run through its paces
15:39:45 <nirik> sounds good
15:39:58 <dgilmore> anything else?
15:40:10 <dgilmore> #info new bodhi in stg please test
15:40:59 <dgilmore> I willa ssume we are done
15:41:03 <dgilmore> thanks all
15:41:07 <dgilmore> #endmeeting