15:43:56 <dgilmore> #startmeeting RELENG (2015-07-20)
15:43:56 <zodbot> Meeting started Mon Jul 20 15:43:56 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:43:56 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:44:08 <dgilmore> #meetingname releng
15:44:08 <zodbot> The meeting name has been set to 'releng'
15:44:09 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
15:44:09 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
15:44:11 <dgilmore> #topic init process
15:44:21 <maxamillion> \o/
15:44:25 * pbrobinson is here
15:44:31 * sharkcz is here
15:45:08 * bochecha__ is here, but working on something else at the same time
15:45:33 * maxamillion is here
15:46:00 <dgilmore> #topic #6164 bodhi2 status update requested
15:46:07 <nirik> morning
15:46:08 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6164
15:46:19 <dgilmore> lmacken: where is bodhi2 this week?
15:49:29 <dgilmore> hrrm I guess he is not here
15:49:35 <dgilmore> nirik: any idea?
15:49:46 <nirik> not sure. I'll get a hold of him and find out.
15:49:56 <nirik> I really hope we can land some stuff this week before alpha next
15:50:09 <dgilmore> nirik: right me too
15:50:16 <maxamillion> #halp
15:50:24 <maxamillion> ....
15:50:24 <dgilmore> #info nirik to find out from lmacken where things are
15:50:45 * masta is here
15:52:10 <dgilmore> #topic Secondary Architectures updates
15:52:12 <dgilmore> #topic Secondary Architectures update - ppc
15:52:20 <dgilmore> pbrobinson: how is ppc?
15:53:13 <pbrobinson> not bad, I need to look further and sync up with the status of pungi4 on primary this week to work out where that is so as to know how we're going ther and what I need to get in place for alpha
15:53:20 <pbrobinson> mass rebuild is basically complete
15:53:31 <dgilmore> pungi is blocked on mock still
15:53:42 * maxamillion slaps mock around a little
15:53:46 <pbrobinson> looking also at docker/golang 1.5 and friends for docker related stuff on both power/aarch64
15:53:52 <dgilmore> cool
15:54:09 <pbrobinson> so I suspect this week will be busy :)
15:56:18 <dgilmore> this week will likely be insane
15:56:25 <sharkcz> buildrawhide seems to have an issue on ppc -  "No 'src' pkgs in any repo"
15:56:45 <dgilmore> odd
15:57:12 <dgilmore> had branched been enabled?
15:57:27 <pbrobinson> sharkcz: I believe we're seeing the same on aarch64, and yes branched  has been enable
15:57:36 <sharkcz> but haven't seen a report yet
15:57:52 <sharkcz> maybe another issue ...
15:58:00 <pbrobinson> I was going to have a look at it but haven't got to it yet today
15:58:06 <pbrobinson> s/was/am
16:03:03 <maxamillion> anything else on the topic of ppc?
16:03:18 <pbrobinson> not from me
16:03:19 <sharkcz> not from me
16:03:21 <sharkcz> :-)
16:04:57 <masta> so dgilmore is in another meeting, as am I.... lets  just go forward
16:05:14 <maxamillion> where's the agenda for this meeting posted?
16:05:17 <masta> #topic Secondary Architectures update - s390
16:05:22 <maxamillion> masta: +
16:05:24 <maxamillion> +1*
16:05:32 <masta> maxamillion: https://fedoraproject.org/wiki/ReleaseEngineering/Meeting_Process
16:05:40 <maxamillion> thanks
16:05:48 <sharkcz> s390 looks good, up-to-date with primary for both f23 and 24
16:05:59 <dgilmore> sorry got disracted
16:06:11 <sharkcz> buildbranched works, but buildrawhide fails, will look on that
16:06:18 <dgilmore> sharkcz: fun
16:06:31 <sharkcz> and we have a plan with nirik how to move the hub to the new host
16:06:58 <nirik> rawhide in primary also failed... might be due to a hawkey packaging issue.
16:07:08 <sharkcz> dgilmore: maybe an issue with EL-6 host with rawhide chroot
16:07:16 <nirik> when would you like to schedule the migration?
16:07:58 <sharkcz> nirik: almost any time, let me know and I will stop the old hub
16:08:23 <nirik> ok, let me figure out what my week looks like and we can schedule it later this week?
16:08:30 <sharkcz> ok
16:08:36 * nirik wanted to do mass updates/reboots tomorrow and wed nights...
16:08:54 <dgilmore> sharkcz: quite possible
16:09:09 <dgilmore> sharkcz: where are we at moving the hub?
16:09:29 <dgilmore> sharkcz: I really do not think it should block on the shared koji-shadow setup
16:09:36 <sharkcz> dgilmore: sometime this week ^^
16:09:43 <dgilmore> I think we should get the hub migrated asap
16:09:46 <dgilmore> sharkcz: okay
16:09:56 <sharkcz> dgilmore: yes, the shadow setup will be the 2nd phase later
16:10:16 <dgilmore> perfect
16:10:31 <dgilmore> anything else?
16:10:39 <sharkcz> nope
16:10:53 <dgilmore> #topic Secondary Architectures update - arm
16:11:02 <dgilmore> pbrobinson: how is things?
16:11:04 <pbrobinson> catching up on the backlog
16:11:16 <pbrobinson> we're about 3500 older so down a lot from last week
16:11:51 <pbrobinson> late this week we have the builders being shipped to PHX so we'll be down some capacity for a bit but I think we should be OK
16:12:07 <dgilmore> cool
16:12:16 <dgilmore> that should help things a bit when they are in place
16:12:28 <masta> not the best time, but yeah.. will help
16:12:39 <masta> (there is no great time ever)
16:12:57 <pbrobinson> yes, pwhalen will upgrade firmwares before they ship
16:13:28 <pbrobinson> I've got most bits in place so we can just reinstall them the moment they are ready to go with ansible that was done for the others ages ago
16:13:51 <pbrobinson> dgilmore: I'll work with you on new certs for them
16:14:00 <dgilmore> pbrobinson: okay awesome
16:14:37 <dgilmore> pbrobinson: if we know the hostnames we can make certs and put in ansible. setup dns and everything else beforehand
16:14:46 <dgilmore> add the hosts to koji
16:14:53 <pbrobinson> I did a bunch of ansible stuff for both aarch64 and power end of last week, will be finishing that up over the next couple of days
16:15:02 <dgilmore> so all thats left to do is pxe install and run the playbook
16:15:07 <pbrobinson> dgilmore: dns/IPs already done
16:15:18 <pbrobinson> dgilmore: i have the host list for koji
16:15:21 <dgilmore> pbrobinson: okay.
16:15:39 <pbrobinson> dgilmore: basically all I have left of that to do is certs.... and you were off Friday ;-)
16:15:56 <pbrobinson> I've not added them to koji yet, but that's a few mins
16:16:53 <dgilmore> pbrobinson: okay
16:17:17 <dgilmore> that will take all of about 15-20 minutes to make the certs and put them in ansible-private
16:18:07 <pbrobinson> yep, so I'm mostly ready and basically plan to finish up the power stuff this week, have it ready for aarch64 stuff next week when smooge in in PHX
16:18:43 <dgilmore> awesome
16:18:51 <dgilmore> anything else?
16:19:12 <pbrobinson> no, I don't think so
16:19:17 <dgilmore> #topic Open Floor
16:19:31 <dgilmore> anyone got anything else?
16:19:32 <maxamillion> I have a couple small things
16:19:41 <bochecha__> I have one thing
16:19:44 * nirik had also a few small ones. ;)
16:20:11 <dgilmore> maxamillion: you're first
16:20:12 <maxamillion> one thing, I just wanted to mention that I'm taking over as the Docker Fedora Base Image maintainer, so now someone from rel-eng will be the official point of contact for that stuff --> https://github.com/docker-library/official-images/pull/909
16:20:33 <maxamillion> and the other thing I just realized is better for a mailing list thread so I'll go there with it
16:20:37 <nirik> cool. :)
16:20:40 <pbrobinson> maxamillion: what does that entail?
16:21:17 <maxamillion> pbrobinson: I'll have permission to update the base docker images in the hub
16:21:25 <pbrobinson> ah, OK
16:21:50 <pbrobinson> maxamillion: need to sync with you on non x86 naming again at some point
16:21:55 <maxamillion> pbrobinson: +1
16:22:08 <dgilmore> maxamillion: Fedora's docker base image is maintained by the BaseWG
16:22:21 <dgilmore> but it is a mess
16:22:54 <dgilmore> maxamillion: for non= x86 naming I think we should go with what was discussed on the list
16:23:01 <maxamillion> dgilmore: yeah, rgr
16:23:50 <maxamillion> dgilmore: I probably need to sync with BaseWG at some point and figure out how we want to handle that and if we want to just replace me with the BaseWG mailing list as the maintainer point of contact (which would ultimately make more sense)
16:24:08 <dgilmore> maxamillion: BaseWG has no list
16:24:11 <dgilmore> just devel@
16:24:13 <maxamillion> ah
16:24:15 <maxamillion> hrmmm
16:24:20 <dgilmore> and is largely non functinal :(
16:24:43 <dgilmore> maxamillion: what is really needed is a way we can just upload new base images upstream at will
16:24:45 <maxamillion> in the mean time though, lsm5 wanted to get it in the hands of someone in rel-eng and he and I have worked on docker quite a bit over the last few years so he asked if I would mind doing it
16:24:58 <maxamillion> dgilmore: yeah, docker doesn't do that :(
16:25:07 <maxamillion> dgilmore: not for the base images anyways
16:25:40 <dgilmore> maxamillion: right
16:25:51 <dgilmore> its one of the horribly broken things about docker
16:26:01 <dgilmore> among many :(
16:26:19 <maxamillion> it could ultimately be fixed, it's just an oddity about the way their distribution avenue is setup
16:26:28 <maxamillion> anyhoo, on to who ever was next for open floor
16:26:43 <maxamillion> #info maxamillion to be maintainer of Fedora Docker Base Image in Docker Hub
16:26:54 <maxamillion> #link https://github.com/docker-library/official-images/pull/909
16:27:29 <dgilmore> bochecha__: your floor
16:27:31 <maxamillion> #info this will ultimately be transitioned to a WG that will shepard the docker base image
16:27:53 <bochecha__> I just wanted to mention it here, that I'd love a review of my fedpkg pull-request: https://pagure.io/fedpkg/pull-request/2/
16:28:17 <bochecha__> this is pretty much the last code change needed before we can actually flip the switch over to sha512 :)
16:28:41 <dgilmore> bochecha__: will try to get on it this week
16:28:58 <masta> bochecha__: I did try that and it worked fine for me
16:29:06 <bochecha__> masta: from the copr? cool!
16:29:31 <bochecha__> dgilmore: once this one (or something like it) is merged, how do you want to proceed? do we actually move to sha512?
16:29:34 <dgilmore> #info bochecha__ has a pull request https://pagure.io/fedpkg/pull-request/2/ to be reviewed, last thing needed to be able to move lookaside cache off of md5sum
16:29:55 <dgilmore> bochecha__: I still want a flag day
16:30:13 <bochecha__> what do you want as a flag day?
16:30:13 <dgilmore> bochecha__: as we will have to be sure people are using the latest fedpkg
16:30:20 <bochecha__> right
16:30:22 <bochecha__> so there are two things
16:30:34 <bochecha__> 1. make a new fedpkg which uploads as sha512
16:30:43 <dgilmore> I would like to roll out the ca cert changes
16:30:45 <bochecha__> 2. on the server, refuse new md5 uploads
16:30:52 <bochecha__> the two could be done independently
16:31:05 <bochecha__> we could push a fedpkg update that uploads as sha512
16:31:32 <bochecha__> and after a certain time of it being in all Fedora/EPEL branches, we could remove the md5 support from the server
16:31:44 <bochecha__> the flag day you want would be the second one, right?
16:31:46 <dgilmore> bochecha__: my fear is that we will get complaints from people using old fedpkgs that do not support teh new sources format
16:31:50 <pbrobinson> presumably for 2. that would enforce a minimum fedpkg version?
16:31:55 <bochecha__> pbrobinson: yes
16:32:18 <maxamillion> I don't think I follow ... what is this "flag day"?
16:32:20 <pbrobinson> is there any logging on server side as to the versions of clients used to connect?
16:32:23 <dgilmore> bochecha__: some people use really old versions of tools
16:32:27 <bochecha__> pbrobinson: which is why we should wait "after a certain time of it being in all Fedora/EPEL branches"
16:32:37 <dgilmore> pbrobinson: there is no such logging
16:32:40 <bochecha__> pbrobinson: no, but that could be added
16:32:50 <pbrobinson> for old clients as well?
16:32:53 <bochecha__> pbrobinson: however, there is logging of what hashtype (md5 or sha512) was used for an upload
16:33:08 <bochecha__> dgilmore: right, but they will need to upgrade, at some point, won't they?
16:33:17 <bochecha__> dgilmore: or do you want to go on supporting those old versions forever? :-/
16:33:21 <dgilmore> maxamillion: there is a few changes that we want/need to make to the workflow that will require people to update the client side of things
16:33:32 <maxamillion> dgilmore: ok
16:33:34 <dgilmore> new configs and new setup
16:33:48 <dgilmore> bochecha__: they will. the flag day
16:33:51 <maxamillion> dgilmore: I'm not familiar with the terminology of "flag day" though ... what does that mean?
16:33:54 <pbrobinson> my query about the logging is presumably the new sha func is already available on a client side. It would be use from a % PoV to know how many would be affected by the change
16:34:02 <bochecha__> dgilmore: right, so that "flag day" would be when we remove the md5 support from the server
16:34:12 <dgilmore> maxamillion: its a day where we flag that changes are happening and the users need to do things
16:34:14 <bochecha__> dgilmore: there is no reason to couple that removal on the server to the update on the client, is there?
16:34:19 <maxamillion> dgilmore: oh ok
16:34:20 <dgilmore> bochecha__: no
16:34:56 <masta> how long do we wait from #1 to #2 ?
16:35:00 <bochecha__> pbrobinson: current tools are already able to upload/download with sha512, and the server is already capable of accepting both
16:35:07 <dgilmore> bochecha__: if people use client tools that only support the old style sources format and they get a checkout that has the new one we will get complaints
16:35:29 <bochecha__> dgilmore: well, yes, but there's nothing we can do about it
16:35:33 <dgilmore> bochecha__: we have not yet changed the format of sources file right?
16:35:38 <dgilmore> bochecha__: there is
16:35:42 <bochecha__> dgilmore: not yet, that's the next pull request
16:35:44 <dgilmore> bochecha__: we wait until the flag day
16:36:01 <bochecha__> dgilmore: I'm planning on sending a single pull-request that will both move to sha512 and change the sources format
16:36:06 <pbrobinson> could we enable it and see after a few weeks what % of new uploads were md5? If there's zero we know it's low impact, if it's a high % we know we have an issue
16:36:07 <dgilmore> bochecha__: it has been the plan all along to make the change all at once
16:36:25 <dgilmore> bochecha__: we can not change that until the flag day
16:36:36 <bochecha__> pbrobinson: that's along the lines of what I was suggesting, but dgilmore doesn't want that
16:37:03 <bochecha__> the thing is, there's no way you can enforce such a flag day
16:37:11 <dgilmore> bochecha__: we can
16:37:24 <pbrobinson> well you can in that old fedpkg will stop working
16:37:25 <bochecha__> one issue is that the new fedpkg (with new source format and sha512) will land in rawhide about 2 weeks before it lands in epel6
16:37:32 <dgilmore> bochecha__: because once we change the ca cert etc, the existing setups will not work
16:37:38 <bochecha__> so that's not a flag day, that's a "flag 2 weeks"
16:38:14 <masta> yeah, there was a meeting a while ago and we had agreement to flag-day this stuff, and we thought back then we could do it.
16:38:22 <dgilmore> bochecha__: with the changes we are making the changes break totally the current setup
16:38:24 <masta> has something changed since then bochecha__ ?
16:38:35 <bochecha__> masta: since what?
16:38:39 <dgilmore> masta: bochecha__ does not want to wait
16:38:39 <pbrobinson> can it not be controlled from server side? IE get the new clients out and then start accepting them on the server from the flag day?
16:38:48 <dgilmore> pbrobinson: nope
16:38:52 <bochecha__> dgilmore: i didn't say I don't want to wait
16:39:03 <bochecha__> dgilmore: I'm saying I don't see the point of waiting :)
16:39:07 <dgilmore> bochecha__: sorry, but it seems that way
16:39:18 <pbrobinson> it seems we're just going around in circles here
16:39:30 <bochecha__> no matter when you do it, you're going to break old setups, you'll get complaints
16:39:38 <dgilmore> bochecha__: I want to wait purly because I do not want to have to force people to update things twice
16:40:00 <dgilmore> pbrobinson: it is a discussion we have had multiple times
16:40:05 <bochecha__> in any case, just to be clear : the pull request I already submitted could be merged and released as an update **without breaking anything**
16:40:12 <pbrobinson> dgilmore: I'm aware
16:40:17 <dgilmore> and it is always the same
16:40:30 <bochecha__> dgilmore: ok, so flip the sha512/sources switch at the same time as the new CAs, then
16:40:47 <pbrobinson> it seems to me the fact this is controlled on the client side is bad, the fact we have no means of telling from the server side what versions of the client people are using is alos bad
16:40:48 <dgilmore> bochecha__: that has been the plan
16:41:23 <bochecha__> the only thing that bothers me, is that I've been working on this migration for the past 17 months, I don't think we're going to migrate to these new CAs any time soon, and I don't really feel like keeping at this migration for one (or more) more years
16:42:14 <dgilmore> bochecha__: I get that
16:42:35 <bochecha__> I mean, we'll always have forced client updates in this way, every couple of years, as we improve things
16:42:55 <dgilmore> bochecha__: and if we can be sure that changing the format of the sources file will not break clients I would say go and do it
16:42:59 <dgilmore> but we can not be
16:43:03 <bochecha__> oh, we can be
16:43:11 <bochecha__> we can be absolutely sure that it will break old clients :)
16:43:28 <dgilmore> bochecha__: we have not forced changes on clients
16:43:35 <dgilmore> not where the old clients broke
16:43:45 <dgilmore> we have to enable new features
16:44:41 <bochecha__> just FYI, the oldest client that can support the new sources file format is **pyrpkg** 1.32
16:44:50 <bochecha__> i.e it's been supported for some time already
16:45:03 <bochecha__> there might still be older clients in use for sure, just providing the data point :)
16:45:52 <bochecha__> in any case, that's a future, yet unsubmitted change, the pull-request I already submitted doesn't break anything, it doesn't change to sha512 or the new sources format, it just finishes the preparations for it
16:46:02 <dgilmore> bochecha__: 1.32 was only on rawhide and oin March this year
16:46:23 <bochecha__> I know
16:46:24 <dgilmore> 1.33 went everywhere in April
16:46:34 <dgilmore> that is not very long when it comes to what people run
16:46:55 <bochecha__> sure, again, I was just providing the actual data point about what's the oldest compatible client version
16:48:18 <dgilmore> as we know we have some changes coming up that will require people get new versions of the tools and use new configs. I want to do it at once
16:48:27 <bochecha__> anyway, if I can just get that pull-request reviewed, merged, and released in all Fedora/EPEL branches, then I'll be happy, I'll send the pull-request to flip the switch and it can get merged whenever it gets merged
16:49:05 <dgilmore> anything else here?
16:49:09 <bochecha__> nothing from me
16:49:16 <lmacken> dgilmore: sorry, was afk before..
16:49:21 <bochecha__> and I have to go now, sorry for leaving early
16:49:27 <lmacken> dgilmore: I'll be working on getting the bodhi2 backend/masher deployed today/tomorrow
16:50:00 <lmacken> there were some API changes in the fedmsg consumer API that I need to work on, but we should hopefully be able to do test pushes shortly
16:50:35 <dgilmore> lmacken: okay, we have a week to get it in production if we are going to do it before f23 is released
16:51:11 <dgilmore> nirik: you had something?
16:51:28 <lmacken> dgilmore: okay, it's going pretty smooth so far but we'll see how the push process works in stg
16:51:34 <nirik> yeah, lets see really quickly...
16:52:02 <nirik> 1) the broken deps reports are broken for f23/rawhide. It's a change in mash on perms on spam-o-matic.
16:52:19 <dgilmore> nirik: hrrm okay, they should not have changed
16:52:20 <nirik> we need to either do a new mash with 755 for it, or change the releng scripts to call 'python ...'
16:52:30 <dgilmore> nirik: it will have to be fixed in the package
16:52:31 <nirik> I put in a PR to change the compose part.
16:52:33 <nirik> ok.
16:52:40 <nirik> It's in /usr/share tho, so 755 seems poor
16:52:44 <dgilmore> it was not an intentional change
16:52:44 <nirik> but ok.
16:53:07 <dgilmore> it probably should go in /usr/libexec
16:53:31 <nirik> 2) I'm planning on doing mass updates/reboots tuesday for releng/build stuff (tomorrow) night. Will send outage announce right after this meeting.
16:53:43 <nirik> 3) (and then all the rest of things wed night)
16:53:58 <nirik> there was something else, but I cannot recall it now. ;) so, move on
16:54:09 <nirik> oh, fedora-24 keys? going to make them soon?
16:54:20 <dgilmore> can we hold that off until we know that we can actually do a compose
16:54:30 <dgilmore> i would hate for some change to break things more
16:54:40 <nirik> well, I would really like to get stuff done before freeze next week.
16:54:44 <nirik> but we can if need be.
16:54:45 <dgilmore> going to make fedora 24 keys this week
16:55:16 <nirik> I'd be inclined to do the compose stuff now and make sre we can compose with the updated stuff.
16:55:30 <nirik> because we are going to have to apply someday anyhow. ;)
16:55:39 <maxamillion> I need to duck out so I can get lunch before my next meeting ... see you folks later
16:57:25 <dgilmore> honestly I would be okay not updating them until after freeze unless there is something security criticak;ll[Cl
16:58:36 <nirik> there are a number of things.
16:58:44 <nirik> we have been holding off. they haven't been rebooted in 2 months
16:58:55 <nirik> nothing is super urgent, but lots of little thigns.
16:59:09 <nirik> I can exclude machines you like I suppose.
17:00:07 <nirik> we can talk more on it out of meeting.
17:00:12 <nirik> no need to keep meeting open.
17:03:14 <dgilmore> sorry network dropped out here
17:03:26 <dgilmore> lets talk out of meeting
17:03:40 <dgilmore> anyone have anything else?
17:03:45 <dgilmore> or end the meeting?
17:03:50 <nirik> +1 to end
17:05:34 <dgilmore> #endmeeting