14:36:01 <dgilmore> #startmeeting RELENG (2016-07-11)
14:36:01 <zodbot> Meeting started Mon Jul 11 14:36:01 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:36:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:36:01 <zodbot> The meeting name has been set to 'releng_(2016-07-11)'
14:36:01 <dgilmore> #meetingname releng
14:36:01 <zodbot> The meeting name has been set to 'releng'
14:36:01 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
14:36:01 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
14:36:05 <dgilmore> #topic init process
14:36:26 <dgilmore> #chair mboddu
14:36:26 <zodbot> Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll
14:36:50 * nirik is sort of here
14:39:57 * sharkcz is here
14:40:18 <dgilmore> win 6
14:40:20 <dgilmore> opps
14:43:11 <maxamillion> .hello maxamillion
14:43:11 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
14:43:33 * nirik has a few things about updates/atomic composes
14:44:22 <bochecha_> afternoon
14:45:00 <dgilmore> lets get started
14:45:32 <dgilmore> nirik: we can cover that now then do the tickets with meeting keyword
14:45:56 <dgilmore> #topic nirik updates/atomic
14:46:01 <nirik> sure
14:46:13 <nirik> so, we have two issues. I filed tickets for both saturday:
14:46:42 <dgilmore> I think the f24 is a bug in bodhi
14:46:44 <nirik> 1) fedora 22 updates-testing the atomic compose is failing. I have no idea why, but it looks like ostree is crashing. We need to gather a core
14:46:55 <dgilmore> its supposed to make a new repo if one does not exist
14:47:04 <nirik> 2) we just enabled f24 atomic, and yeah, it's not working...
14:47:25 <maxamillion> wait, what is this?
14:47:28 <nirik> so, I am hoping to catch lmacken when he gets in to work on those two items...
14:47:47 <nirik> maxamillion: atmoic updates/updates-testing
14:47:57 <dgilmore> the content in https://kojipkgs.fedoraproject.org/atomic/24/ is from the branched nightly composes
14:48:20 <dgilmore> we did not make any atomic bits as part of the f24 ga composes
14:48:31 <maxamillion> nirik: ah ok
14:48:43 <dgilmore> maxamillion: the ostree repo has not been getting built
14:49:10 <dgilmore> so what is in place and in the two week compose is the last branched compose
14:49:20 <nirik> https://fedorahosted.org/rel-eng/ticket/6439
14:49:56 <dgilmore> #info ostree repos for f24 were not getting built and bodhi is not doing what it is supposed to
14:49:57 <nirik> I don't know whats supposed to be in the place where there's nothing now. I think bodhi is supposed to make an empty repo there tho.
14:50:12 <dgilmore> #info f22 updatest-testing failing in weird ways
14:50:31 <dgilmore> nirik: that is my understanding
14:50:42 <dgilmore> nirik: and thats what it looks like in the config
14:50:55 <dgilmore> it has some snippets for initing a repo
14:50:56 <nirik> so if we can get a hold of masta we should have him hold off on pushing until we sort these... unless it's going to take a long time
14:51:21 <nirik> I disabled the f24 stuff so we could get that push finished...
14:51:35 <dgilmore> nirik: when I changed the configs the other week I removed soem weird repo that was configured for f22
14:51:51 <nirik> oh? huh. wonder if thats related.
14:52:11 <dgilmore> its possible that something was setup o make it use different bits that I broke by disabling some temp repo that was defined
14:52:32 <dgilmore> but if that is teh case we should not have been doing what we were doing
14:54:11 <dgilmore> https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/roles/bodhi2/backend/templates/atomic-config.py.j2?id=119db0826e013fda1899589607b89f06be49f9f5
14:54:17 <nirik> well, we had some side repos when we first setup things to get it working, but I thought we dropped all of them long ago
14:54:18 <dgilmore> I removed f22-temprepo
14:54:37 <nirik> huh.
14:54:45 <nirik> we could try readding it, or look whats in there.
14:54:48 <dgilmore> I also fixed up the git urls
14:55:47 <nirik> yeah, looks like it has a newer rpm-ostree
14:56:18 <nirik> rpm-ostree-2015.7-6.fc22
14:56:31 <dgilmore> :sigh:
14:56:35 <nirik> and f22 has rpm-ostree-2015.3-8.fc22
14:57:13 <nirik> There's one in between the two in testing...
14:57:24 <nirik> rpm-ostree-2015.7-3.fc22
14:57:34 <dgilmore> #info there seems to have been a f22 rpm-ostree update in a side repo rather than gone through the regular channels
14:57:58 <nirik> I suppose since f22 has only a week to live... we might just re-enable that and let it die with f22?
14:58:05 <dgilmore> that is really bad practice
14:58:12 * nirik agrees
14:58:22 <dgilmore> I guess
14:58:44 * dgilmore does not like it
14:59:03 <nirik> so which one would it be using there? the one in updates-testing? or the one in updates?
14:59:08 <nirik> it's in a mock chroot
14:59:19 <dgilmore> the repo is all dated 2015-07-28
14:59:24 <dgilmore> its been there a year
14:59:29 <dgilmore> almost
14:59:41 <dgilmore> updates
14:59:48 <dgilmore> well updates-testing
15:00:03 <dgilmore> nirik: afaik bodhi makes custom mock configs on the fly
15:00:13 <maxamillion> shouldn't we just drop f22 atomic? ... I thought one of the big goals of atomic was to not track older releases and only push the latest
15:00:18 <nirik> yeah, so perhaps we could unpush that f22-updates-testing one?
15:00:41 <dgilmore> maxamillion: that is not really how we are tooled and setup as a project
15:00:57 <dgilmore> nirik: thats likely a good option
15:01:22 <nirik> I can give that a quick whirl.
15:01:58 <dgilmore> maxamillion: the refs would need to change, we would have to change the workflows
15:02:45 <dgilmore> maxamillion: some people expressed the desire to do what you mentioned, but we have not gone down a road that will let us do that
15:03:11 <dgilmore> maxamillion: and some users want to follow paths we do not really want to go down
15:03:55 <maxamillion> I'm not sure I follow, but I just feel like there's a lot of work being done to support things that we we're not on the hook to support
15:04:06 <dgilmore> #info nirik to unpush f22 updates-testing rpm-ostree build and attempt to restart the updates push
15:04:45 <dgilmore> maxamillion: as a project we have set expectations and communicated how things will work
15:05:04 <dgilmore> maxamillion: some people have expressed wanting to do things differently
15:05:29 <dgilmore> but we have not communicated that and have not setup the tooling and processes to allow it
15:06:08 <nirik> I have no idea what our atomic consumers want. ;) We might be able to get some stats on usage...
15:06:16 <dgilmore> I honestly do not know if anyone is still using f22 ostree
15:06:24 <dgilmore> but it does have one week to live
15:07:49 <dgilmore> nirik: and I guess for f24 we could as masher make an empty repo
15:08:02 <dgilmore> but bodhi is supposed to be doing that
15:08:11 <nirik> unpushing that rpm-ostree didn't help. Still failed the same way
15:08:37 <nirik> oh, but I bet it's using the existing updates-testing repo, so it still has that package.
15:08:46 <nirik> not sure how to get around that
15:09:04 <nirik> I guess enable the tmp one again, get it to finish, then disable it.
15:09:57 <dgilmore> I guess
15:10:33 <dgilmore> #info we need to stop doing hacks
15:13:20 * nirik nods
15:13:50 <dgilmore> anything else here?
15:14:11 <nirik> nope.
15:14:54 <dgilmore> #topic #6420 Hypothetical maximum speed mirroring
15:15:03 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6420
15:16:44 <nirik> I think the only things really left here are fedmsgs and cleanup and announcing
15:17:23 <maxamillion> that looks pretty cool
15:17:29 <dgilmore> we need to update all the secondary processes to update the file
15:17:44 <nirik> ah yeah, that too
15:17:52 <dgilmore> and to emit fedmsgs
15:18:05 <dgilmore> I think it looks worthwhile
15:18:07 <nirik> download-ib01 is using it. It's working great... aside from one time when it's disk filled
15:20:25 <dgilmore> cool
15:20:42 <nirik> I really hope we can get people who hit dl.fedoraproject.org to use it.
15:21:02 <dgilmore> nirik: indeed, it seems like it should help a lot
15:21:21 <dgilmore> there was the mirror that reported an issue where a bunch of content got deleted
15:22:22 <nirik> yeah, saw that.
15:22:34 <nirik> unclear what happened... but also we did delete a bunch of content
15:23:08 <nirik> the f24 alpha/beta/staged stuff
15:23:18 <dgilmore> yeah
15:23:51 <dgilmore> sounded like they may have grabbed the file while it was being updated
15:25:22 <dgilmore> #info Releng is all for getting mirroring smarter
15:26:44 <dgilmore> #topic #6437 Update documentation
15:26:59 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6437
15:27:30 <dgilmore> I think the main thing here was we needed to decide what if how we update the documentation versioning
15:27:42 <dgilmore> there is a bunch of docs that need some love
15:28:31 <dgilmore> I am planning to go over the mass branching one and update it, likely breaking it down to a few docs that cover the different portions of tasks that need to eb done together
15:29:22 <mboddu> but how do you think we can version the docs?
15:29:43 <dgilmore> right now the conf.py for docs has
15:29:44 <dgilmore> # The short X.Y version.
15:29:44 <dgilmore> version = '0.0.1'
15:29:44 <dgilmore> # The full version, including alpha/beta/rc tags.
15:29:44 <dgilmore> release = '0.0.1'
15:29:58 <dgilmore> which really does not make much sense
15:30:23 <dgilmore> though we could use the fedora release version in there
15:30:46 <dgilmore> and increment release whenever we make a change
15:30:55 <dgilmore> or we could just use the date in there
15:30:58 <nirik> yeah, ot just date?
15:31:01 <bochecha_> why not just call it « 1.0 », never bump it, and just version in git?
15:31:02 <mboddu> then, are you planning to change entire documentation to say version 0.0.2 or will you version 0.0.2, the documentation that we changed
15:32:03 <dgilmore> we have never versioned docs in the past at all
15:32:12 <dgilmore> so its something new
15:32:32 <dgilmore> bochecha_: we could
15:32:40 <dgilmore> I am open to suggestions
15:32:56 <dgilmore> on both how to version the docs if we even do
15:33:19 <maxamillion> I don't have any real opinions on that either
15:33:25 <maxamillion> if anything we could just date-stamp it
15:33:25 <dgilmore> I think that versioning may only make sense if we made a pdf
15:33:50 <dgilmore> maxamillion: we could
15:34:06 <bochecha_> or if there were multiple versions (branches) that would be current at one point in time (like stable releases of a piece of software)
15:34:21 <bochecha_> but for something like the releng docs, it kinda feels like « the latest is the right one
15:34:36 <dgilmore> I think with teh html version we produce today its really only useful to make sure that teh rendered docs match commits
15:35:05 <dgilmore> bochecha_: yeah and we only have master branch in git
15:35:07 <maxamillion> I reference the html version periodically actually
15:35:21 <maxamillion> I've also pointed some people at them
15:35:33 <bochecha_> you wouldn't make releases, so when would you bump the version?
15:36:51 <dgilmore> bochecha_: when you commited changes in git
15:37:03 <bochecha_> automatically?
15:37:09 <dgilmore> it would be useful to know that whats committed is what is rendered
15:37:15 <dgilmore> ideally
15:37:20 <bochecha_> then use the commit hash as version?
15:37:22 <dgilmore> or it will be forgotten
15:37:33 <maxamillion> we could do that if we had integrations with readthedocs.org
15:38:03 <maxamillion> so maybe we could go that route ... it wouldn't be hosted on our infra but it is a FOSS stack
15:38:15 <dgilmore> perhaps
15:38:54 <dgilmore> maxamillion: perhaps we can have the shinx build process do something for us
15:39:21 <dgilmore> inject date and ime stamp or git hash at build time
15:39:56 <maxamillion> dgilmore: I've not ever looked at something like that so I'm not sure, but maybe? :)
15:39:57 <dgilmore> #info not sure the best way to version the releng docs
15:40:17 <mboddu> can we do something like x.y.z version number and update z number for every commit automatically and update y when we make lot of changes and update x when we make huge changes in git?
15:40:20 <dgilmore> #info since fedora releng docs, code is a ongoing thing that never has a release
15:41:18 <bochecha_> since the config file is python, you can probably set version to the commit hash with something like `version = subprocess.check_output(['git', 'rev-parse', 'HEAD'])`
15:41:52 <bochecha_> (untested)
15:43:12 <dgilmore> that may be sufficient
15:43:37 <dgilmore> mboddu: do you think that would give you the info you needed?
15:44:18 <mboddu> I think so
15:45:26 <dgilmore> does someone want to test what it will take to implement?
15:47:33 <dgilmore> #action dgilmore to investigate putting the git hash in the docs versioning
15:47:46 <mboddu> dgilmore and I are going through knowledge transfer process, so I might need help from you guys as well, so keep an eye out for my questions
15:48:41 <dgilmore> #topic welcome mboddu
15:49:03 <dgilmore> for those that do not know mboddu started two weeks ago
15:50:05 <bochecha_> started what? :)
15:50:14 <dgilmore> he will be taking over my day to day tasks ass I work on unifying Fedora and RHEL processes and working on how we build new things
15:50:28 <mboddu> dgilmore: thank you very much for welcoming me (again) :)
15:50:32 <dgilmore> bochecha_: at Red Hat in RCM to do fedora release engineering
15:50:48 <bochecha_> ah, great, welcome mboddu
15:51:09 <mboddu> bochecha_: thank you
15:53:04 <dgilmore> I am looking forward to pull requests from mboddu for updating docs where things have gotten stale :)
15:53:23 <dgilmore> #topic Secondary Architectures updates
15:53:24 <dgilmore> #topic Secondary Architectures update - ppc
15:53:26 <mboddu> bochecha_: FYI, we can use git describe for getting version number, subprocess.check_output(["git", "describe"]) (quick google search)
15:53:42 <dgilmore> pbrobinson: how is ppc?
15:53:55 <pbrobinson> not bad
15:54:10 <pbrobinson> close to sync, composing
15:54:40 <pbrobinson> I think I'm mostly ready to rebuild builders to F-24 when we're ready (need to deal with tftp bits still)
15:55:13 <pbrobinson> will have some virt-host rebuilds to do at that point too (two not quite up to spec)
15:55:34 <dgilmore> cool. need to sit down with nirik and come up with a plan to test f24 builders for primary
15:55:55 <nirik> I switched the stg ones last week
15:56:14 <nirik> they seemed to be doing ok with koschei builds
15:56:22 <pbrobinson> nirik: yes, I saw, that's what triggered me to poke too
15:57:09 <nirik> it would be nice to get the prod ones before flock/f25alpha
15:57:17 <pbrobinson> yes
15:57:20 <dgilmore> that would be good
15:58:03 <dgilmore> #topic Secondary Architectures update - s390
15:58:11 <dgilmore> sharkcz: how is s390x?
15:58:43 <sharkcz> looks good, no issues so far, build-wise just hours behind primary
15:59:02 <dgilmore> awesome.
15:59:15 <dgilmore> any testing of f24 on the builders?
15:59:32 <sharkcz> not yet
15:59:32 <dgilmore> sharkcz: any changes planned in the compose process for f25?
16:00:22 <pbrobinson> some
16:00:28 <pbrobinson> we'll be adding docker
16:00:57 <pbrobinson> will rebuild the s390 builders with ansible if we can get the ssh changes to the firewall :)
16:00:57 <sharkcz> and making the process more stable
16:01:12 <pbrobinson> yes, the filesystem stuff
16:02:04 <sharkcz> yep, but it proved that it works with sshfs
16:02:20 <dgilmore> cool
16:02:38 <dgilmore> #info s390x is close build wise
16:02:53 <pbrobinson> yep, so with luck it should settle out nicely soon
16:03:13 <dgilmore> #info looking at docker for f25 and making the builder deployment more like the rest of fedora
16:04:23 <dgilmore> #topic Secondary Architectures update - arm
16:04:36 <dgilmore> pbrobinson: how is aarch64?
16:04:42 <pbrobinson> arm is pretty much up with primary
16:05:04 <dgilmore> #info 16:04 < pbrobinson> arm is pretty much up with primary
16:05:10 <dgilmore> cool
16:05:15 <pbrobinson> I'm working on kernel bits for moonshot to get that into production
16:05:26 <pbrobinson> and also u-boot for ARMv7 virt builders
16:05:36 <dgilmore> #info working to get moonshot chasis into production
16:05:59 <pbrobinson> should have a new upstream u-boot today which I'm going to use to hand over to internal ARM team for some assistance there
16:06:12 <pbrobinson> those two items are this week's primary rel-eng task
16:06:30 <dgilmore> sounds good
16:07:15 <pbrobinson> if we don't quite get there initially on the u-boot I'll work out an initial v7 run book to get devices deployed so we can rebuild to end game later
16:07:27 <dgilmore> #info u-boot for armv7, handing off to internal arm team to simplify 32 bit arm virt builkders on aarch64
16:08:04 <pbrobinson> it'll simplify a lot, not just builders :-D
16:08:15 <dgilmore> indeed
16:09:03 <dgilmore> #topic Secondary Architectures update
16:09:40 <dgilmore> pbrobinson: did you want to give a quick update on the fesco ticket you filed on redefinig secondary?
16:09:54 <pbrobinson> sure
16:10:12 <dgilmore> just to put you on the spot
16:10:50 <pbrobinson> overall FESCo seems OK with it, I need to do a summary to send out to devel@ for wider discussion, I've dug out the asbestos suit and should send that out tomorrow
16:11:21 <bochecha_> any link to that ticket?
16:11:24 <dgilmore> https://fedorahosted.org/fesco/ticket/1592
16:11:32 <bochecha_> thanks
16:11:41 <pbrobinson> the plan is once that's discussed and reviewed is to put in the first of the arch imports as a F-26 change
16:12:44 <pbrobinson> of course any feedback/questions actively welcome, here, directly or on the ticket
16:16:47 <dgilmore> pbrobinson: thanks
16:16:59 <dgilmore> #info feedback questions welcome
16:17:00 <bochecha_> I can see how having builds fail entirely if an « exotic » buildArch failed will be controversial on devel@ :)
16:17:44 <dgilmore> bochecha_: indeed
16:17:56 <dgilmore> #topic Open Floor
16:18:06 <dgilmore> does anyonbe have anything?
16:18:23 <bochecha_> yeah, I was wondering about https://fedorahosted.org/rel-eng/ticket/6111
16:18:47 <bochecha_> any idea when you want to do those cert changes, so we can finally move to sha512 for the lookaside cache?
16:21:24 * maxamillion has something after bochecha_
16:22:08 <dgilmore> bochecha_: no one is working on migrating the ca infra
16:22:39 <bochecha_> do you still want to bundle those changes together?
16:22:48 <dgilmore> bochecha_: patrick is trying to migrate to a token based auth option for koji
16:23:12 <dgilmore> we should perhaps see if he has a timeline for that
16:23:26 <dgilmore> if he doesnt then maybe we just need two flag days
16:24:10 <dgilmore> the CA cert we have expires in 2018
16:24:44 <nirik> yeah, not sure the timeline for the oauth stuff... but perhaps we should just have 2 flag days.
16:25:07 <pbrobinson> might be easier to co-ordinate over all
16:25:22 <pbrobinson> even if not ideal
16:25:59 <bochecha_> well, if the two flag days are quickly one after the other, it's going to be a pain for the build infra users
16:26:19 <nirik> right, so we should see how far out it might be...
16:27:23 <dgilmore> #action dgilmore to talk to patrick on oauth token support for koji and se if he has a timeline
16:27:45 <dgilmore> I think if we have at least 6 months 2 flag days will not be terrible
16:28:07 <dgilmore> if its less than that it will likely get confusing
16:28:27 <nirik> the lookaside changes are pretty much ready right?
16:28:41 <dgilmore> yeah, we need to chnage a config option
16:28:44 <bochecha_> nirik: yes, they've been ready for some time
16:28:51 <nirik> ok
16:29:11 <bochecha_> see here for the detailed status: https://fedorahosted.org/rel-eng/ticket/5846#comment:23
16:29:44 <bochecha_> anyway, that's it for me, I just wanted to ask about it, and I think maxamillion is waiting in line :)
16:29:50 <dgilmore> we can use the wildcard cert for koji.fp.o, it just requires all users update their configs
16:30:17 <nirik> and pkgs?
16:30:24 <maxamillion> I was just wanting to get some eyes on https://pagure.io/releng/pull-request/85 and get it reviewed/merged ... I was hoping to go live with that soon
16:31:15 <dgilmore> maxamillion: my main concern is that its doing work it shouldnt be, but i think that is not your fault
16:31:38 <dgilmore> autocloud should be emitting a message saying a composed passed testing
16:31:58 <maxamillion> dgilmore: yeah, I have filed a RFE with kushal about that
16:32:09 <maxamillion> dgilmore: acarter is tracking it
16:32:38 <maxamillion> dgilmore: so I have high hopes of a resolution ... and then I'll switch over to using that information and remove the ugliness from the script
16:34:38 <dgilmore> maxamillion: perms may be funky
16:34:53 <dgilmore> maxamillion: as we have to run the two week composes as root
16:35:25 <maxamillion> dgilmore: oh
16:35:26 <maxamillion> :/
16:36:15 <pbrobinson> dgilmore: is that because pungi?
16:36:52 <dgilmore> pbrobinson: yeah
16:37:27 <dgilmore> mind you on compose-x86-01 all the filed are owned by nobody:nobody due to nfsv4
16:37:59 <dgilmore> files
16:38:27 <maxamillion> hrmmm
16:38:38 <dgilmore> nirik: think we can make sure we have everything configured correctly so nfsv4 works correctly?
16:38:46 <maxamillion> alright, so maybe I'll do some test runs without sending emails or fedmsgs first
16:39:06 <nirik> odd. I thought that was working right, but sure, we can investigate.
16:39:13 <dgilmore> idmapd.conf is possibly not correctly setup on all the boxes
16:39:14 <nirik> might be root squash ?
16:39:34 <nirik> ok. Can someone file a infra ticket on it?
16:39:36 <dgilmore> its not root squashed
16:39:38 <nirik> or I can in a while
16:39:42 <dgilmore> sure
16:40:05 <dgilmore> I will file a ticket
16:41:32 <dgilmore> anything else?
16:41:40 <dgilmore> or can we wrap up?
16:42:31 <maxamillion> nothing else from me
16:43:09 <dgilmore> #endmeeting