14:30:24 <mboddu> #startmeeting RELENG (2017-04-03)
14:30:24 <zodbot> Meeting started Mon Apr  3 14:30:24 2017 UTC.  The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:30:24 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:30:24 <zodbot> The meeting name has been set to 'releng_(2017-04-03)'
14:30:24 <mboddu> #meetingname releng
14:30:24 <zodbot> The meeting name has been set to 'releng'
14:30:24 <mboddu> #chair dgilmore nirik tyll sharkcz masta pbrobinson pingou puiterwijk maxamillion mboddu Kellin
14:30:24 <zodbot> Current chairs: Kellin dgilmore masta maxamillion mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll
14:30:24 <mboddu> #topic init process
14:30:32 <contyk> o/
14:30:36 * masta is here
14:30:39 <contyk> .hello psabata
14:30:40 <zodbot> contyk: psabata 'Petr Šabata' <psabata@redhat.com>
14:30:44 <jkaluza> .hello jkaluza
14:30:46 <zodbot> jkaluza: jkaluza 'Jan Kaluža' <jkaluza@redhat.com>
14:31:01 * nirik is here mostly
14:31:25 * puiterwijk is here partially
14:32:21 <bowlofeggs> .hello bowlofeggs
14:32:23 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <randy@electronsweatshop.com>
14:32:39 <mboddu> Okay, lets start
14:32:59 <mboddu> #topic #6645 Make sure OSTree repos are sync'ed correctly after the masher completes
14:33:07 <mboddu> #link https://pagure.io/releng/issue/6645
14:34:21 <sharkcz> .hello sharkcz
14:34:22 <zodbot> sharkcz: sharkcz 'Dan Horák' <dan@danny.cz>
14:34:39 <nirik> I'm not sure what there is to discuss here. we need to replace parts that are currently cron jobs with fedmsg based stuff...
14:35:59 <mboddu> nirik: I like your idea. But who will generally work on this stuff?
14:36:34 <nirik> anyone in infra or releng that is willing and has time to do so as far as I am concerned. ;)
14:36:42 <nirik> I'm not sure who that would be or when.
14:38:01 <bowlofeggs> it does seem related to bodhi, but i can't commit to working on that any time reasonably soon
14:38:31 <bowlofeggs> could be something to get an infra apprentice to look at, maybe?
14:38:32 <mboddu> Do we know how pressing is this thing is when it comes to priority?
14:38:49 <bowlofeggs> i don't think anything is really broken about how it is now, it's just inefficient
14:39:12 <nirik> right.
14:39:27 <bowlofeggs> which means it realyl could be a good apprentice task (it's not a pressing matter, we can let it take the time needed)
14:39:32 <nirik> so I would remove meeting keyword from this for now and we can work on it as we can.
14:39:38 <puiterwijk> Except for Atomic host updates for 30 minutes per two weeks
14:40:44 <masta> internally I switched to using a few separate steps to rsync atomic host, to ironically ensure it's atomically copied.
14:40:55 <puiterwijk> Well, every day after the new trees are generated, for up to 30 minutes people can't pull any tree
14:41:03 <masta> seems like the same issue
14:41:42 <puiterwijk> So, reading the ticket, the reason this is a problem for them is that they use the "ref has changed" as trigger to run tests
14:41:57 <maxamillion> .hello maxamillion
14:41:57 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
14:42:00 <maxamillion> sorry I'm later
14:42:05 <maxamillion> late*
14:42:09 <nirik> I... don't think thats true.
14:42:19 <bowlofeggs> puiterwijk: oh, so it's not just inefficent, it's a "30 minute window of errors" for users?
14:42:20 <nirik> it's only when 2 of them finish nearly at the same time I thought?
14:42:34 <puiterwijk> bowlofeggs: for testers, yes
14:42:42 <bowlofeggs> ah
14:42:46 <bowlofeggs> but not end users at least?
14:42:54 <puiterwijk> nirik: no. It happens if the bodhi ostree compose finishes while a sync is in progress
14:43:02 <puiterwijk> bowlofeggs: not anymore.
14:43:26 <masta> puiterwijk, right. But that seems sensible to me, because the ref should be the last bit to copy into place... since the objects can copy-in first, and the refs moving into place is effectively an atomic operation that needs fencing.
14:43:27 <mboddu> I thought its when the 30 min cron job and bodhi clash
14:43:35 <nirik> but that should be a much smaller window...
14:43:45 <bowlofeggs> cool. imo, it could still be a good thing to pitch as an apprentice idea since nobody appears to have time for it
14:43:51 <puiterwijk> nirik: correct. But do this twice everyday, and the odds increase :)
14:44:15 <puiterwijk> masta: yes. The problem occurs when the new tree gets put in place while the script is running
14:44:31 <nirik> I guess it would only be when: rsync starts, atomic compose finishes, rsync stats only part of the atomic compose changes and syncs them.
14:44:38 <puiterwijk> Yep.
14:44:49 <nirik> but thats a pretty small window I would think.
14:45:01 <masta> there are multiple ways to handle this, maybe try --delete-after ?
14:45:17 <nirik> won't help in this case...
14:45:20 <puiterwijk> WEll, it happens if the new tree gets put in place between it syncing the refs and syncing the objects
14:45:34 <puiterwijk> masta: there's no deletes at all... Only more and more new objects..
14:46:02 <nirik> anyhow, we all agree we should move to a fedmsg based one that just syncs what it has seen a new fedmsg for right?
14:46:12 <puiterwijk> nirik: and since the number of objects in the ostree repos is quite big (because of how the stuff works), getting the list of files of objects takes a while
14:46:17 <puiterwijk> Yes
14:46:25 <puiterwijk> At least, I would agree that that's the right way
14:47:00 <nirik> so it's just a matter of getting cycles to implement it. I don't know that we can promise anything right now here.
14:47:01 <puiterwijk> I was just disagreeing with the "it is just an annoyance", since with the current scripts etc it seems to be giving test failures :)
14:47:12 <masta> puiterwijk, fair enough, but what I'm saying is to leverage rsync's .~tmp~ thingy to strive for granular copy... but perhaps I'm not appreciating the race condition aspects
14:47:17 <nirik> ok, so that likely bumps it more in priority
14:48:12 <bowlofeggs> i'm +1 to fedmsging it
14:48:12 <mboddu> nirik: yeah, since there is a chance it might happen every day
14:48:55 <nirik> I wonder... could we just add this to bodhi to do? once it's done, it syncs it? it would have to know compose location and publish location, but...
14:49:06 <nirik> then it doesn't even need a fedmsg.
14:49:34 <puiterwijk> Well, bodhi doesn't have permissions to write to that place as the right user
14:50:01 <nirik> ah true.
14:50:06 <mboddu> puiterwijk: ^^ thats the reason why there is that cron job?
14:50:07 <puiterwijk> But other than that, there's nothing at all preventing us from just getting rid of the /compose tree, and having Bodhi sync directly to and from the final location
14:50:19 <nirik> well, thats bad...
14:50:22 <puiterwijk> mboddu: well, that and the fact that "it's the way we've always done things"
14:50:29 <nirik> because it could fail and leave things broken
14:50:44 <puiterwijk> nirik: well, no. Since it only syncs it out there *after* the full tree is generated...
14:50:45 <nirik> it should compose to a compose place, make sure it's successfull, then sync to published location
14:50:55 <puiterwijk> Yes. It already does that for ostrees
14:51:18 <nirik> well, I was thinking you meant just operate on the published location all the time.
14:51:24 <puiterwijk> the "published location" as far as Bodhi concerned is just in /mash, and then the sync script moves it out of there
14:51:38 * nirik nods, yes, I am aware.
14:52:18 * nirik is done, we need to just replace this script, we will get to it when we do.
14:52:30 <puiterwijk> Agreed.
14:52:51 <puiterwijk> (even though I can't actually vote, not being releng et al :) )
14:53:04 <mboddu> I think the best option here is syncing based on fedmsg then
14:53:47 <mboddu> #info Replace the cron job(https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/bodhi2/backend/files/fedora-updates-push#n52) with fedmsg driven event.
14:54:09 <mboddu> #info Someone from infra or releng will look into it based on the availability.
14:54:26 <mboddu> Okay, lets move on
14:54:45 <mboddu> #topic #6600 Mock and sigined rawhides
14:54:53 <mboddu> #link https://pagure.io/releng/issue/6600
14:55:06 <dustymabe> hey guys, sorry i'm late
14:57:00 <nirik> IMHO there's no perfect solutiion here... we need to hear back from reporter on what we proposed. Nothing will do all they want.
14:57:27 <mboddu> nirik: how about using one gpg key for rawhide?
14:57:42 <mboddu> nirik: which never changes
14:58:25 <puiterwijk> mboddu: I'm -1 to that, since that means that the same key is going to be used for eternity, and if at one point we then decide to cycle it, it's going to cause even more problems
14:58:40 <maxamillion> I'm with puiterwijk on that one
14:58:45 <Kellin> same
14:59:06 <nirik> yeah, we thought of that, but yeah...
14:59:26 <puiterwijk> Right now, we at least practice the key rollover once every 6 months. If it's not a regular thing, it's going to be much bigger deal every time
14:59:30 <masta> I'm not against it. We still have to always setup a new key regardless
14:59:40 <nirik> we could possibly in theory sign everything with 2 keys in the branching period... but I don't know that rpm or dnf handles this well.
15:00:47 <nirik> it might be something to ask for from them tho... as that would also help with mirroring around branch time which last time was... bad.
15:00:48 <mboddu> nirik: that would be the best if rpm and dnf can support it, but I thought rpm already supports it I am not sure about dnf
15:01:19 <puiterwijk> I don't think dnf cares about signatures at all
15:01:26 <puiterwijk> That's all RPM
15:01:46 <nirik> puiterwijk: it does in that it points to a key for that repo... but I guess we could just make it point to the newer one
15:01:56 <puiterwijk> Right
15:02:03 <masta> yep
15:03:26 <nirik> if we can sign with 2 keys, we could sign everything with f28 and f27 a few weeks before branching, then f27 will be all set and f28 will be all set and they can be hardlinked.
15:04:36 <puiterwijk> I think that's just going to require changes to koji... Since currently writeSignedRPM can only write with a single key, I think
15:04:49 <nirik> yeah.
15:04:58 <puiterwijk> But I do think that's a good plan
15:05:33 <nirik> I can file a RFE on koji... no telling when they might be able to look at it tho.
15:06:44 <mboddu> Lets see when koji will be able to get this thing done and I think we can put it on hold until then
15:07:34 * jkaluza would like to discuss Modular Compose in the end of meeting if possible... :)
15:07:50 <mboddu> #info nirik will file a RFE about koji support for multiple gpg keys
15:08:08 * dustymabe would like to discuss separate f26 atomic pungi compose at the end of the meeting if possible
15:08:31 <puiterwijk> Note that you should file a releng ticket with the "meeting" keyword. Or use the open floor.
15:08:52 <dustymabe> puiterwijk: indeed, open floor is the goal
15:09:42 <mboddu> #topic #5911 Rules for private/topic branches in dist-git
15:09:52 <mboddu> #link https://pagure.io/releng/issue/5911
15:10:41 <mboddu> I can see an advantage in doing that, but I am not sure what it takes to make it work
15:12:31 <Kellin> didn't we already discuss this in a previous meeting and determined it was not needed?
15:12:55 <nirik> Kellin: yeah
15:13:25 <nirik> IMHO it could be made to work by enhancing koji to never build from a non standard branch... and then just disallow deleting those...
15:13:52 <nirik> but there was not seen to be much point to private branches anyhow.
15:14:03 <masta> There is work with koji upstream to implement support for builder policy to block private branch builds, should that be a desired policy.
15:14:45 <masta> (presumably for non-scratch builds)
15:15:10 <nirik> there is?
15:15:27 <masta> yes, it's being worked on currently.
15:16:39 <mboddu> masta: I think thats what they are looking for and then we add no deleting of branches, that should do it
15:16:48 <nirik> ok, cool. I had no idea it was on their radar
15:17:13 <bowlofeggs> fwiw, i made a side branch in dist-git once to do some longer running change to a spec file
15:17:25 <bowlofeggs> and i was suprised that i couldn't delete it when done (i now understand why)
15:17:51 <puiterwijk> Just one problem if we'd move to that new policy: what do we do with already-pushed private branches? People might have built from them
15:17:58 <bowlofeggs> most of my spec file changes don't take much time, but occasionally they take a lot of time and i might want to collaborate with others
15:18:42 <bowlofeggs> of course, another option is to tell people they can make a new remote on pagure/gitlab/github/their own server
15:18:52 <masta> puiterwijk, not sure we can re-write history, but good question.
15:19:01 <bowlofeggs> i think that might be the best thing - telling people to just make their own remotes
15:19:06 <nirik> (side note back to the signing topic: I just realized this also will need changes in pungi and possibly bodhi if we still have any multiple signed packages there)
15:19:10 <puiterwijk> masta: pretty sure we shouldn't, since that's tricky :)
15:19:19 <bowlofeggs> it would be nice if dist-git also disallowed pushing non-official branches
15:19:28 <puiterwijk> nirik: well, that depends on how Koji implements multi-key, but sure :)
15:19:34 <bowlofeggs> allowing to push but disallowing to delete i think is more confusing
15:20:06 <puiterwijk> bowlofeggs: note that we are planning t put Pagure in front of dist-git, so people would get forks where they can do that stuff :)
15:20:29 <bowlofeggs> oh right
15:20:36 <bowlofeggs> i'm looking forward to that
15:20:44 <puiterwijk> So, honestly, with that in mind, I'd say we just don't care about private branches
15:20:47 <bowlofeggs> if anything, just having PR review on spec file changes could be nice
15:20:55 <mboddu> puiterwijk: Oh, if pagure is in front of dist-git then we can ask people to use it for collaboration and never worry about dist-git branches
15:20:55 <nirik> yeah, disallowing non standard would be nice too. it should in theory be possible in gitolite
15:20:56 <masta> puiterwijk, aagree
15:21:10 <puiterwijk> nirik: I thought we already did that... :)
15:21:15 <puiterwijk> If not, we should indeed
15:21:25 <masta> pretty soon we wont' have to care about them, if we choose to ignore them in builder policy.
15:21:37 <nirik> puiterwijk: no, pretty sure you can still make em. ;(
15:21:57 <puiterwijk> nirik: ah. Then we should fix the script generating the gitolite policy indeed. But that should be simple enough
15:22:24 * mboddu dont know what is a gitolite
15:22:43 <puiterwijk> mboddu: that's a script we use that implements ACLs for git
15:22:51 <Kellin> mboddu: it's a wrapper that helps control access to a git repository
15:22:57 <puiterwijk> (and yes, that is way oversimplified, so sorry for other people that know the details)
15:23:12 <mboddu> puiterwijk, Kellin : thanks
15:23:40 <mboddu> so, lets put this on hold and I think using pagure is the best option and never worry about private dist-git branches
15:24:11 <puiterwijk> Well, we should make sure to configure gitolite to deny private branch pushes from now on... But otherwise, yep
15:24:32 <mboddu> puiterwijk: yeah, but who has to take that decision?
15:24:39 <masta> mboddu, it would be great to know what builds have sourced from private dist-git branches, just to know.
15:24:42 <puiterwijk> mboddu: releng I'd say..
15:25:11 <puiterwijk> masta: I'm not that that is recorded anywhere in koji. So then we'd need to go down every single build in koji, and check the branches it's in for the distgit commits
15:25:16 <nirik> I'd say lets adjust gitolite and get pagure in place and then close the issue telling them to use forks.
15:25:22 <puiterwijk> I'm not sure that that's*
15:25:40 <masta> probably not hard to script around git's 'branch  --contains=sha' feature
15:25:53 <puiterwijk> Sure. But it'll take a long while to run over every single build
15:26:16 <masta> it would be slow, yea
15:26:34 <masta> but it's trivial, conceptually.
15:27:05 <puiterwijk> RSA with 16k keys is also simple conceptually. :)
15:27:06 <maxamillion> masta: famouse last words :P
15:27:10 <mboddu> #info mboddu will talk to dgilmore about gitolite changes to deny private branch pushes and get pagure in place and ask people to use pagure forks for collaboration
15:27:33 <masta> a variation of the mass rebuild script could do it.
15:27:36 <nirik> hey, there's less than a million builds. ;)
15:28:15 <mboddu> #info Also, look into figuring out how to know what builds have sourced from private dist-git branches
15:28:24 <nirik> BTW, if anyone decides to run such a script, please notify in advance... we don't want to overload koji.
15:28:25 <mboddu> okay, lets move on, we dont have much time
15:28:45 <mboddu> #topic Alternative Architectures updates
15:28:54 <mboddu> sharkcz: quick update please?
15:29:02 <mboddu> nirik: anything from admins about s390x?
15:29:32 <nirik> yes.
15:29:48 <sharkcz> mboddu: build-wise there are no issues, both f26 and rawhide are close to primary
15:29:51 <nirik> I talked with the admin friday, he said he was going to work on it this weekend and let me know today what the status is/was.
15:30:44 <mboddu> sharkcz: thanks
15:30:54 <mboddu> nirik: awesome
15:31:29 <mboddu> #info nirik talked to admin on Friday and he planned to work on it this weekend and will give an update to nirik today
15:31:51 <mboddu> #info alternative arches are all fine
15:31:57 <mboddu> #topic Open Floor
15:32:04 <dustymabe> open floor :)
15:32:20 <jkaluza> who's first? :)
15:32:20 <dustymabe> been trying to get an update on https://pagure.io/releng/issue/6713
15:32:21 <mboddu> jkaluza, dustymabe : sorry guys, but can you guys make it quick
15:32:46 <dustymabe> just trying to get an update there, that's all
15:33:00 <mboddu> dustymabe: sorry, its on dgilmore plate and he is right now traveling
15:33:15 <dustymabe> maybe update the ticket with that info?
15:33:17 <dustymabe> thanks
15:33:23 <dustymabe> jkaluza: all you
15:33:27 <mboddu> dustymabe: sure
15:33:28 <nirik> so this would be like the f25 ones in bodhi...
15:33:35 <mboddu> jkaluza: your turn
15:33:49 <jkaluza> Well, I suppose my question is also for dgilmore then... I was trying to find out wheter the pungi-4.1.14 is already used to build rawhide composes
15:34:09 <jkaluza> Because that contains PRs which I need to start building Modular Compose
15:34:22 <mboddu> nirik: sort of, but atomic compose happens after the release, so its different from f25 in that way
15:34:41 <jkaluza> and after I start with that, we can get rid of all those module-* repos at https://kojipkgs.fedoraproject.org/repos/
15:34:58 <mboddu> jkaluza: I can check what version of pungi is being used on composers
15:34:59 <dustymabe> nirik: right
15:35:18 <puiterwijk> jkaluza: it should still be using the old pungi, given that the infra is still frozen
15:35:20 <puiterwijk> mboddu: ^
15:35:55 <mboddu> puiterwijk: yeah thats right I forgot, thanks
15:35:55 <masta> indeed
15:35:56 <jkaluza> puiterwijk: Is it frozen also for rawhide, is it the same machine generating those composes?
15:36:09 <masta> we done here?
15:36:12 <puiterwijk> jkaluza: the infra is frozen. Aka, the compose boxes themselves.
15:36:21 <jkaluza> puiterwijk: well, ok :)
15:36:24 <puiterwijk> jkaluza: so, that is regardless for what compsoe is being run
15:36:34 <puiterwijk> That freeze lifts the day after Alpha release, so Wednesday
15:36:41 <bowlofeggs> i've got an announcement/proposal: i'd like to deploy bodhi 2.5.0 to prod when the freeze is over
15:36:42 <jkaluza> I will follow-up with dgilmore. All from me.
15:36:54 <mboddu> jkaluza: Freeze will be over on Wed, so it can updated on Wed
15:37:30 <mboddu> Okay, thank you all for joining
15:37:33 <mboddu> #endmeeting