14:33:38 <mboddu> #startmeeting RELENG (2017-04-24)
14:33:38 <zodbot> Meeting started Mon Apr 24 14:33:38 2017 UTC.  The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:33:38 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:33:38 <zodbot> The meeting name has been set to 'releng_(2017-04-24)'
14:33:38 <mboddu> #meetingname releng
14:33:38 <zodbot> The meeting name has been set to 'releng'
14:33:38 <mboddu> #chair dgilmore nirik tyll sharkcz masta pbrobinson pingou puiterwijk maxamillion mboddu Kellin
14:33:38 <mboddu> #topic init process
14:33:38 <zodbot> Current chairs: Kellin dgilmore masta maxamillion mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll
14:34:04 <nirik> morning
14:35:33 <masta> morning
14:36:03 <mboddu> lets give couple more min
14:36:03 <Kellin> .hello kellin
14:36:06 <zodbot> Kellin: kellin 'None' <kellin@retromud.org>
14:36:34 * sharkcz has another meeting running
14:36:49 * pbrobinson o/
14:37:39 <mboddu> Okay, lets start
14:37:55 <mboddu> #topic #6756 "ActionNotAllowed" when attempting "koji untag-build f27"
14:38:03 <mboddu> #link https://pagure.io/releng/issue/6756
14:38:36 <mboddu> So the reason is, autosigning, since autosigning requires autosign permission and not all users dont have that perm to untag their own builds
14:39:22 <mboddu> and we were not able to get to it in time, so that build got shipped in rawhide
14:39:25 <dgilmore> mboddu: thats not at all the reason
14:39:50 <dgilmore> mboddu: the reason the permission is set on the tag is to make sure that only signed builds get tagged in
14:40:05 <nirik> well, it's autosign related anyhow. ;)
14:40:11 <pbrobinson> but tag/untag don't have separate perm tags right
14:40:18 <dgilmore> mboddu: once a build has shipped in rawhide it can not be unshipped
14:40:27 <dgilmore> nvrs are not allowed to go backwards
14:40:58 <dgilmore> nirik: sure, but the details are important
14:41:27 <dgilmore> we may be able to override the perm for untagging with the hub policy
14:41:28 <dgilmore> maybe
14:41:32 <nirik> IMHO, this is just the way it is moving forward for now, and in the future we need to act on untag tickets before the next rawhide.
14:42:11 <dgilmore> nirik: right. we should also probably document how things work, what people need to do and set proper expectations
14:42:27 <nirik> yes.
14:42:34 <mboddu> dgilmore: thanks for the explanation, wrong word usage :)
14:42:37 <nirik> and don asbestos suits and announce it.
14:42:54 <dgilmore> if we had 24/7 coverage I wuld feel better about just saying you need to untag you need to file an issue
14:43:19 <dgilmore> nirik: you think I ever take the asbestos pants off :D
14:43:53 <dgilmore> as is if someone late in the US filed an issue it would never get handled before the compose kicked off
14:44:17 <pbrobinson> and a SLA too
14:45:02 <nirik> well, not sure what the alternative is... opening it could allow someone to mistakenly tag in something not signed or untag something thats already gone out
14:45:37 <nirik> we could add another tag I suppose...
14:45:38 <dgilmore> nirik: we could look at allowing untagging only via hub policy
14:46:04 <dgilmore> or scope out and ask someone to develop a service that people can request untagging from
14:46:05 <nirik> signed -> whatever tag -> rawhide compose moves things from whatever tag to f27 before composing
14:46:15 <dgilmore> that enforces the policy of it being in a compse
14:46:53 <nirik> yeah, but another service: 👎
14:47:06 <dgilmore> nirik: that is a possibility
14:47:16 <mboddu> Cant we allow tagging of unsigned builds and as a part of compose sign the builds and move to a new tag and use that new tag to make a compose?
14:47:53 <dgilmore> mboddu: potentially slowing the compose down by multiple hours
14:48:10 <dgilmore> somne of the bigger packages take hours to sign
14:48:48 <dgilmore> its done the way it is for two reasons
14:49:35 <dgilmore> race conditions between builds finishing, builds being signed and builds being included in teh compose
14:50:10 <dgilmore> if you sign at compose time, once you have done signing you have to check if more builds turned up
14:50:20 <dgilmore> potentially never starting a compose
14:51:01 <dgilmore> anyway we need to have the current state documented and shared out so that people have clear expectations
14:51:05 <mboddu> dgilmore: yeah, that makes sense
14:51:07 <puiterwijk> Well, that would help us get rid of failed composes... No composes => no failed composes
14:51:37 <dgilmore> puiterwijk: sure. does not help get content out there
14:51:47 <puiterwijk> true
14:53:38 <nirik> so, easist would be the koji hub policy... shall we investigate that first?
14:54:09 <dgilmore> nirik: yeah, it will at least get us to a point of self service
14:54:46 <mboddu> nirik, dgilmore : I am not sure what is a koji hub policy, can you point me to some info on it?
14:54:53 <dgilmore> #info current state is that users can not untag builds from rawhide, and need to file a releng issue to untag builds
14:55:01 <nirik> as I recall it's very poorly documented. ;)
14:55:21 <mboddu> nirik: something is better than nothing :)
14:55:23 <dgilmore> #action investigate adjusting hub policy to allow users to untag builds
14:55:50 <nirik> https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/koji_hub/templates/hub.conf.j2#n108
14:55:54 <mboddu> dgilmore: thanks for the notes
14:55:59 <nirik> the policy starts there on line 108
14:56:04 <dgilmore> #info possible other solutions are a microservice to allow users request untagging of builds, that enforces policy.
14:56:10 <dgilmore> mboddu: someone has to
14:56:25 <mboddu> nirik: thanks
14:56:39 <dgilmore> #action document and communicate the current status with devel@
14:57:41 <dgilmore> #info the f27 tag was limited in permsisons when we enabled autosigning to ensure only signed builds make it into rawhide
14:58:48 <dgilmore> #info likely worth filing some RFE's with koji to have finer graied controls
14:59:28 <dgilmore> I think thats covered
14:59:37 <mboddu> dgilmore: thanks
14:59:45 <dgilmore> we do need people to take ownership of the action items
15:00:46 <mboddu> I can take the 2nd action item, document and communicating the current status to devel list
15:01:28 <dgilmore> okay
15:01:39 <dgilmore> mboddu: please also make sure that the issue gets updated
15:01:48 <dgilmore> lets move to the next item
15:01:49 <mboddu> dgilmore: sure
15:01:55 <dgilmore> meeting is half over
15:02:26 <mboddu> #topic #6739 F26 Alpha image paths and CHECKSUM files changed partially from underscore to dash
15:02:29 <nirik> is someone taking the action to investigate the policy?
15:02:32 <mboddu> #link https://pagure.io/releng/issue/6739
15:02:52 <dgilmore> nirik: I will
15:02:59 <nirik> ok, thanks.
15:03:26 <dgilmore> so this needs documenting in the releng sops properlly
15:03:36 <mboddu> The problem here is, we use _ for pre release composes and - for final compose
15:04:05 <dgilmore> the config has to be different in Alpha/Beta to GA
15:04:37 <Kellin> I'll take up the documentation and clarification on this
15:04:40 <dgilmore> we either need to get pungi changed, or we need to properlly document and make sure we have things configured right
15:05:07 <dgilmore> it is a clunkiness of how we are kinda abusing some values
15:05:32 <dgilmore> it may be that there is a different variable we could set in the config that suits our needs better
15:05:42 <mboddu> dgilmore: I think Kellin can use an existing PR https://pagure.io/releng/pull-request/6733
15:07:57 <mboddu> #action Kellin will document the right values to be used for Alpha/Beta configs and for GA config
15:08:03 <mboddu> moving on
15:08:19 <mboddu> #topic #6663 Create another new tag for Modularity
15:08:21 <Kellin> dgilmore: tag me offline in #releng and we can talk about how pungi is being abused to get vars in - will want the history so I grok what's happening
15:08:30 <mboddu> #link https://pagure.io/releng/issue/6663
15:09:23 <dgilmore> there are many issues here when inheritance comes into play
15:09:27 <dgilmore> and you block packages
15:10:38 <dgilmore> basically in the modular world we need a better way to deal with packages
15:11:25 <dgilmore> we likely need to and package lists
15:11:47 <dgilmore> the issue is that mbs was managing it itself.
15:12:04 <dgilmore> and puiterwijk said that having admin perms to do so was a security risk
15:12:14 <dgilmore> and so some onther way had to be found
15:12:56 <dgilmore> I also do not know how far inheritance would go here
15:13:10 <mboddu> dgilmore: quick question, do you think the koji setup we have right now is the best way to handle modularity in long term?
15:13:17 <dgilmore> the bigger the inheritance chain the longer it takes koji to get the list of packages for a package
15:13:31 <dgilmore> mboddu: absoultly its the wrong way
15:13:52 <dgilmore> say we have package foo
15:13:54 <masta> sounds like there would only be inheritance inside the scope of the new tag.
15:13:55 <jkaluza> I'm not sure why do we need that tag
15:14:12 <dgilmore> we add it to the base tag, all is good it can be built in any module
15:14:30 <jkaluza> hm, the admin rights could be the reason
15:14:41 <dgilmore> but doen the road its decided foo is no good and has to go
15:14:49 <dgilmore> you can not just block it in the base tag
15:14:57 <contyk> o hai
15:15:07 <dgilmore> as it has to still exist in teh tags for modules we have shipped and are currently supported
15:15:19 <dgilmore> but you need to make sure that its blocked and not included in new tags
15:15:38 <dgilmore> if all the different modules inherit from teh package list tag
15:15:55 <dgilmore> you then need to make sure its blocked in every new version of the module
15:16:21 <dgilmore> jkaluza: its all about the admin perms needed to add remove packages
15:16:32 <jkaluza> dgilmore: yeah, I see
15:16:55 <dgilmore> I think that RFE's for koji need to be filed so we can figure out how to propelly support modules
15:17:35 <masta> agree,  but we need to articulate the enhancement request.
15:17:44 <dgilmore> it may be that we need two microsevices
15:17:53 <dgilmore> one for doing the module builds
15:18:06 <dgilmore> and another for managing the tags, packages etc
15:18:35 <jkaluza> dgilmore: by managing the tags and packages, you mean creating it for the first one which submits the builds?
15:18:41 <dgilmore> the build service can request new module versions
15:19:03 <masta> seems like a reasonable way to go, and perhaps the micro services might help inform future changes in koji.
15:19:31 <dgilmore> jkaluza: I mean making the tags and targets for a modules, setting up inheritance so it can build from its base modules, and adding appropriate packages to the list to be built
15:19:50 <jkaluza> dgilmore: I like it, +1. I'm only not sure how that helps, you still need admin privs for the manager service
15:20:04 <jkaluza> dgilmore: Sure it will be smaller service with admin rights, so better, but still
15:20:12 <mboddu> dgilmore: ^^ for me it sounds like a new koji with in a koji
15:20:17 <dgilmore> jkaluza: it helps by limiting the scope of what the admin service is coded to do
15:20:23 <jkaluza> dgilmore: yeah, +1 then.
15:20:42 <dgilmore> jkaluza: seperating the privlidged commands from the unprivlidged tasks
15:20:43 <jkaluza> We can certainly split MBS into two services where only one would need admin rights
15:21:06 <dgilmore> it would require two sets of authentication credentials for two seperate users
15:21:23 <jkaluza> I'm now thinking whether the tag creationg for modules could be part of Koji and we would have API for that to call
15:21:38 <dgilmore> jkaluza: perhaps
15:21:45 * puiterwijk notes that I pointed this out during the audit of MBS
15:22:03 <dgilmore> jkaluza: we would have to work with the koji devs to see what they would be willing to accept
15:22:06 <puiterwijk> And that they should have a ticket for this, if I recall correctly
15:22:41 <dgilmore> puiterwijk: yes a issue should be filed in https://pagure.io/koji with RFE's for koji
15:22:57 <puiterwijk> Sure
15:23:44 <jkaluza> dgilmore: I may volunteer to check if the koji issue is there and file one which would do all things we need admin privs for right now
15:23:56 <dgilmore> jkaluza: contyk: can I ask you guys to reach out to koji devs and get things started
15:24:00 <jkaluza> :)
15:24:29 <contyk> since I came a bit late, what is it you want changed in koji?
15:24:45 <jkaluza> dgilmore: I would ask koji devs first for theirs opinions and if things won't get right there, we could at least split MBS as you described.
15:25:35 * contyk needs to add this meeting to his calendar
15:26:07 <dgilmore> contyk: well find out what parts of what mbs needs that needs admin perms today we could either incorporate with different perms etc
15:26:10 <jkaluza> contyk: MBS is now using koji as admin, because it has to setup the tags and inheritance for a module build. The goal is to be able to run MBS as a normal user.
15:26:35 <contyk> ok, I see
15:26:47 <jkaluza> contyk: so probably nothing you are interested in personally :)
15:26:57 <dgilmore> either split up mbs, or move some pieces into koji
15:27:10 <contyk> so you want a role that has permissions to create tags, targets, add packages... and handle secure-boot stuff
15:27:14 <contyk> while still being a normal user
15:27:19 <dgilmore> eitehr way the goal is figure out how to drop the perms needed to manage modules
15:27:24 <jkaluza> dgilmore: yeah, as I said, I'm OK to reach koji devs and start the discussion
15:27:49 <contyk> +1 to jkaluza doing everything
15:27:56 <dgilmore> contyk: :D
15:28:09 <jkaluza> dgilmore: is this weekly meeting?
15:28:40 <dgilmore> #action jkaluza to reach out to koji devs to get the conversation started on figuing out how to drop the permissions of module builds and management
15:28:45 <dgilmore> jkaluza: it is
15:29:00 <dgilmore> jkaluza: we discuss things going on in releng land in this meeting
15:29:25 <mboddu> jkaluza: its every Monday 9:30 AM CST
15:29:28 <dgilmore> the agenda is driven by issues in the releng pagure instance with the meeting keyword
15:29:43 <jkaluza> While I'm here, I plan to deploy new MBS next week which will stop doing those repos you don't like on kojipkgs
15:29:50 <dgilmore> jkaluza: the meeting is always US central time as thats where I am based
15:30:02 <dgilmore> jkaluza: okay :)
15:30:56 <dgilmore> jkaluza: infrastructure feezes next tueday for Beta
15:31:32 <jkaluza> dgilmore: that's also for pungi-fedora, right?
15:31:48 <dgilmore> jkaluza: so any changes after next monday need explicit freeze break requests
15:31:51 <dgilmore> jkaluza: yes
15:31:55 <jkaluza> ok, good to know
15:32:18 <dgilmore> jkaluza: though the processes for infra changes and pungi-fedora changes are very different
15:32:57 <mboddu> dgilmore, jkaluza : anything else on this topic?
15:33:06 <dgilmore> nope
15:33:17 <mboddu> dgilmore: thanks
15:33:23 <jkaluza> ok, you can continue with your normal agenda, I don't want to waste others time here
15:33:34 <nirik> dgilmore: incorrect
15:33:44 <dgilmore> nirik: which bit?
15:33:45 <nirik> we have slipped... so we freeze on 2017-05-16
15:33:59 * jkaluza likes that, if true
15:34:15 <dgilmore> nirik: gahh miss read the schedule with all the dashes
15:34:29 <dgilmore> translation deadline is the 2nd
15:34:29 <nirik> yeah... slip sliding away. ;)
15:34:41 <dgilmore> its right at the Beta Freeze line
15:34:47 <dgilmore> https://fedoraproject.org/wiki/Releases/26/Schedule
15:35:19 <dgilmore> jkaluza: ignore me. I am ahead of myself
15:36:12 <mboddu> Okay, we actually went beyond the scheduled time.
15:36:38 <mboddu> And we haven't covered all the topics as usual :)
15:36:45 <dgilmore> mboddu: I know pbrobinson had an open floor item
15:36:54 <mboddu> Any quick updates anyone?
15:37:02 <dgilmore> #topic Open Floor
15:37:07 * nirik has 2 (hopefully quick items) can go after pbrobinson
15:37:26 <dgilmore> nirik: how about you go first
15:37:50 <pbrobinson> wfm
15:38:05 <nirik> well, one of them I was going to ask pbrobinson about. ;) So... the new armv7 buildvm's seem to be working well yes?
15:38:10 <pbrobinson> so the ARMv7 virt builders have been live in prod for nearly a week, there's 24 of them. Not seen any issues, just wanted to highlight they were there
15:38:25 <pbrobinson> nirik: I think we're on the same page ;-)
15:38:39 <nirik> so, when can we disable the old ones? Or is there any reason to keep them on?
15:38:45 <pbrobinson> they've been exclusive there on kernel and eclipse channel.
15:38:51 <pbrobinson> there's 24 of them
15:39:04 <pbrobinson> I've not tested them yet on the image build side of things
15:39:36 <pbrobinson> I wanted to get nirik's and other people's general feedback, and whether it's worth removing the old ones from the general build channel
15:39:55 <dgilmore> maybe we should disable the old ones for a week
15:40:04 <nirik> yeah, works for me. If you want to do some more testing first thats fine, but hopefully they can do everything.
15:40:15 <dgilmore> see how things go, and remove them from active service if things are okay
15:40:21 <nirik> which brings up my second related item... I want to do a mass update/reboot cycle next week.
15:40:35 <nirik> if we can power off the old arm builders that would save updating them. :)
15:40:53 <dgilmore> nirik: sure.
15:40:54 <pbrobinson> well the new ones have been live since last Tues, so that's nearly a week
15:40:58 <pbrobinson> I think that works
15:41:00 <dgilmore> so lets disable the old ones today
15:41:08 * dgilmore can handle that
15:41:13 <nirik> +1 from me. Just need to make sure they can do images.
15:41:25 <pbrobinson> I was going to get a koji cmd from dgilmore so I can test arm image builds in stage too
15:41:33 <masta> I'd say maybe change the load/capacity settings of the old ones, so they are less likely to take new tasks?
15:41:39 <nirik> dgilmore: also, can we disable that one at your house now? :)
15:41:47 <pbrobinson> masta: that means they still need to be updated
15:41:53 <masta> right
15:42:01 <dgilmore> nirik: nope :(
15:42:11 <nirik> bummer. It needs virt?
15:42:14 <pbrobinson> nirik: no, that's going to need a little more work for ARMv7 on aarch64 docker builds
15:42:16 <pbrobinson> on my list
15:42:18 <dgilmore> nirik: we need to be able to run imagefactory to do that
15:42:23 <nirik> ok
15:42:31 <dgilmore> which is different to running appliance-creator
15:42:52 <pbrobinson> I don't think that should take too much work, I suspect just a couple of bugfixes
15:43:01 <nirik> well, I am in favor of disabling the old ones today and seeing how it goes. If all is well we can power them off next week and leave them around for a few months in case we need em
15:43:16 <dgilmore> we do need to put some of the new machines in the compose channel
15:43:35 <nirik> we might make a short announcement tomorrow or whenever that they are there...
15:43:39 <pbrobinson> dgilmore: I think there's at least 3 there?
15:44:01 <pbrobinson> or maybe I removed them when I enabled them just to be sure they were stable first
15:44:04 <dgilmore> pbrobinson: there are 0
15:44:05 <nirik> also, if we wanted we could kill the eclipse and gcc channels.
15:44:05 <pbrobinson> that's on my list
15:44:10 <dgilmore> https://koji.fedoraproject.org/koji/channelinfo?channelID=9
15:44:29 <pbrobinson> nirik: I was going to do that once we had pure VMs in the standard channel :)
15:44:35 <dgilmore> we could kill the eclipse and gcc channel
15:44:43 <nirik> simplify things. ;)
15:45:11 <nirik> anyhow, thats all I had. Happy to help with any of that as needed.
15:45:39 <dgilmore> cool
15:45:41 <pbrobinson> nirik: I'll co-ord with you post meeting
15:45:51 <nirik> sounds good.
15:46:06 * nirik notes the old arm builders went out of warentee a bit over a year ago
15:46:24 <dgilmore> oops
15:46:31 <mboddu> #info ARMv7 virt builders have been live in prod for nearly a week
15:46:45 <nirik> well, calexida no longer exists either. :)
15:46:53 <mboddu> #info Old arm builders will be disabled today
15:47:50 <mboddu> Anything else?
15:48:06 * nirik has nothing.
15:48:42 <mboddu> Thank you all for joining
15:48:48 <mboddu> #endmeeting