14:31:17 <dgilmore> #startmeeting RELENG (2016-10-03)
14:31:17 <zodbot> Meeting started Mon Oct  3 14:31:17 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:31:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:31:17 <zodbot> The meeting name has been set to 'releng_(2016-10-03)'
14:31:17 <dgilmore> #meetingname releng
14:31:17 <zodbot> The meeting name has been set to 'releng'
14:31:17 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion mboddu
14:31:17 <zodbot> Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll
14:31:20 <dgilmore> #topic init process
14:31:21 <dgilmore> hi all
14:31:49 <nirik> morning
14:31:51 <maxamillion> .hello maxamillion
14:31:52 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
14:32:05 <mboddu> morning everyone
14:33:08 <dgilmore> guess we have a small crew today
14:33:20 <masta> hello
14:33:22 * sharkcz is here
14:33:36 <dgilmore> I wanted to go over a few things really quickly
14:33:57 <dgilmore> #topic Atomic Host CD
14:34:58 <dgilmore> On the weekend i put together a script to continuosly rebuild atomic host from teh rawhide buildroot
14:35:01 <dgilmore> https://pagure.io/releng/blob/master/f/scripts/build-test-ostree
14:35:19 <dgilmore> I sent in some FBR requests to enable having it run every 15 minutes
14:35:57 <dgilmore> it is related to https://fedorahosted.org/rel-eng/ticket/6350
14:36:32 <dgilmore> #info Atomic Host is being rebuilt continuosly from teh rawhide buildroot to enable rapid testing
14:36:37 <nirik> dgilmore: I do see that it's kinda chatty sending to releng-cron, we may want to adjust that.
14:36:45 <dgilmore> nirik: yeah
14:36:55 <maxamillion> dgilmore: ah cool
14:36:56 <maxamillion> +1
14:37:01 <dgilmore> nirik: really need to get it just logging everything
14:37:20 <dgilmore> it is building ostrees for x86_64, aarch64 and armhfp
14:37:37 <dgilmore> however we seem to be hitting a weird issue on armhfp
14:37:44 <masta> neat script
14:37:55 <dgilmore> rpm-ostree tries to allocate 2GiB ram and fails sometimes
14:38:09 <dgilmore> masta: some is hackish
14:38:17 <dgilmore> we should not need to tweak the package set
14:38:54 <masta> dgilmore, yeah that is true
14:38:59 <dgilmore> hopefully with the outcome of the meeting i plan to call later in the week it will enable it to run quicker
14:39:16 <dgilmore> but anyway I hope it is useful
14:39:40 <dgilmore> I am planning to setup a armhfp box and a aarch64 vm to enable testing and trying it out
14:40:11 <dgilmore> #topic comps and dnf
14:40:11 <masta> I think the pruning can be a separate script, but anyhoo...
14:40:46 <masta> does dnf even do comps?
14:40:48 <dgilmore> https://bugzilla.redhat.com/show_bug.cgi?id=1380945
14:40:54 <dgilmore> masta: it does
14:41:14 <dgilmore> dnf enabled strict group listings again
14:41:35 <dgilmore> comps does not support saying that a package is only for some arches
14:41:54 <dgilmore> there was a discussion about it in #fedora-releng this morning
14:42:06 <masta> wait, what?
14:42:14 <dgilmore> Comps needs to be properlly documented, currently very few people understand it fully
14:42:20 <masta> oh! it does, but only on package groups, right?
14:42:21 <maxamillion> +1
14:42:24 * nirik would say no one. ;)
14:42:24 <dgilmore> then a  change needs to be proposed
14:42:31 <maxamillion> there's definitely things about comps I don't fully understand
14:42:34 <dgilmore> and it needs commuincation
14:42:39 <maxamillion> and I feel as though I probably should :)
14:42:47 <dgilmore> as it will cause pain for users relying on the current behaviour
14:43:13 <dgilmore> nirik: notting is about teh only one that I would say fully does
14:43:14 <nirik> dgilmore: can you update the bug based on whatever discussion? The bug makes it sound like comps does support that
14:43:18 <dgilmore> I mostly do I think
14:43:25 <dgilmore> and I could be wrong on that
14:43:47 <masta> so I could have sworn comps does have a parameter for package arch
14:43:51 <dgilmore> nirik: yeah. the dnf guys did not understand it
14:43:55 * masta goes to look
14:44:15 <dgilmore> masta: there is some use internally, but its only for compose tooling used in rhel
14:44:25 <dgilmore> its not supported in yum
14:44:39 <dgilmore> and its not supported by the schema
14:45:18 <masta> <packagereq arch="foo">package</packagereq>
14:46:19 <dgilmore> masta: right its not actually part of comps though
14:46:30 <dgilmore> its something people internally do for compose reasons
14:46:38 <dgilmore> it does not pass schema validation
14:47:01 <masta> dgilmore, ah, that is interesting. It sucks, but interesting.
14:47:26 <nirik> I think nim-nim wrote the validation stuff... but he's not been around in a long time either
14:48:19 <masta> so do we need xml scheme gurus to take a look at this?
14:48:36 <dgilmore> well we need to document everything we have
14:48:55 <dgilmore> if someone wants to work with the dnf guys to do so that would be awesome
14:49:16 <masta> ok, document what is factually true. Maybe conceive what we want, beyond that.
14:50:31 <dgilmore> masta: right
14:50:42 <dgilmore> I am not opposed to making the change dnf wants to make
14:51:00 <dgilmore> but shoving it in a back door with no documentation will upset people
14:51:22 <maxamillion> +1
14:51:26 <dgilmore> I can see people out in the wild relying on the behaviour
14:51:46 <dgilmore> they have a repo with all sorts of stuff and a home brewed comps
14:51:57 <nirik> there is one other option thats a middle of the road thing...
14:52:00 <masta> yeah, I can see that too
14:52:06 <nirik> but would possibly upset users.
14:52:51 <nirik> keep dnf group install strict, but adjust lorax/anaconda/appliance-creator/etc to pass 'strict=false' for installs only.
14:53:23 <maxamillion> dgilmore: doesn't rpmfusion and other third party repos offer comps too? would any of this impact users of those?
14:53:26 <masta> Any Anaconda folks about?
14:54:00 <dgilmore> maxamillion: yep
14:54:10 <dgilmore> masta: doubt it
14:54:22 <masta> maxamillion, it would help them for sure. Having an updated scheme doc can only be good thing.
14:54:55 <maxamillion> masta: +1
14:55:05 <dgilmore> I may need to set up a call to make sure comps is documented
14:55:21 <masta> nirik, I appreciate the middle ground idea, always good to have options like that
14:55:51 <masta> yeah, comps is really important, I'd like to be on that call dgilmore
14:56:08 <nirik> just another option, sure. Would also be more work and likely to annoy users, so I think just turning it off for now would be best...
14:56:25 <masta> at least I say that now, cringe!
14:58:20 <dgilmore> #info work needs to be done to fully document comps
14:59:36 <dgilmore> #info any changes will need to be well communicated
14:59:52 <dgilmore> another related bug is https://bugzilla.redhat.com/show_bug.cgi?id=1369129
15:00:16 <dgilmore> #topic #6482 fix bodhi buildroot overrides with secure-boot packages
15:00:23 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6482
15:01:06 <nirik> oh yeah.. this.
15:01:35 <nirik> dgilmore: you mentioned wanting to remove admin from bodhi user... what will it take to do that?
15:01:44 <dgilmore> I think to fix this we make a adjustment ot the policy allowing bodhi to tag the builds
15:01:54 <dgilmore> nirik: not 100% sure
15:02:09 <dgilmore> nirik: last I knew bodhi was using --force when taging packages
15:02:19 <nirik> I think thats gone now, but not sure.
15:02:27 <nirik> perhaps we fiile a issue for bowlofeggs to look at?
15:02:35 <dgilmore> if its not doing it we could probably remove it from admin
15:02:44 <dgilmore> and grant extra perms if needed
15:02:51 <dgilmore> nirik: yeah
15:03:11 <dgilmore> I would rather not give bodhi secure-boot perm
15:03:32 <bowlofeggs> seems fine to me - these are just bodhi's permissions in koji we're talking about?
15:03:52 <nirik> bowlofeggs: yeah. right now it cannot do overrides for things in the 'secure-boot' channel.
15:04:05 <nirik> because the bodhi user isn't in that permission
15:04:35 <dgilmore> https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/koji_hub/templates/hub.conf.j2#n110
15:04:39 <bowlofeggs> but it sounds like dgilmore also doesn't want it to do that?
15:05:01 <masta> so long as the koji admin permissions are broken out enough, yeah
15:05:17 <dgilmore> https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/koji_hub/templates/hub.conf.j2#n98
15:05:17 <nirik> ideally we would reduce bodhi user permissions to just the bare minimum it needs to do it's work...
15:05:28 <dgilmore> nirik: indeed
15:05:29 <nirik> (in case someone compromised it and got access to it's user)
15:05:30 <bowlofeggs> yeah that makes sense to me
15:05:51 <masta> we all agree on that, yeah
15:06:15 <dgilmore> we can set in the policy above "has_perm secure-boot && package kernel shim grub2 fedora-release fedora-repos pesign :: allow" granting bodhi permission to tag the builds
15:06:37 <nirik> is_user bodhi && package ...
15:06:40 <nirik> or something
15:06:46 <dgilmore> nirik: uep
15:06:46 <masta> wow.. that is cool
15:07:06 <nirik> bowlofeggs: bodhi used to use --force with it's koji operations (taking advantage of being an admin), but I think bodhi2 cleaned a lot of that up.
15:07:15 <nirik> it may just not need admin anymore.
15:07:19 * nirik isn't sure
15:07:20 <dgilmore> if is_user does not work, we could set up a bodhi perm
15:07:23 <dgilmore> then add that
15:07:39 <dgilmore> nirik: indeed
15:07:51 <dgilmore> we should look through the bodhi code
15:07:59 <masta> uh... but then we have to kinda hardcode the secure-boot package set in code, right?
15:07:59 <dgilmore> and likely just remove admin
15:08:11 <dgilmore> masta: its hardcoded in the policy
15:08:29 <dgilmore> we add remove a package we need to do it in all the places
15:08:36 <dgilmore> it is contained to one file
15:08:46 <dgilmore> it is one config file
15:09:01 <masta> that kinda sucks in other ways, configuration creep, but it's mild and I'd be fine with it.
15:09:35 <masta> that is what ansible is for.
15:12:57 <dgilmore> masta: well it is on the hubs
15:13:09 <dgilmore> it is also how koji defines the policy
15:13:26 <dgilmore> another option would be making koji use the db for configuration
15:13:33 <dgilmore> but that would be a massive koji change
15:14:03 <nirik> I prefer a config file to db
15:14:10 <dgilmore> #action dgilmore to ensure that the koji hub policy is changed so bodhi can tag the packages in secure-boot
15:14:12 <nirik> much easier to automate and make sure it's the way you want it
15:14:18 <dgilmore> nirik: sure
15:14:32 <dgilmore> I do not see koji making that change
15:15:07 <dgilmore> #topic Alternative Architectures updates
15:15:16 <dgilmore> sharkcz: how is everything?
15:15:33 <sharkcz> afaik all is good
15:16:04 <dgilmore> #info all okay
15:16:30 <dgilmore> #info atomic host enabled in rawhide for aarch64
15:16:46 <dgilmore> I enabled in rawhide making atomic host for aarch64
15:16:50 <dgilmore> there is one issue
15:17:03 <sharkcz> pbrobinson plans to enable it for ppc64le soon
15:17:05 <dgilmore> the hardcoded package set specifies grub2
15:17:08 <masta> dgilmore, that is great
15:17:24 <masta> dgilmore, yeah, noticed that in your script from before. +(
15:17:29 <dgilmore> grub2 is not available on aarch64 as there is no bios support
15:17:44 <masta> hum...
15:18:03 <dgilmore> I sent an email to the ostree-devel list awhile back about supporting multiple arches
15:18:42 <masta> dgilmore, I plan to use the aarch64 ostree, thanks!
15:19:32 <dgilmore> https://github.com/cockpit-project/cockpit/pull/5110
15:19:37 <dgilmore> I sent that to cockpit
15:19:55 <dgilmore> and given the pull request only changed the spec file I think their CI is broken
15:21:39 <dgilmore> anyway
15:21:44 <masta> yeah
15:21:57 <dgilmore> #topic beta readyness
15:22:11 <dgilmore> not sure I spelt that right
15:22:18 <dgilmore> Go no go is Thursday
15:22:29 <dgilmore> https://qa.fedoraproject.org/blockerbugs/milestone/25/beta/buglist
15:22:48 <dgilmore> there is 3 accepted blockers, 3 proposed blockers
15:23:17 <dgilmore> I think that  https://bugzilla.redhat.com/show_bug.cgi?id=1381045 will be rejected
15:23:23 <dgilmore> the X bump will not be in Beta
15:23:40 <dgilmore> the other 2 proposed blockers have patches posted
15:23:49 * nirik has been running it here, seems ok, but thats only 1 device
15:24:12 <dgilmore> adamw: any wisdom on when we will get a compose request?
15:24:19 <dgilmore> we really need it today
15:24:25 <masta> hehe
15:24:40 <dgilmore> that would give tuesday and wednesday for testing
15:24:56 <dgilmore> if we do not get the compose request by tomorrow we will have to slip
15:25:15 <masta> I figured they test the nightlies
15:25:25 <nirik> we will see what today brings us.
15:25:32 <dgilmore> there is an outstanding request to the workstation WG on ostree workstation
15:25:55 <dgilmore> https://lists.fedoraproject.org/archives/list/desktop@lists.fedoraproject.org/thread/HSNUVGAR5R7WHGZBS5N6D7WE4SUWJZ46/
15:26:16 <dgilmore> hopefully they can get that resolved so we can ship as part of Beta
15:27:44 <dgilmore> anyone aware of anything else holding up Beta?
15:28:36 <dgilmore> #topic open floor
15:28:46 <dgilmore> does anyone want to bring up anything?
15:29:08 <maxamillion> not I
15:29:29 <nirik> I have one item
15:29:32 <nirik> hopefully quick
15:29:48 <nirik> We are close to bringing the new buildhw hw online
15:29:55 <dgilmore> I have one quick thing, it is pretty widely announced by now but pbrobinson is moving toa  new role, which will interact with releng a lot still. however we likely will need to replace him from the push rotation, unless he chooses to remain in the rotation
15:30:01 <dgilmore> nirik: awesome
15:30:04 <nirik> Do we still need buildhw builders? or can we do everything in vm's now?
15:30:19 <dgilmore> nirik: we need to test nested virt
15:30:41 <nirik> yeah. if that works perhaps we don't need buildhw builders...
15:30:42 <dgilmore> I think we did some minimal testing in stg koji
15:30:50 <nirik> stg should be setup for it...
15:31:02 <nirik> anyhow, something to think about before we add them.
15:31:11 <dgilmore> we should probably test livemedia also
15:31:20 <dgilmore> as that worked in stg but was really slow
15:31:39 <masta> dgilmore, any candidates to replace pbrobinson ?
15:31:47 <nirik> also, we have 16 of them and only 11 of the old buildhw's... so we have extras. Smooge was thinking we use some for autocloud testers... and I don't know if there were other needs, but we could use them for those as well if well defined.
15:32:06 <dgilmore> masta: maybe maxamillion if he wants
15:32:20 <maxamillion> I'll need some training, but that'd be great
15:32:22 <masta> dgilmore, I ask, because I'm interested myself... but I'd have to coordinated that with my internal obligations.
15:32:34 <dgilmore> masta: you are already in the rotation
15:32:49 <nirik> oh, a note on pushes too for everyone:
15:32:55 <masta> oh, my bad... this is for push duty
15:33:15 <masta> yeah, that is so easy it's silly
15:33:17 <maxamillion> dgilmore: yeah, let's pencil me in for push duty pending someone can teach me (or are there docs now?)
15:33:21 <nirik> autosign should take care of almost all the signing, but you should still sign just in case. There shouldn't be a need to remove repodata dirs or I think restart the fedmsg hub anymore.
15:33:23 <masta> maxamillion, you need to get on this dude.
15:33:27 <maxamillion> masta: rgr that
15:33:34 <maxamillion> I'm in, let's do it
15:33:35 <dgilmore> maxamillion: there is docs and some recent changes
15:33:40 <maxamillion> dgilmore: rocking
15:33:51 <mboddu> maxamillion: I updated the docs: https://docs.pagure.org/releng/sop_pushing_updates.html
15:33:58 <maxamillion> mboddu: yes!!!! :D
15:34:01 <maxamillion> mboddu++
15:34:03 <zodbot> maxamillion: Karma for mohanboddu changed to 2 (for the f24 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:34:04 <dgilmore> nirik: yeah, mboddu said he did not need to sign anything last week
15:34:21 <nirik> yeah. it's keeping up nicely.
15:34:22 <maxamillion> what does that mean?
15:34:32 <masta> maxamillion, we can meet some place locally, and I'll show you how personally
15:34:43 <dgilmore> maxamillion: puiterwijk's auto signing is working
15:34:54 <nirik> dgilmore: did the fedpkg update with the changes for gating go out? did we want to decide on a cutover date for that?
15:35:04 <dgilmore> nirik: nope
15:35:05 <mboddu> maxamillion: you can let me know if you need anything else, and it also consists of some of the commonly happening issues and how to resolve them
15:35:20 <nirik> I'd really love to finish that
15:35:23 <maxamillion> mboddu: +1
15:35:34 <maxamillion> dgilmore: ah, swank
15:35:58 <dgilmore> nirik: https://pagure.io/fedpkg/c/293dc31427d55961222534ae26e8a97aed2d77a1?branch=master it was merged but just after 1.25 was cut and released
15:36:02 <masta> maxamillion, yeah, I'm all with mboddu .. we can help you get on board signing & pushing
15:36:35 <nirik> we should push a 1.26 out. ;)
15:36:51 <mboddu> maxamillion: before we need to sign the packages before we make the push, we have a script for it, puiterwijk auto signing eliminated that, but we are still running the script for now just in case if any package is not signed
15:37:02 <dgilmore> nirik: I will bug the guys doing the upstream stuff to cut a release
15:37:04 <maxamillion> mboddu: ahhh ok
15:37:35 <nirik> dgilmore: and do we want to tie this to the checksum thing? basically thats just a switchover to new sometime...
15:37:37 <dgilmore> lets get back to nirik's questions on the new builders
15:37:45 <dgilmore> nirik: yeah
15:37:46 <nirik> yeah, sorry, drifting.
15:38:11 * nirik had nothing more, we can test if we want to make them buildhw's or larger buildvms or something.
15:38:13 <dgilmore> nirik: no problems having one or two of the boxes used for automated testing
15:38:39 <dgilmore> nirik: I guess start with buildvms
15:38:46 <dgilmore> then we can go back if we have issues
15:39:04 <nirik> ok, in any case we wont do anything until after beta is out
15:39:16 <dgilmore> yep
15:39:32 <dgilmore> nirik: we should add it as a issue for next week
15:39:45 <dgilmore> to follow up with something concrete
15:39:48 <nirik> ok
15:39:56 <dgilmore> which reminds me
15:40:12 <dgilmore> what is the status of fedorahosted to pagure issue migration?
15:40:32 <masta> is it too early to talk about risc-v ?
15:40:56 <nirik> I need to check on that fedmsg for tickets thing...
15:42:24 <dgilmore> masta: I think we should be in discussions with them on what they need to do
15:42:33 <masta> can we do qemu koji builders?
15:42:37 <dgilmore> masta: but it is way to early talking about it being official
15:43:03 <dgilmore> masta: we could, right now they are using forks of core components
15:44:11 <masta> dgilmore, ok, it's just a feeler discussion. Hopefully things improve with the port soon
15:44:22 <dgilmore> nirik: is there a issue i can add myself to?
15:44:56 <dgilmore> hopefully in the next month we will bring ppc64 and ppc64le into koji.fp.o for rawhide
15:44:59 <nirik> yeah, but need to find it.
15:45:01 * nirik looks
15:47:24 <masta> dgilmore, that would be nice from the perspective of signing
15:48:49 <masta> dgilmore, do we have any power9 plans?
15:49:12 <dgilmore> masta: afaik it should all work just fine
15:49:38 <dgilmore> gcc is new enough
15:49:54 <sharkcz> yep, and real power9 hw is still 1 year ahead
15:49:56 <dgilmore> assuming that the kernel has the power9 config options enabled it should all just work
15:50:02 <sharkcz> ack
15:50:07 <dgilmore> Ia ssume IBM will test and report/fix as need be
15:50:36 <dgilmore> I do not think fedora needs a special power9 plan
15:50:48 <masta> good
15:51:16 <dgilmore> if there is nothing else I will wrap up
15:51:27 <dgilmore> #info no special power9 plan needed
15:51:44 <masta> wrap it
15:51:46 <dgilmore> #info will work with riscv team to ensure they do things correctly
15:52:05 * masta goes off
15:52:15 <dgilmore> #action nirik to get dgilmore ticket info fo fedmsg bug on fedorahosted to pagure issue migration
15:52:25 <dgilmore> #endmeeting