14:35:13 <dgilmore> #startmeeting RELENG (2016-09-26)
14:35:13 <zodbot> Meeting started Mon Sep 26 14:35:13 2016 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:35:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:35:13 <zodbot> The meeting name has been set to 'releng_(2016-09-26)'
14:35:13 <dgilmore> #meetingname releng
14:35:13 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion mboddu
14:35:13 <zodbot> The meeting name has been set to 'releng'
14:35:13 <zodbot> Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll
14:35:16 <dgilmore> #topic init process
14:35:38 * sharkcz is here
14:35:40 <masta> hello
14:36:03 * nirik is here
14:37:13 <mboddu> .hello mohanboddu
14:37:14 <zodbot> mboddu: mohanboddu 'Mohan Boddu' <mboddu@bhujji.com>
14:38:06 <puiterwijk> Hi
14:38:55 <dgilmore> I doubt pbrobinson will be here today and I know maxamillion will not be
14:39:01 * nirik runs to get coffee, back in a min
14:39:01 <dgilmore> lets get started
14:39:11 <dgilmore> #topic Communications
14:39:34 <dgilmore> So we had issues a few weeks back with pushing updated ostrees
14:39:55 <dgilmore> the person on push duty worked with ostree folks to resolve the issue
14:40:12 <dgilmore> however it was done in a unsupportable and uncommunicated way
14:40:31 <dgilmore> it also broke the ability to do ostrees as part of rawhide or branched composes
14:41:00 <dgilmore> we 100% have to communicate when changing things
14:41:32 <dgilmore> it would not have taken much to realise it was not suitable
14:42:04 <dgilmore> it also highlighted that the way we are doing ostree is disjoint and not really great yet again
14:42:21 <dgilmore> so we need a longer term plan to redo how we make ostrees
14:42:25 <dgilmore> and how we manage them
14:43:03 <puiterwijk> Do we have a documentation of the current ways we do this?
14:43:27 <dgilmore> puiterwijk: not really
14:43:37 <dgilmore> bodhi does it one way and pungi another
14:43:55 <nirik> could we move to some way in koji that both fire off and use?
14:43:58 <dgilmore> the dijointness was due to under resourcing and having to shove things in at the last minute
14:44:07 <puiterwijk> Yeah, I noticed that. I think we should have a single codebase for both... maybe it would be an idea to let bodhi call pungi with a minified config?
14:44:08 <dgilmore> nirik: that needs to be the plan
14:44:27 <dgilmore> nirik: how its done in bodhi will fail as soon as we add any extra arches
14:44:51 <dgilmore> it only works today as the bodhi box is x86_64 and so is the repo
14:45:00 <nirik> fun.
14:45:13 <puiterwijk> dgilmore: yeah, agreed. So, would it be an idea to have bodhi call pungi's ostree phase code?
14:45:17 <dgilmore> puiterwijk: that could be one way to do it
14:45:35 <mboddu> cccccceinnejlegejnhjdlnnneeggetfvffkrvtknllc
14:45:47 <dgilmore> puiterwijk: there is massive issues with how pungi does it also
14:46:05 <dgilmore> puiterwijk: so I would really like to sit back. and do it all over
14:46:07 <puiterwijk> dgilmore: oh, okay. I thought that was at least somewhat more sane than bodhi's
14:46:18 <dgilmore> puiterwijk: yes and no
14:46:24 <dgilmore> it uses runroot to do it in koji
14:46:26 <puiterwijk> dgilmore: right. But have we fixed the underlying problem about under resourcing?
14:46:41 <dgilmore> but its syncing the repo and updating then syncing back
14:47:17 <puiterwijk> Right.
14:47:29 <puiterwijk> Bodhi does the same
14:47:46 <dgilmore> puiterwijk: not really. but there is more people than just me and lmacken, than we had when it was done
14:48:02 <puiterwijk> dgilmore: right, fair enough. If you need help, I'd also be willing to chime in some
14:48:06 <dgilmore> puiterwijk: it really is a bolt on thing to everywhere
14:48:35 <puiterwijk> Yep, totally agreed.
14:48:46 <dgilmore> #info ostree repo management needs redesigned and unified in pungi and bodhi
14:49:45 <dgilmore> #info we are expecting ostree deliverables in the next release or two for more arches than x86_64
14:50:02 <dgilmore> as to the communication
14:50:08 <puiterwijk> dgilmore: are or are not?
14:50:15 <dgilmore> puiterwijk: are
14:50:25 <puiterwijk> Since "are expecting" and "in the next release or two" is strange. Wouldn't we keep it after we enable it?
14:50:29 <dgilmore> puiterwijk: pbrobinson is working on ppc64 and ppc64le for f25
14:50:43 <dgilmore> puiterwijk: it is not strange
14:51:02 <dgilmore> we are expecting we will have to deliver non x86_64 ostrees in the next release or two
14:51:12 <puiterwijk> oooooh, wait.\
14:51:22 <dgilmore> puiterwijk: it is when we expect it to arrive
14:51:26 <puiterwijk> You mean we will deliver it as first in the next release or the one after that?
14:51:28 <puiterwijk> Right.
14:51:35 <dgilmore> right
14:51:38 <puiterwijk> I thought you mean "We will only have this for the next release, and maybe the one after that"
14:51:46 <dgilmore> not at all
14:51:56 <dgilmore> #undo
14:51:56 <zodbot> Removing item from minutes: INFO by dgilmore at 14:49:45 : we are expecting ostree deliverables in the next release or two for more arches than x86_64
14:52:00 <puiterwijk> Okay, cool. Sorry for confusion then
14:52:25 <dgilmore> #info we are expecting to begin delivering ostree deliverables in the next release or two for arches other than x86_64
14:52:34 <dgilmore> perhaps that is clearer
14:52:39 <puiterwijk> dgilmore: thanks for the explanation. That's better
14:53:13 <dgilmore> afaik flatpack is supposed to eb delivered as ostree layers
14:53:31 <dgilmore> which will add significantly to the ostree repos also
14:54:13 <dgilmore> #info please file a issue in pagure or send an email to the releng list when dealing with issues changing what or how we deliver things
14:54:55 * nirik is sorry if he was the cause there. :( will try and make sure to file issues on these things
14:55:57 <dgilmore> nirik: it was not you
14:56:29 <masta> was it me?
14:56:36 <dgilmore> masta: yes it was
14:57:07 <nirik> well, I was helping on that too... but doesn't matter, lets try and fix it. ;)
14:57:15 <dgilmore> right
14:57:22 <dgilmore> I am not trying to assign blame
14:57:32 <dgilmore> I am trying to make sure we work better going forward
14:57:54 <masta> Yeah, I was on push duty that week, and remember we had some drama around moving bodhi backend to fedora
14:58:07 <dgilmore> mocks --new-chroot is not something we can enable in koji
14:58:08 <nirik> it would be really nice to have some kind of written up docs on the ostree stuff too... theres multiple streams and I never know what it where...
14:58:26 <dgilmore> livemedia-creator breaks with it in terrible ways
14:58:50 <dgilmore> I know because I tested way back when getting it implemented and working
14:59:09 <dgilmore> nirik: yeah it needs better documentation
14:59:27 <dgilmore> but there is fundamental flaws in the ways it is implemented
14:59:33 <nirik> and a plan. So is there going to be some kind of meeting and coming up with a plan? or ?
14:59:38 <masta> I hope we can go back to traditional mock chroot them
15:00:04 <dgilmore> nirik: needs to be a plan, likely a meeting with randy, mikem, and some releng people
15:00:13 <dgilmore> and ostree people also
15:00:34 <dgilmore> I think we will need a plugin to koji or a new task in it to make the ostrees
15:00:42 <nirik> +1
15:00:44 <dgilmore> and then adjust bodhi and pungi to use it
15:02:13 <mboddu> cccccceinnejgnviuunuerjrkvufnkjfitnrbrveijju
15:02:34 <dgilmore> #action dgilmore to set up a meeting with ostree, koji, bodhi, pungi and releng stakeholders to redo from the ground up ostree support in fedora
15:03:10 <dgilmore> okay I think that is all we have on this unless there is other ideas
15:04:20 <dgilmore> #topic rawhide signing
15:04:34 <dgilmore> so puiterwijk wanted to discuss automated signing
15:05:02 <puiterwijk> So, as of last week we have automated infra tag signing, and we can configure rawhide and enable it when we want.
15:05:14 <puiterwijk> So, the main question is: do we want? And if yes, when?
15:05:22 <dgilmore> puiterwijk: yes and asap
15:05:32 <puiterwijk> dgilmore: you and me, after the meeting, #fedora-releng? :)
15:05:44 <puiterwijk> Sincei t just need the tag and some configuration
15:05:44 <dgilmore> there si some longstanding bugs of people complaining about rawhide not being signed
15:05:51 <dgilmore> puiterwijk: sure
15:06:03 <dgilmore> we can then make fedora-repo changes to enable gpg checking
15:06:21 <nirik> and pungi needs to use it
15:06:24 <puiterwijk> I already submitted a PR to pungi
15:06:34 <puiterwijk> well, our pungi config
15:06:42 <dgilmore> puiterwijk: once rawhide is done we can look at enabling repodata signing as well as ostree and the other things we want
15:06:49 <nirik> and I think we should have pungi remove the pending tag (but we can do that later)
15:07:11 <puiterwijk> dgilmore: for ostree: that should be done, just waiting for a bodhi release for 23/24 and a working ostree compose for branched/rawhide
15:07:13 <dgilmore> nirik: what do you mean pungi needs to use it?
15:07:30 <puiterwijk> dgilmore: https://pagure.io/pungi-fedora/pull-request/78
15:07:31 <nirik> dgilmore: it needs to have the right key configured when it's gathering packages/making trees.
15:07:51 <dgilmore> nirik: there is zero support there for checking gpgkeys
15:08:23 <nirik> dgilmore: uh, I thought it needed to know what it gathers? otherwise wouldn't it just always get unsigned rpms?
15:08:31 <dgilmore> nirik: yes
15:08:44 <dgilmore> nirik: there is zero support for it checking rpms as it installs things
15:09:00 <dgilmore> afaik anaconda lacks all support also
15:09:07 * nirik is confused.
15:09:12 <dgilmore> maybe we are talking about differenmt things
15:09:22 <nirik> could be. ;)
15:09:38 <nirik> I just meant we need to tell it: "use packages signed by this key and if they aren't fail"
15:10:44 <dgilmore> nirik: that is ot entirely clear
15:11:02 <nirik> just like we do for branched...
15:11:18 <dgilmore> do you mean "include only packages signed by this tree in the repos that the compose makes" ?
15:11:40 <dgilmore> there is many places where gpg checking could be enabled
15:11:51 <dgilmore> okay
15:12:23 <nirik> sure, but I meant only the simple one of gathering packages signed with key XYZ when making repos and using packages, etc
15:12:32 <nirik> so, end users get the signed rpms
15:12:42 <nirik> yes, there's a bunch of other places we could enable someday...
15:13:20 <dgilmore> #info asap likely today we will have rawhide fail if unsigned packages get included
15:14:07 <dgilmore> puiterwijk: do we have docs on the interfaces for auto signing?
15:14:31 <nirik> my understand is that would we change the target for rawhide to f26-pending and autosign would look at that tag, sign, and move things to f26 when signed. Is that right?
15:15:03 <dgilmore> nirik: that is how I understand it
15:15:14 <puiterwijk> dgilmore: I do not yet. But I can write that today
15:15:19 <dgilmore> I am more talking about repodata, ostrees, etc
15:15:21 <puiterwijk> And yep.
15:15:35 <nirik> IMHO it would be nice if we could leave them tagged f26-pending until they actually went out in a compose... then taskotron or other qa stuff could test over them, and pungi untags them from that at the end of it's compose.
15:16:18 <dgilmore> nirik: I would like to have taskotron tests passing be part of teh buidl gating process
15:16:30 <dgilmore> gahh can not spell today
15:17:01 <dgilmore> I think step 1 we autosign
15:17:15 <dgilmore> step 2 we have qa testing be involved
15:17:25 <nirik> yep. we can add as we go...
15:17:36 <puiterwijk> We could just add yet another tag -testcandidate or something.
15:17:39 <dgilmore> then we can look at things like automated side tag rebuilds for soname bumps etc
15:17:54 <puiterwijk> And then have autosign move it from -pending -> -testcandidate, and taskotron -testcandidate -> normal
15:18:06 <nirik> the nice thing about having a tag for 'pending' is that we could do things like compose images from it and test them if desired, etc.
15:18:14 <puiterwijk> (just an idea)
15:18:22 <nirik> but yeah, we don't need to decide this today
15:18:26 <dgilmore> the question then is at what point do we add the builds to the buildroot
15:18:51 <dgilmore> I think given we are only autosigning we could add to the buildroot
15:18:51 <nirik> yeah...
15:19:10 <dgilmore> but if we want to gate on testing we should wait until testing passes
15:19:16 <nirik> right now we can just do the simple way... and talk with qa and see how we want it to look moving forward
15:19:16 <dgilmore> and the testing has to be fast
15:19:32 <nirik> yeah
15:19:51 <dgilmore> #action dgilmore to talk to adamw and tflink about how to work with automated testing going forward
15:20:35 <dgilmore> anything we nedd to cover puiterwijk, nirik or anyone else?
15:21:02 <puiterwijk> dgilmore: not that I know.
15:21:09 <nirik> dgilmore: we should sit down sometime and sort various security stuff for autosign/sigul... but it can be after freeze.
15:21:18 <nirik> ie, make sure the right people have all the right passwords, etc.
15:21:22 <dgilmore> #topic Alternative Architectures updates
15:21:33 <dgilmore> nirik: indeed
15:21:44 <dgilmore> sharkcz: how are the other arches looking?
15:21:57 <sharkcz> all good afaik
15:22:12 <sharkcz> we have composes on s390 since last week
15:22:18 <dgilmore> #info alt arches in good state
15:22:39 <dgilmore> #info s390 composes working as of last week
15:23:10 <dgilmore> I think in the next 2-3 weeks we should look at importing ppc64 and ppc64le into koji.fp.o for rawhide
15:24:14 <nirik> after beta is out sounds likely...
15:24:26 <dgilmore> sharkcz: any idea if we would be able to import s390x in the f26 cycle?
15:24:32 <dgilmore> or should we hold off for f27?
15:24:49 <dgilmore> nirik: likely it is a good window
15:25:14 <sharkcz> dgilmore: f27 looks more realistic, the infra things (builders/networks/..) are being discussed
15:25:20 <dgilmore> okay
15:25:46 <dgilmore> #topic Open Floor
15:25:55 <dgilmore> does anyone have anything else
15:26:12 <dgilmore> nirik: what is the status of the fedorahosted to pagure issue importer?
15:26:34 <nirik> I used it this weekend to import over the fedora-infrastructure trac...
15:26:42 <nirik> however... it has a big issue we need to sort out.
15:26:56 <nirik> You cannot currently disable fedmsgs to the issues repos.
15:27:10 <nirik> which means when you import tickets it fedmsgs everyone involved in every ticket ever.
15:27:20 <dgilmore> that is noisy
15:27:22 <puiterwijk> People might have noticed... a few emails or irc notifs going out :)
15:27:23 <nirik> we need to fix that before any more big imports
15:27:40 <nirik> we had to take extraordinary measures to stop the infra one from spewing
15:27:45 <dgilmore> 6500 odd messages would be bad
15:27:55 <puiterwijk> dgilmore: let's add two zeroes to that?
15:28:00 <puiterwijk> (for infra)
15:28:36 <dgilmore> puiterwijk: :) we have ~6500 tickets
15:28:39 <nirik> yeah, it's not only filing the ticket, but every comment or attachment or action.
15:28:50 <dgilmore> not sur eif its a msg per ticket or comment to ticket
15:28:54 <puiterwijk> dgilmore: lucky you!
15:29:07 <dgilmore> so yeah likely an order of magnitude more
15:29:09 <nirik> every comment/action. ;(
15:29:14 <puiterwijk> It depends on the ticket, some only send 1 (still open, non-commented), some may send MANY more
15:29:57 <nirik> anyhow, we need to fix that before any more large imports.
15:30:50 <dgilmore> cool thanks
15:31:49 <dgilmore> anything else?
15:31:55 <dgilmore> or should we wrap up?
15:32:35 * nirik has nothing off hand
15:32:48 <dgilmore> #endmeeting