14:35:13 <dgilmore> #startmeeting RELENG (2016-09-26) 14:35:13 <zodbot> Meeting started Mon Sep 26 14:35:13 2016 UTC. The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:35:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:35:13 <zodbot> The meeting name has been set to 'releng_(2016-09-26)' 14:35:13 <dgilmore> #meetingname releng 14:35:13 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion mboddu 14:35:13 <zodbot> The meeting name has been set to 'releng' 14:35:13 <zodbot> Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll 14:35:16 <dgilmore> #topic init process 14:35:38 * sharkcz is here 14:35:40 <masta> hello 14:36:03 * nirik is here 14:37:13 <mboddu> .hello mohanboddu 14:37:14 <zodbot> mboddu: mohanboddu 'Mohan Boddu' <mboddu@bhujji.com> 14:38:06 <puiterwijk> Hi 14:38:55 <dgilmore> I doubt pbrobinson will be here today and I know maxamillion will not be 14:39:01 * nirik runs to get coffee, back in a min 14:39:01 <dgilmore> lets get started 14:39:11 <dgilmore> #topic Communications 14:39:34 <dgilmore> So we had issues a few weeks back with pushing updated ostrees 14:39:55 <dgilmore> the person on push duty worked with ostree folks to resolve the issue 14:40:12 <dgilmore> however it was done in a unsupportable and uncommunicated way 14:40:31 <dgilmore> it also broke the ability to do ostrees as part of rawhide or branched composes 14:41:00 <dgilmore> we 100% have to communicate when changing things 14:41:32 <dgilmore> it would not have taken much to realise it was not suitable 14:42:04 <dgilmore> it also highlighted that the way we are doing ostree is disjoint and not really great yet again 14:42:21 <dgilmore> so we need a longer term plan to redo how we make ostrees 14:42:25 <dgilmore> and how we manage them 14:43:03 <puiterwijk> Do we have a documentation of the current ways we do this? 14:43:27 <dgilmore> puiterwijk: not really 14:43:37 <dgilmore> bodhi does it one way and pungi another 14:43:55 <nirik> could we move to some way in koji that both fire off and use? 14:43:58 <dgilmore> the dijointness was due to under resourcing and having to shove things in at the last minute 14:44:07 <puiterwijk> Yeah, I noticed that. I think we should have a single codebase for both... maybe it would be an idea to let bodhi call pungi with a minified config? 14:44:08 <dgilmore> nirik: that needs to be the plan 14:44:27 <dgilmore> nirik: how its done in bodhi will fail as soon as we add any extra arches 14:44:51 <dgilmore> it only works today as the bodhi box is x86_64 and so is the repo 14:45:00 <nirik> fun. 14:45:13 <puiterwijk> dgilmore: yeah, agreed. So, would it be an idea to have bodhi call pungi's ostree phase code? 14:45:17 <dgilmore> puiterwijk: that could be one way to do it 14:45:35 <mboddu> cccccceinnejlegejnhjdlnnneeggetfvffkrvtknllc 14:45:47 <dgilmore> puiterwijk: there is massive issues with how pungi does it also 14:46:05 <dgilmore> puiterwijk: so I would really like to sit back. and do it all over 14:46:07 <puiterwijk> dgilmore: oh, okay. I thought that was at least somewhat more sane than bodhi's 14:46:18 <dgilmore> puiterwijk: yes and no 14:46:24 <dgilmore> it uses runroot to do it in koji 14:46:26 <puiterwijk> dgilmore: right. But have we fixed the underlying problem about under resourcing? 14:46:41 <dgilmore> but its syncing the repo and updating then syncing back 14:47:17 <puiterwijk> Right. 14:47:29 <puiterwijk> Bodhi does the same 14:47:46 <dgilmore> puiterwijk: not really. but there is more people than just me and lmacken, than we had when it was done 14:48:02 <puiterwijk> dgilmore: right, fair enough. If you need help, I'd also be willing to chime in some 14:48:06 <dgilmore> puiterwijk: it really is a bolt on thing to everywhere 14:48:35 <puiterwijk> Yep, totally agreed. 14:48:46 <dgilmore> #info ostree repo management needs redesigned and unified in pungi and bodhi 14:49:45 <dgilmore> #info we are expecting ostree deliverables in the next release or two for more arches than x86_64 14:50:02 <dgilmore> as to the communication 14:50:08 <puiterwijk> dgilmore: are or are not? 14:50:15 <dgilmore> puiterwijk: are 14:50:25 <puiterwijk> Since "are expecting" and "in the next release or two" is strange. Wouldn't we keep it after we enable it? 14:50:29 <dgilmore> puiterwijk: pbrobinson is working on ppc64 and ppc64le for f25 14:50:43 <dgilmore> puiterwijk: it is not strange 14:51:02 <dgilmore> we are expecting we will have to deliver non x86_64 ostrees in the next release or two 14:51:12 <puiterwijk> oooooh, wait.\ 14:51:22 <dgilmore> puiterwijk: it is when we expect it to arrive 14:51:26 <puiterwijk> You mean we will deliver it as first in the next release or the one after that? 14:51:28 <puiterwijk> Right. 14:51:35 <dgilmore> right 14:51:38 <puiterwijk> I thought you mean "We will only have this for the next release, and maybe the one after that" 14:51:46 <dgilmore> not at all 14:51:56 <dgilmore> #undo 14:51:56 <zodbot> Removing item from minutes: INFO by dgilmore at 14:49:45 : we are expecting ostree deliverables in the next release or two for more arches than x86_64 14:52:00 <puiterwijk> Okay, cool. Sorry for confusion then 14:52:25 <dgilmore> #info we are expecting to begin delivering ostree deliverables in the next release or two for arches other than x86_64 14:52:34 <dgilmore> perhaps that is clearer 14:52:39 <puiterwijk> dgilmore: thanks for the explanation. That's better 14:53:13 <dgilmore> afaik flatpack is supposed to eb delivered as ostree layers 14:53:31 <dgilmore> which will add significantly to the ostree repos also 14:54:13 <dgilmore> #info please file a issue in pagure or send an email to the releng list when dealing with issues changing what or how we deliver things 14:54:55 * nirik is sorry if he was the cause there. :( will try and make sure to file issues on these things 14:55:57 <dgilmore> nirik: it was not you 14:56:29 <masta> was it me? 14:56:36 <dgilmore> masta: yes it was 14:57:07 <nirik> well, I was helping on that too... but doesn't matter, lets try and fix it. ;) 14:57:15 <dgilmore> right 14:57:22 <dgilmore> I am not trying to assign blame 14:57:32 <dgilmore> I am trying to make sure we work better going forward 14:57:54 <masta> Yeah, I was on push duty that week, and remember we had some drama around moving bodhi backend to fedora 14:58:07 <dgilmore> mocks --new-chroot is not something we can enable in koji 14:58:08 <nirik> it would be really nice to have some kind of written up docs on the ostree stuff too... theres multiple streams and I never know what it where... 14:58:26 <dgilmore> livemedia-creator breaks with it in terrible ways 14:58:50 <dgilmore> I know because I tested way back when getting it implemented and working 14:59:09 <dgilmore> nirik: yeah it needs better documentation 14:59:27 <dgilmore> but there is fundamental flaws in the ways it is implemented 14:59:33 <nirik> and a plan. So is there going to be some kind of meeting and coming up with a plan? or ? 14:59:38 <masta> I hope we can go back to traditional mock chroot them 15:00:04 <dgilmore> nirik: needs to be a plan, likely a meeting with randy, mikem, and some releng people 15:00:13 <dgilmore> and ostree people also 15:00:34 <dgilmore> I think we will need a plugin to koji or a new task in it to make the ostrees 15:00:42 <nirik> +1 15:00:44 <dgilmore> and then adjust bodhi and pungi to use it 15:02:13 <mboddu> cccccceinnejgnviuunuerjrkvufnkjfitnrbrveijju 15:02:34 <dgilmore> #action dgilmore to set up a meeting with ostree, koji, bodhi, pungi and releng stakeholders to redo from the ground up ostree support in fedora 15:03:10 <dgilmore> okay I think that is all we have on this unless there is other ideas 15:04:20 <dgilmore> #topic rawhide signing 15:04:34 <dgilmore> so puiterwijk wanted to discuss automated signing 15:05:02 <puiterwijk> So, as of last week we have automated infra tag signing, and we can configure rawhide and enable it when we want. 15:05:14 <puiterwijk> So, the main question is: do we want? And if yes, when? 15:05:22 <dgilmore> puiterwijk: yes and asap 15:05:32 <puiterwijk> dgilmore: you and me, after the meeting, #fedora-releng? :) 15:05:44 <puiterwijk> Sincei t just need the tag and some configuration 15:05:44 <dgilmore> there si some longstanding bugs of people complaining about rawhide not being signed 15:05:51 <dgilmore> puiterwijk: sure 15:06:03 <dgilmore> we can then make fedora-repo changes to enable gpg checking 15:06:21 <nirik> and pungi needs to use it 15:06:24 <puiterwijk> I already submitted a PR to pungi 15:06:34 <puiterwijk> well, our pungi config 15:06:42 <dgilmore> puiterwijk: once rawhide is done we can look at enabling repodata signing as well as ostree and the other things we want 15:06:49 <nirik> and I think we should have pungi remove the pending tag (but we can do that later) 15:07:11 <puiterwijk> dgilmore: for ostree: that should be done, just waiting for a bodhi release for 23/24 and a working ostree compose for branched/rawhide 15:07:13 <dgilmore> nirik: what do you mean pungi needs to use it? 15:07:30 <puiterwijk> dgilmore: https://pagure.io/pungi-fedora/pull-request/78 15:07:31 <nirik> dgilmore: it needs to have the right key configured when it's gathering packages/making trees. 15:07:51 <dgilmore> nirik: there is zero support there for checking gpgkeys 15:08:23 <nirik> dgilmore: uh, I thought it needed to know what it gathers? otherwise wouldn't it just always get unsigned rpms? 15:08:31 <dgilmore> nirik: yes 15:08:44 <dgilmore> nirik: there is zero support for it checking rpms as it installs things 15:09:00 <dgilmore> afaik anaconda lacks all support also 15:09:07 * nirik is confused. 15:09:12 <dgilmore> maybe we are talking about differenmt things 15:09:22 <nirik> could be. ;) 15:09:38 <nirik> I just meant we need to tell it: "use packages signed by this key and if they aren't fail" 15:10:44 <dgilmore> nirik: that is ot entirely clear 15:11:02 <nirik> just like we do for branched... 15:11:18 <dgilmore> do you mean "include only packages signed by this tree in the repos that the compose makes" ? 15:11:40 <dgilmore> there is many places where gpg checking could be enabled 15:11:51 <dgilmore> okay 15:12:23 <nirik> sure, but I meant only the simple one of gathering packages signed with key XYZ when making repos and using packages, etc 15:12:32 <nirik> so, end users get the signed rpms 15:12:42 <nirik> yes, there's a bunch of other places we could enable someday... 15:13:20 <dgilmore> #info asap likely today we will have rawhide fail if unsigned packages get included 15:14:07 <dgilmore> puiterwijk: do we have docs on the interfaces for auto signing? 15:14:31 <nirik> my understand is that would we change the target for rawhide to f26-pending and autosign would look at that tag, sign, and move things to f26 when signed. Is that right? 15:15:03 <dgilmore> nirik: that is how I understand it 15:15:14 <puiterwijk> dgilmore: I do not yet. But I can write that today 15:15:19 <dgilmore> I am more talking about repodata, ostrees, etc 15:15:21 <puiterwijk> And yep. 15:15:35 <nirik> IMHO it would be nice if we could leave them tagged f26-pending until they actually went out in a compose... then taskotron or other qa stuff could test over them, and pungi untags them from that at the end of it's compose. 15:16:18 <dgilmore> nirik: I would like to have taskotron tests passing be part of teh buidl gating process 15:16:30 <dgilmore> gahh can not spell today 15:17:01 <dgilmore> I think step 1 we autosign 15:17:15 <dgilmore> step 2 we have qa testing be involved 15:17:25 <nirik> yep. we can add as we go... 15:17:36 <puiterwijk> We could just add yet another tag -testcandidate or something. 15:17:39 <dgilmore> then we can look at things like automated side tag rebuilds for soname bumps etc 15:17:54 <puiterwijk> And then have autosign move it from -pending -> -testcandidate, and taskotron -testcandidate -> normal 15:18:06 <nirik> the nice thing about having a tag for 'pending' is that we could do things like compose images from it and test them if desired, etc. 15:18:14 <puiterwijk> (just an idea) 15:18:22 <nirik> but yeah, we don't need to decide this today 15:18:26 <dgilmore> the question then is at what point do we add the builds to the buildroot 15:18:51 <dgilmore> I think given we are only autosigning we could add to the buildroot 15:18:51 <nirik> yeah... 15:19:10 <dgilmore> but if we want to gate on testing we should wait until testing passes 15:19:16 <nirik> right now we can just do the simple way... and talk with qa and see how we want it to look moving forward 15:19:16 <dgilmore> and the testing has to be fast 15:19:32 <nirik> yeah 15:19:51 <dgilmore> #action dgilmore to talk to adamw and tflink about how to work with automated testing going forward 15:20:35 <dgilmore> anything we nedd to cover puiterwijk, nirik or anyone else? 15:21:02 <puiterwijk> dgilmore: not that I know. 15:21:09 <nirik> dgilmore: we should sit down sometime and sort various security stuff for autosign/sigul... but it can be after freeze. 15:21:18 <nirik> ie, make sure the right people have all the right passwords, etc. 15:21:22 <dgilmore> #topic Alternative Architectures updates 15:21:33 <dgilmore> nirik: indeed 15:21:44 <dgilmore> sharkcz: how are the other arches looking? 15:21:57 <sharkcz> all good afaik 15:22:12 <sharkcz> we have composes on s390 since last week 15:22:18 <dgilmore> #info alt arches in good state 15:22:39 <dgilmore> #info s390 composes working as of last week 15:23:10 <dgilmore> I think in the next 2-3 weeks we should look at importing ppc64 and ppc64le into koji.fp.o for rawhide 15:24:14 <nirik> after beta is out sounds likely... 15:24:26 <dgilmore> sharkcz: any idea if we would be able to import s390x in the f26 cycle? 15:24:32 <dgilmore> or should we hold off for f27? 15:24:49 <dgilmore> nirik: likely it is a good window 15:25:14 <sharkcz> dgilmore: f27 looks more realistic, the infra things (builders/networks/..) are being discussed 15:25:20 <dgilmore> okay 15:25:46 <dgilmore> #topic Open Floor 15:25:55 <dgilmore> does anyone have anything else 15:26:12 <dgilmore> nirik: what is the status of the fedorahosted to pagure issue importer? 15:26:34 <nirik> I used it this weekend to import over the fedora-infrastructure trac... 15:26:42 <nirik> however... it has a big issue we need to sort out. 15:26:56 <nirik> You cannot currently disable fedmsgs to the issues repos. 15:27:10 <nirik> which means when you import tickets it fedmsgs everyone involved in every ticket ever. 15:27:20 <dgilmore> that is noisy 15:27:22 <puiterwijk> People might have noticed... a few emails or irc notifs going out :) 15:27:23 <nirik> we need to fix that before any more big imports 15:27:40 <nirik> we had to take extraordinary measures to stop the infra one from spewing 15:27:45 <dgilmore> 6500 odd messages would be bad 15:27:55 <puiterwijk> dgilmore: let's add two zeroes to that? 15:28:00 <puiterwijk> (for infra) 15:28:36 <dgilmore> puiterwijk: :) we have ~6500 tickets 15:28:39 <nirik> yeah, it's not only filing the ticket, but every comment or attachment or action. 15:28:50 <dgilmore> not sur eif its a msg per ticket or comment to ticket 15:28:54 <puiterwijk> dgilmore: lucky you! 15:29:07 <dgilmore> so yeah likely an order of magnitude more 15:29:09 <nirik> every comment/action. ;( 15:29:14 <puiterwijk> It depends on the ticket, some only send 1 (still open, non-commented), some may send MANY more 15:29:57 <nirik> anyhow, we need to fix that before any more large imports. 15:30:50 <dgilmore> cool thanks 15:31:49 <dgilmore> anything else? 15:31:55 <dgilmore> or should we wrap up? 15:32:35 * nirik has nothing off hand 15:32:48 <dgilmore> #endmeeting