14:35:13 #startmeeting RELENG (2016-09-26) 14:35:13 Meeting started Mon Sep 26 14:35:13 2016 UTC. The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:35:13 Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:35:13 The meeting name has been set to 'releng_(2016-09-26)' 14:35:13 #meetingname releng 14:35:13 #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion mboddu 14:35:13 The meeting name has been set to 'releng' 14:35:13 Current chairs: bochecha dgilmore masta maxamillion mboddu nirik pbrobinson pingou sharkcz tyll 14:35:16 #topic init process 14:35:38 * sharkcz is here 14:35:40 hello 14:36:03 * nirik is here 14:37:13 .hello mohanboddu 14:37:14 mboddu: mohanboddu 'Mohan Boddu' 14:38:06 Hi 14:38:55 I doubt pbrobinson will be here today and I know maxamillion will not be 14:39:01 * nirik runs to get coffee, back in a min 14:39:01 lets get started 14:39:11 #topic Communications 14:39:34 So we had issues a few weeks back with pushing updated ostrees 14:39:55 the person on push duty worked with ostree folks to resolve the issue 14:40:12 however it was done in a unsupportable and uncommunicated way 14:40:31 it also broke the ability to do ostrees as part of rawhide or branched composes 14:41:00 we 100% have to communicate when changing things 14:41:32 it would not have taken much to realise it was not suitable 14:42:04 it also highlighted that the way we are doing ostree is disjoint and not really great yet again 14:42:21 so we need a longer term plan to redo how we make ostrees 14:42:25 and how we manage them 14:43:03 Do we have a documentation of the current ways we do this? 14:43:27 puiterwijk: not really 14:43:37 bodhi does it one way and pungi another 14:43:55 could we move to some way in koji that both fire off and use? 14:43:58 the dijointness was due to under resourcing and having to shove things in at the last minute 14:44:07 Yeah, I noticed that. I think we should have a single codebase for both... maybe it would be an idea to let bodhi call pungi with a minified config? 14:44:08 nirik: that needs to be the plan 14:44:27 nirik: how its done in bodhi will fail as soon as we add any extra arches 14:44:51 it only works today as the bodhi box is x86_64 and so is the repo 14:45:00 fun. 14:45:13 dgilmore: yeah, agreed. So, would it be an idea to have bodhi call pungi's ostree phase code? 14:45:17 puiterwijk: that could be one way to do it 14:45:35 cccccceinnejlegejnhjdlnnneeggetfvffkrvtknllc 14:45:47 puiterwijk: there is massive issues with how pungi does it also 14:46:05 puiterwijk: so I would really like to sit back. and do it all over 14:46:07 dgilmore: oh, okay. I thought that was at least somewhat more sane than bodhi's 14:46:18 puiterwijk: yes and no 14:46:24 it uses runroot to do it in koji 14:46:26 dgilmore: right. But have we fixed the underlying problem about under resourcing? 14:46:41 but its syncing the repo and updating then syncing back 14:47:17 Right. 14:47:29 Bodhi does the same 14:47:46 puiterwijk: not really. but there is more people than just me and lmacken, than we had when it was done 14:48:02 dgilmore: right, fair enough. If you need help, I'd also be willing to chime in some 14:48:06 puiterwijk: it really is a bolt on thing to everywhere 14:48:35 Yep, totally agreed. 14:48:46 #info ostree repo management needs redesigned and unified in pungi and bodhi 14:49:45 #info we are expecting ostree deliverables in the next release or two for more arches than x86_64 14:50:02 as to the communication 14:50:08 dgilmore: are or are not? 14:50:15 puiterwijk: are 14:50:25 Since "are expecting" and "in the next release or two" is strange. Wouldn't we keep it after we enable it? 14:50:29 puiterwijk: pbrobinson is working on ppc64 and ppc64le for f25 14:50:43 puiterwijk: it is not strange 14:51:02 we are expecting we will have to deliver non x86_64 ostrees in the next release or two 14:51:12 oooooh, wait.\ 14:51:22 puiterwijk: it is when we expect it to arrive 14:51:26 You mean we will deliver it as first in the next release or the one after that? 14:51:28 Right. 14:51:35 right 14:51:38 I thought you mean "We will only have this for the next release, and maybe the one after that" 14:51:46 not at all 14:51:56 #undo 14:51:56 Removing item from minutes: INFO by dgilmore at 14:49:45 : we are expecting ostree deliverables in the next release or two for more arches than x86_64 14:52:00 Okay, cool. Sorry for confusion then 14:52:25 #info we are expecting to begin delivering ostree deliverables in the next release or two for arches other than x86_64 14:52:34 perhaps that is clearer 14:52:39 dgilmore: thanks for the explanation. That's better 14:53:13 afaik flatpack is supposed to eb delivered as ostree layers 14:53:31 which will add significantly to the ostree repos also 14:54:13 #info please file a issue in pagure or send an email to the releng list when dealing with issues changing what or how we deliver things 14:54:55 * nirik is sorry if he was the cause there. :( will try and make sure to file issues on these things 14:55:57 nirik: it was not you 14:56:29 was it me? 14:56:36 masta: yes it was 14:57:07 well, I was helping on that too... but doesn't matter, lets try and fix it. ;) 14:57:15 right 14:57:22 I am not trying to assign blame 14:57:32 I am trying to make sure we work better going forward 14:57:54 Yeah, I was on push duty that week, and remember we had some drama around moving bodhi backend to fedora 14:58:07 mocks --new-chroot is not something we can enable in koji 14:58:08 it would be really nice to have some kind of written up docs on the ostree stuff too... theres multiple streams and I never know what it where... 14:58:26 livemedia-creator breaks with it in terrible ways 14:58:50 I know because I tested way back when getting it implemented and working 14:59:09 nirik: yeah it needs better documentation 14:59:27 but there is fundamental flaws in the ways it is implemented 14:59:33 and a plan. So is there going to be some kind of meeting and coming up with a plan? or ? 14:59:38 I hope we can go back to traditional mock chroot them 15:00:04 nirik: needs to be a plan, likely a meeting with randy, mikem, and some releng people 15:00:13 and ostree people also 15:00:34 I think we will need a plugin to koji or a new task in it to make the ostrees 15:00:42 +1 15:00:44 and then adjust bodhi and pungi to use it 15:02:13 cccccceinnejgnviuunuerjrkvufnkjfitnrbrveijju 15:02:34 #action dgilmore to set up a meeting with ostree, koji, bodhi, pungi and releng stakeholders to redo from the ground up ostree support in fedora 15:03:10 okay I think that is all we have on this unless there is other ideas 15:04:20 #topic rawhide signing 15:04:34 so puiterwijk wanted to discuss automated signing 15:05:02 So, as of last week we have automated infra tag signing, and we can configure rawhide and enable it when we want. 15:05:14 So, the main question is: do we want? And if yes, when? 15:05:22 puiterwijk: yes and asap 15:05:32 dgilmore: you and me, after the meeting, #fedora-releng? :) 15:05:44 Sincei t just need the tag and some configuration 15:05:44 there si some longstanding bugs of people complaining about rawhide not being signed 15:05:51 puiterwijk: sure 15:06:03 we can then make fedora-repo changes to enable gpg checking 15:06:21 and pungi needs to use it 15:06:24 I already submitted a PR to pungi 15:06:34 well, our pungi config 15:06:42 puiterwijk: once rawhide is done we can look at enabling repodata signing as well as ostree and the other things we want 15:06:49 and I think we should have pungi remove the pending tag (but we can do that later) 15:07:11 dgilmore: for ostree: that should be done, just waiting for a bodhi release for 23/24 and a working ostree compose for branched/rawhide 15:07:13 nirik: what do you mean pungi needs to use it? 15:07:30 dgilmore: https://pagure.io/pungi-fedora/pull-request/78 15:07:31 dgilmore: it needs to have the right key configured when it's gathering packages/making trees. 15:07:51 nirik: there is zero support there for checking gpgkeys 15:08:23 dgilmore: uh, I thought it needed to know what it gathers? otherwise wouldn't it just always get unsigned rpms? 15:08:31 nirik: yes 15:08:44 nirik: there is zero support for it checking rpms as it installs things 15:09:00 afaik anaconda lacks all support also 15:09:07 * nirik is confused. 15:09:12 maybe we are talking about differenmt things 15:09:22 could be. ;) 15:09:38 I just meant we need to tell it: "use packages signed by this key and if they aren't fail" 15:10:44 nirik: that is ot entirely clear 15:11:02 just like we do for branched... 15:11:18 do you mean "include only packages signed by this tree in the repos that the compose makes" ? 15:11:40 there is many places where gpg checking could be enabled 15:11:51 okay 15:12:23 sure, but I meant only the simple one of gathering packages signed with key XYZ when making repos and using packages, etc 15:12:32 so, end users get the signed rpms 15:12:42 yes, there's a bunch of other places we could enable someday... 15:13:20 #info asap likely today we will have rawhide fail if unsigned packages get included 15:14:07 puiterwijk: do we have docs on the interfaces for auto signing? 15:14:31 my understand is that would we change the target for rawhide to f26-pending and autosign would look at that tag, sign, and move things to f26 when signed. Is that right? 15:15:03 nirik: that is how I understand it 15:15:14 dgilmore: I do not yet. But I can write that today 15:15:19 I am more talking about repodata, ostrees, etc 15:15:21 And yep. 15:15:35 IMHO it would be nice if we could leave them tagged f26-pending until they actually went out in a compose... then taskotron or other qa stuff could test over them, and pungi untags them from that at the end of it's compose. 15:16:18 nirik: I would like to have taskotron tests passing be part of teh buidl gating process 15:16:30 gahh can not spell today 15:17:01 I think step 1 we autosign 15:17:15 step 2 we have qa testing be involved 15:17:25 yep. we can add as we go... 15:17:36 We could just add yet another tag -testcandidate or something. 15:17:39 then we can look at things like automated side tag rebuilds for soname bumps etc 15:17:54 And then have autosign move it from -pending -> -testcandidate, and taskotron -testcandidate -> normal 15:18:06 the nice thing about having a tag for 'pending' is that we could do things like compose images from it and test them if desired, etc. 15:18:14 (just an idea) 15:18:22 but yeah, we don't need to decide this today 15:18:26 the question then is at what point do we add the builds to the buildroot 15:18:51 I think given we are only autosigning we could add to the buildroot 15:18:51 yeah... 15:19:10 but if we want to gate on testing we should wait until testing passes 15:19:16 right now we can just do the simple way... and talk with qa and see how we want it to look moving forward 15:19:16 and the testing has to be fast 15:19:32 yeah 15:19:51 #action dgilmore to talk to adamw and tflink about how to work with automated testing going forward 15:20:35 anything we nedd to cover puiterwijk, nirik or anyone else? 15:21:02 dgilmore: not that I know. 15:21:09 dgilmore: we should sit down sometime and sort various security stuff for autosign/sigul... but it can be after freeze. 15:21:18 ie, make sure the right people have all the right passwords, etc. 15:21:22 #topic Alternative Architectures updates 15:21:33 nirik: indeed 15:21:44 sharkcz: how are the other arches looking? 15:21:57 all good afaik 15:22:12 we have composes on s390 since last week 15:22:18 #info alt arches in good state 15:22:39 #info s390 composes working as of last week 15:23:10 I think in the next 2-3 weeks we should look at importing ppc64 and ppc64le into koji.fp.o for rawhide 15:24:14 after beta is out sounds likely... 15:24:26 sharkcz: any idea if we would be able to import s390x in the f26 cycle? 15:24:32 or should we hold off for f27? 15:24:49 nirik: likely it is a good window 15:25:14 dgilmore: f27 looks more realistic, the infra things (builders/networks/..) are being discussed 15:25:20 okay 15:25:46 #topic Open Floor 15:25:55 does anyone have anything else 15:26:12 nirik: what is the status of the fedorahosted to pagure issue importer? 15:26:34 I used it this weekend to import over the fedora-infrastructure trac... 15:26:42 however... it has a big issue we need to sort out. 15:26:56 You cannot currently disable fedmsgs to the issues repos. 15:27:10 which means when you import tickets it fedmsgs everyone involved in every ticket ever. 15:27:20 that is noisy 15:27:22 People might have noticed... a few emails or irc notifs going out :) 15:27:23 we need to fix that before any more big imports 15:27:40 we had to take extraordinary measures to stop the infra one from spewing 15:27:45 6500 odd messages would be bad 15:27:55 dgilmore: let's add two zeroes to that? 15:28:00 (for infra) 15:28:36 puiterwijk: :) we have ~6500 tickets 15:28:39 yeah, it's not only filing the ticket, but every comment or attachment or action. 15:28:50 not sur eif its a msg per ticket or comment to ticket 15:28:54 dgilmore: lucky you! 15:29:07 so yeah likely an order of magnitude more 15:29:09 every comment/action. ;( 15:29:14 It depends on the ticket, some only send 1 (still open, non-commented), some may send MANY more 15:29:57 anyhow, we need to fix that before any more large imports. 15:30:50 cool thanks 15:31:49 anything else? 15:31:55 or should we wrap up? 15:32:35 * nirik has nothing off hand 15:32:48 #endmeeting