15:34:00 #startmeeting RELENG (2017-02-20) 15:34:00 Meeting started Mon Feb 20 15:34:00 2017 UTC. The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:34:00 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:34:00 The meeting name has been set to 'releng_(2017-02-20)' 15:34:00 #meetingname releng 15:34:00 The meeting name has been set to 'releng' 15:34:00 #chair dgilmore nirik tyll sharkcz masta pbrobinson pingou puiterwijk maxamillion mboddu 15:34:00 Current chairs: dgilmore masta maxamillion mboddu nirik pbrobinson pingou puiterwijk sharkcz tyll 15:34:03 #topic init process 15:34:14 .hello ignatenkobrain 15:34:15 ignatenkobrain: ignatenkobrain 'Igor Gnatenko' 15:34:19 hi everyone 15:34:27 .hello mohanboddu\ 15:34:28 mboddu: Sorry, but you don't exist 15:34:34 .hello mohanboddu 15:34:36 mboddu: mohanboddu 'Mohan Boddu' 15:34:42 .hello kevin 15:34:43 nirik: kevin 'Kevin Fenzi' 15:34:48 .hello puiterwijk 15:34:49 .hello sharkcz 15:34:49 puiterwijk: puiterwijk 'Patrick "マルタインアンドレアス" Uiterwijk' 15:34:52 sharkcz: sharkcz 'Dan Horák' 15:36:36 .hello maxamillion 15:36:39 maxamillion: maxamillion 'Adam Miller' 15:36:40 #topic #6601 Migrate the rel-eng tickets from (old) trac 15:36:44 .hello ngompa 15:36:45 Son_Goku: ngompa 'Neal Gompa' 15:36:48 https://pagure.io/releng/issue/6601 15:36:53 lets get moving 15:37:39 nirik: I asked pingou to join, while we wait a quick sec do you have any update? 15:37:55 no. I also asked him to join. :) 15:38:02 he's around, so hopefully soon here. 15:38:36 .hello bowlofeggs 15:38:37 bowlofeggs: bowlofeggs 'Randy Barlow' 15:38:41 to note here 15:39:08 hey pingou. Any chance to look at the releng ticket mess ? :) 15:39:34 yeah, we should 15:39:39 Last week threebean set up a process to sync the pagure issues into a internal red hat issue tracker so that mboddu's and my management chain have more visibility into what's going on in Fedora 15:40:09 that might be fun if we push the old tickets 15:40:16 pingou: indeed 15:40:21 yeah, they will get a flood. ;) 15:40:36 or I'll poke threebean before doing the final push 15:40:55 we likely want open ones synced 15:41:11 pingou: so I guess you are saying you have not looked at it? 15:41:26 fedorahosted.org is being shut down next week right? 15:41:44 does that also include the project /releases folders that people are still using? 15:41:57 dgilmore: I started, you know when I was asking for the read access back 15:42:05 because there are a number of projects that are still using those 15:42:19 Son_Goku: that is off topic for this discussion 15:42:29 pingou: okay 15:42:32 for shutting down fedorahosted.org? 15:42:32 yes, it includes everything (except lists.fedorahosted.org) 15:42:39 pingou: what is it going to take to get resolved? 15:42:50 time and some hair splitting 15:43:08 do we know the problem issues? 15:43:25 issue identifiers is the issue 15:43:36 we'll need to decide which identifiers we want to keep stable 15:43:49 I am confused now 15:43:51 because I doubt we'll be able to squeeze all the FH tickets in b/w the pagure ones 15:44:11 as I understood the problem there was some conflicts with attachments to issues? 15:44:38 pingou: what do you mean by b/w? 15:44:39 conflicts w/ attachments? no idea what you're referring to 15:45:04 the last problem was usernames of ticket filers 15:45:11 pingou: that when nirik tried to import the issues it failed as there was issue with some attachements 15:45:13 because email2trac had been used 15:45:32 so instead of username they had "Foo bar baz " 15:45:50 afaik most of that was all spam 15:46:00 there was an attachment issue at one point... but yeah lots of issues. 15:46:03 in b/w as in: there is a gap in the identifier used on the pagure side, so we thought we would squeeze the FH tickets in this gap on pagure, but I'm not 100% sure it'll work 15:46:12 dgilmore: all the ones left were legit. 15:46:23 we have now: 15:46:45 +++++ 15:46:53 tickets filed in pagure when it first was setup === tickets imported from trac, but incompletely === more tickets after that 15:47:10 pingou: it should still work if we can fix the imported ones 15:47:43 so what we need to do is dump them all out, fix the issue filer on those email2trac ones and reimport then entire thing? 15:47:51 yup 15:47:58 that's at least my idea 15:48:20 * nirik nods. 15:48:40 not sure if every email would resolve to a fas account 15:48:40 +++ 15:48:45 ignatenkobrain: ? 15:48:52 oh god 15:48:58 my keyboard is broken 15:49:00 sorry 15:49:05 okay 15:49:08 dgilmore: yeah, some are obvious 15:49:12 some not so much so 15:49:13 I'll likely want to release 2.13 before we import so that loading the tickets is done via its own service 15:49:31 if we cant figure it we should just set them to nobody or something 15:49:36 + 15:49:38 1 15:50:14 #info current status is that issues had been filed via email2trac, we can not resolve all emails to fas accounts. 15:50:30 #info we should set all unknown to nobody 15:50:30 https://fedorahosted.org/rel-eng/query?reporter=~%3C&col=id&col=summary&col=status&col=type&col=milestone&col=component&order=priority 15:50:50 109 ticket commits, but less than 109 tickets 15:50:56 #info pingou wants to wait until 2.13 of pagure is deployed 15:51:02 no, that might actually be 109 tickets 15:51:11 pingou: when do you plane to do the deployment 15:51:22 #info we have until Feb 28 to complete task 15:51:51 https://fedorahosted.org/rel-eng/ticket/26 we could probably just delete 15:52:03 dgilmore: got 2 PRs to review and merge, then we'll see what's left in the PR queue but I'd like to do it this week, sooner rather than later 15:53:31 pingou: please do if you can 15:53:59 anything else to cover here? 15:54:24 #topic #6577 Fedora 26 mass rebuild 15:54:25 https://pagure.io/releng/issue/6577 15:54:36 so the mass rebuild from ourside is done 15:55:47 yep. 15:56:07 looks like pkgconf change is fine 15:56:09 I think next time we might want to look at using the second sign vault and another sign bridge so we dont' backlog signing 15:56:34 +1 yeah 15:56:35 #action mboddu to run the scripts that output failure and needs rebuild lists for at least 4 more weeks 15:56:57 nirik: not sure thats really possible 15:57:08 yeah, it might not be easy... just a thought. 15:57:18 dgilmore: it should with the new sigul version... 15:57:34 The new one has automatic syncing between two vaults 15:57:35 http://dl.fedoraproject.org/pub/alt/mass-rebuild/f26-failures.html 15:57:40 http://dl.fedoraproject.org/pub/alt/mass-rebuild/f26-need-rebuild.html 15:58:20 well, as long as we don't change valut content... 15:58:27 https://bugzilla.redhat.com/show_bug.cgi?id=1423041 15:58:46 nirik: you'd still want to sync logs I think :). But yes, that'd make it even easier 15:58:51 puiterwijk: I do not think thats the issue 15:59:05 more I am wondering about how load balancing the requests would work 15:59:14 well, we don't really do anything with logs currently, but meh.. 15:59:22 it wouldn't work to load balance it. 15:59:30 I am saying to make 2 seperate ones 15:59:44 nirik: they would start stomping on each other 15:59:51 trying to sign the same rpms 16:00:07 well, if one is doing mass rebuilds and the other all the rest... there should be no overlap? 16:00:22 Not if you keep the m separate. One bridge01 and one bridge02 16:00:42 nirik: the load balancing of that still has to happen 16:00:45 (you probably don't even need two bridges, but that's another addition for the future) 16:00:50 yeah. but perhaps it's all too much complexity and people should just be more patient. ;) 16:01:07 but I guess running two instances of robosignatory configured differently could do that 16:01:10 dgilmore: the idea was to just add a second bridge that uses the second vault. 16:01:19 Yep 16:01:42 we will hit similiar issues in the next week 16:01:54 when we make a f27 key and have to resign all of rawhide 16:02:43 #info signing everything slowed down signing overall 16:02:56 the other option might be to add some kind of scheduling to robosign... 16:03:08 #info going forward we could look at ways to make signing scale better 16:03:12 ie, prefer some tags over others 16:03:14 dgilmore: for next time, I would suggest to juse use the standard autosigning for the mass rebuild... IF it had been setup in advance, we wouldn't have had any problems 16:03:30 Or at least less 16:03:42 puiterwijk: there will be some 16:03:55 Sure. But way less, and especially no further problems when merging in 16:04:04 and we will hit the probnlem likely worse when moving rawhide to the new key 16:04:15 Yeah, agreed. 16:04:16 as nothing will have been autosigned with the f27 key 16:04:44 well, if we manually sign f27 with a small batch number... then enable it in robosign... 16:04:50 it might not be that disruptive 16:05:18 nirik: as we do things today. we have a window of about 2 days to sign everything with the f27 key 16:05:33 Right. What day are you planning to create that key? Depending on that, I might have chance to look at the prioritization/scaling... 16:05:33 I guess we should make the f27 key this week 16:06:05 #action dgilmore to make f27 key today/tomorrow 16:06:07 Okay, this week is a 99% "no" on time for me, unless I can get an ack from people to drop other stuff 16:06:19 * dgilmore is taking this afternoon off as MAson is off school 16:06:34 if we have a key I can start manually signing with it. ;) 16:06:36 you can use this for the f27 signing key: 367316 16:06:43 bowlofeggs: yeah, no 16:06:46 haha 16:06:49 https://fedoraproject.org/wiki/Releases/26/Schedule 16:07:04 we have branching next week 16:07:10 on Tuesday 16:07:30 lets change topics as I think mass rebuild is done 16:07:47 #topic Fedora 26 movement 16:08:08 #info this week we need to make the Fedora 27 gpg key for rpms 16:08:28 #info this week we need to branch all of the compose configs for Fedora 26 16:08:50 https://pagure.io/pungi-fedora 16:09:04 https://pagure.io/fedora-kickstarts/ 16:09:17 https://pagure.io/fedora-lorax-templates 16:09:29 https://pagure.io/fedora-atomic 16:09:52 https://pagure.io/workstation-ostree-config 16:10:04 all need branching 16:10:12 random side note, do have a doc that references all of those and what their purpose in the releng pipeline is? 16:10:32 https://github.com/rpm-software-management/mock will need an update with new mock configs 16:10:41 maxamillion: likely not 16:10:46 dgilmore: I can help with branching 16:11:05 https://docs.pagure.org/releng/ 16:11:15 dgilmore: we probably should ... if I can find some time this week I'll write one up 16:11:18 mboddu: its all on you to do. 16:11:25 dgilmore: okay 16:12:07 dgilmore: ignatenkobrain had a topic and has to leave soon... 16:12:12 we will also need to branch https://pagure.io/fedora-repos https://pagure.io/fedora-release 16:12:26 nirik: okay thats news to me 16:12:28 I don't need to leave soon, but at least I need to know when I should come back :) 16:12:34 we can look at that after this 16:12:41 oh, sorry, misunderstood then 16:13:06 ignatenkobrain: well random topics are at the end of the meeting in open floor 16:13:14 side note on fedora-repos: it would be nice if we had a link from f26 key to rawhide or f27 key to rawhide when we change it. Would help updates. 16:13:24 today we are going to have to cut some things short as we are running out of time 16:13:24 * nirik should file an issue 16:13:53 dgilmore: I wouldn't name it as random because we discussed that topic on ML and Kevin proposed to join next meeting and discuss thing 16:14:43 #info all packages for rawhide have to be signed with the f27 key and autosigning setup by COB Monday, so that as soon as we land all the changes for branching rawhide switches to f27 keys 16:15:01 ignatenkobrain: its not in a issue with the meeting keyword? 16:15:06 ignatenkobrain: if so its random 16:15:10 as its not in the agenda 16:15:15 dgilmore: no one told me to do this =( 16:15:25 ignatenkobrain: afaik its widely known 16:15:27 moreover, I didn't see agenda in ML for today's meeting 16:15:29 dgilmore, it is not 16:15:37 https://fedoraproject.org/wiki/ReleaseEngineering/Meeting_Process 16:15:41 I didn't even know and I'm only here because ignatenkobrain reminded me 16:15:54 Son_Goku: whats not? 16:16:01 what the agenda was and stuff 16:16:10 hell, I didn't even know a meeting was today until he told me 16:16:13 well, let's don't discuss this now.. 16:16:22 anyway, brr 16:16:54 I think that all gives a quick overview of the f26 branching status and what has to be done this week 16:17:19 ignatenkobrain: what is your topic as I am not aware of it 16:17:41 dgilmore: the porting releng tools to DNF 16:17:53 #topic porting releng tools to DNF 16:17:53 unfortunately I was not able to get overview in ML 16:18:15 ignatenkobrain: there is no clear overview sadly 16:18:16 this is becoming more and more hot topic 16:18:25 some of it is waiting on dnf to be suitable 16:18:34 I tried to outline what I thought the current status was... it's not super clear what to do currently. 16:18:35 some of it is waiting on people to port tools 16:18:49 We still have to fully support yum as well as dnf 16:18:57 and it needs to be configurable 16:19:13 well, live and ARM images are using DNF now 16:19:26 ignatenkobrain: afaik with your working with lsedlar pungi is almost ready 16:19:38 Son_Goku: Live has used dnf for years 16:19:45 well over a year 16:19:51 we (Rust SIG) are started with packaging rust crates and we are just stuck because for our needs we MUST use rich deps 16:19:54 there's no other way 16:19:54 when we switched to livemedia-creator 16:20:19 and arm will be considered supported as soon as we drop using appliance-creator which we will be doing this week 16:20:39 appliance-creator is already using dnf, so what you decide to break by switching tools again is up to you 16:20:42 ignatenkobrain: there's a couple things that are to be ported to dnf that's on my TODO list that I hope to get to this week 16:20:48 dgilmore: is there any current news on koji signed repos? 16:20:55 ignatenkobrain: until we sort out changes for bodhi we are stuck with yum 16:21:08 dgilmore: what kind of changes? 16:21:09 nirik: there was movement on signed repos last week 16:21:11 ignatenkobrain: I'm going to add dnf support to pymultilib and then port mash to use that (hopefully) 16:21:17 ignatenkobrain: its not really known 16:21:23 ignatenkobrain: it could be using signed repos 16:21:28 it could be porting mash 16:21:29 maxamillion: that should resolve the bodhi issue 16:21:43 dgilmore: I need some faster solution 16:21:48 Son_Goku: it'll be a little time before all that stuff goes live in the infra, but hopefully 16:21:48 python-multilib while not directly relying on yum relies on yum 16:22:00 you can only pass it yum repo objects 16:22:22 ignatenkobrain: we have a goal of supporting dnf fully by f27 16:22:23 it needs to be refactored to support dbo too 16:22:45 dgilmore: f27 is quite far =( 16:23:02 Son_Goku: dbo? 16:23:02 Son_Goku: I know that lsedlar was looking at throwing the python-multilib code away and using something cleaner 16:23:09 dgilmore: that might be better 16:23:11 dgilmore: wait, what? 16:23:14 ignatenkobrain: we have 1 week to land changes for f26 16:23:19 maxamillion: I call dnf base objects "dbo" 16:23:38 dgilmore: should I not add dnf support to pymultilib and port mash over then? 16:23:41 Son_Goku: ah 16:24:03 when livecd-tools was ported from yum to dnf, that's what we moved to 16:24:08 maxamillion: basing a multilib python module on pungi's code not mash's as python-multilib does today 16:24:08 dnf.Base objects 16:24:13 dgilmore: so what would be some solution to implement within this week to support rich deps? 16:24:26 I can spend time on this and do work 16:24:29 ignatenkobrain: we do not have any capacity to do anything 16:24:46 dgilmore: you do have capacity -> ignatenkobrain 16:24:47 ignatenkobrain: all of the options need a lot of work. 16:24:57 ignatenkobrain: its all more than 1 weeks work 16:25:04 that suck 16:25:11 well, actually 16:25:12 ignatenkobrain: Ideally we would move to signed repos in koji 16:25:12 I don't think so 16:25:19 we can ignore koji for the most part 16:25:23 that needs signed repos to go in 16:25:26 dgilmore: I think I've seen code of signed repos... 16:25:27 pungi is almost done 16:25:33 and its needs bodhi to be changed to call and use it 16:25:36 and basically it's bloated with yum, no? 16:25:46 and if maxamillion and ignatenkobrain can knock out mash, we're golden for F26 16:25:51 but it would also need a python multilib module that supported yum and dnf 16:26:02 that knocks out all the hot paths that cause issues 16:26:16 ignatenkobrain: we could port mash to configurably use dnf or yum 16:26:26 dgilmore: that works for me as a temporary solution 16:26:51 ignatenkobrain: its a lot of throwaway work 16:26:59 not really 16:27:07 our long term plan has to been to kill mash entirely 16:27:17 that's been said for two years now 16:27:22 at this point, it's not happening 16:27:28 as it has been to kill appliance-tools and livecd-tools 16:27:37 things take time to get done. ;) 16:27:57 particullary when majorly under resourced 16:28:00 dgilmore: for me it doesn't matter because I want to make Fedora 1st class support of Rust and because of this thing I just can't 16:28:34 so even it's throwaway work, but if it will allow me to do my other job -- I'm fine with it 16:28:49 and besides, it's more example code for how to use DNF 16:28:53 which is always good 16:28:54 ignatenkobrain: to me the best thing is to have a multilib python module that can use yum or dnf 16:29:16 then have pungi, koji signed-repos and potentially mash use that 16:29:30 dgilmore: so 1) port pymultilib to support both yum and dnf 2) port mash to pymultilib ? 16:29:31 then we have just one code base for multilib implementation 16:29:45 ignatenkobrain, sounds like that's what he wants 16:29:56 yeah, that's what's on my TODO list 16:30:10 but it sounds like lsedlar is considering going a different route so I'll want to sync with him first 16:30:11 ignatenkobrain: sure, though talk to lsedlar as he was looking at basing a new module from pungi's multilib code and not mash's 16:30:32 well, and 3) port mash to dnf right? 16:30:39 (configurably) 16:30:58 nirik: I thought that once we port mash to pymultilib it will not depend on yum ? 16:31:00 nirik: 3 is implied by 1 and 2 16:31:01 nirik: mash would need some work to have a config option to use dnf 16:31:09 nirik: pymultilib would be dnf or yum on the backend and if mash it ported to that then why 3) ? 16:31:16 * dgilmore notes that it would only come into play for f26 and up 16:31:33 I'm not so sure of that, but ok 16:31:46 as dnf and yum give different results we must not change how multilib is done in stable releases 16:31:49 dgilmore: I don't care THAT much about older versions at this moment 16:32:18 but I care about upcoming release 16:32:21 which is f26 16:32:21 maxamillion: because I am not sure mash only uses yum for multilib... but I could be wrong. I recall it passing around a bunch of yum objects 16:32:27 ignatenkobrain: bodhi will be enabled for f26 on 2017-03-07 16:32:45 ignatenkobrain: we also need to get a dnf using pungi in use this week 16:32:54 and afaik that may not be possible yet 16:32:55 dgilmore: so basically we have 2 weeks to make all things done? 16:33:01 ignatenkobrain: 1 week 16:33:16 half a week 16:33:24 nirik: ah ok 16:33:27 3 is next fri 16:33:28 ignatenkobrain: before branching next tuesday 16:33:31 15minutes! :) sorry... 16:33:32 ah god 16:33:50 ignatenkobrain: you have four days to save the world :) 16:34:06 ignatenkobrain: though the mash changes could land a little after branching 16:34:16 but we really should get pungi in place this week 16:34:41 I don't really agree how it has been done, but doing it differently will take lot more time 16:35:11 ignatenkobrain: mboddu has been lookinga t whats needed to port everything to dnf 16:35:22 which is more than just this problem space 16:35:31 in order to move to python3 16:35:48 dgilmore: for this topic I'm interested only in paths which block us from rich deps 16:35:54 in which case we will hit different issues 16:36:48 ignatenkobrain: I think the only things that will be effected right now is composes and updates, however there is probably corner cases where things will break 16:36:59 like broken deps reports for branched and rawhide 16:37:17 composes == pungi, right? 16:37:24 updates == bodhi + mash + ... 16:37:31 which is generated by the spam-o-matic script in mash 16:37:37 * ignatenkobrain is not that experienced with releng pipeline 16:37:39 ignatenkobrain: correct 16:38:15 * dgilmore would like to wrap this up as we are over time 16:38:53 dgilmore: can you summarize please? 16:39:00 ;) 16:40:05 #info shortest path to using dnf is to have a multilib python module that supports yum and dnf, porting pungi and mash to using those. koji signed repos can then switch to and use it also 16:40:11 was doing so 16:40:29 sounds like a plan 16:40:45 #info that does not account for any other pieces where we have scripts and tools that use yum 16:40:59 #info so there may be unknown breakage if we changed over 16:41:13 spam-o-matic might not be too hard to move... it's just using repoclosure I think... 16:41:38 #info pungi would need to land this week and mash would need to get its way to the bodhi-backend before alpha freeze on 2017-03-07 16:41:47 nirik: its in mash 16:41:52 yep. 16:42:08 so i am assume it will handled in getting mash done 16:42:10 does pungi use repodiff? I think mash does also for it's reports 16:42:44 nirik: the diff is in compose-utils https://pagure.io/compose-utils 16:43:01 ok 16:43:26 I have been meaning to file an issue to make the compose repot more useful 16:43:35 ignatenkobrain: does that tell you what to work on then? 16:43:53 nirik: it does 16:44:02 (and BTW, many thanks for your time on things... it is appreciated. ) 16:44:07 ignatenkobrain: I would really like the long standing dnf usability bugs fixed first 16:44:18 but I guess that they will still languish 16:45:02 dgilmore: badger jsilhan about those :) 16:45:05 part of me does not want to migrate to dnf as we will lose valuable debuging information 16:45:11 wut 16:45:25 dnf 2.0 has nearly all of the info that yum presented 16:45:40 dnf 2.0 is my mind is not close to useful 16:45:50 hopefully, dnf 3.0 will be complete and proper rewrite 16:45:58 its a little better but it is still missing most of what we need 16:46:09 but I would not expect that to happen in near 1yr 16:46:38 mboddu: would you like to present some of your findings? 16:46:41 dgilmore: without complete redesign it's not possible to do some things =( 16:47:31 dgilmore: all of the tools and their respective yum usage is in taiga stories, let me find the stories 16:49:10 https://taiga.fedorainfracloud.org/project/acarter-fedora-docker-atomic-tooling/us/691 16:49:56 the spreadsheet has all the list of tools and yum usage 16:50:20 mboddu: can you please give a brief summary 16:51:37 dgilmore: sure 16:54:31 I created the above spreadsheet which has the releng tools that needs dnf porting. I looked at Pungi, Mash, Fedora-Packager and Releng scripts and listed out all the imports of yum. I am currently looking at releng scripts to port to dnf. 16:56:28 what spreadsheet? 16:56:50 lets really wrap up we are nearly 30 minutes over 16:57:08 dgilmore: sorry, I linked to the taiga ticket which has the link to the spreadsheet. https://mohanboddu.fedorapeople.org/yum-dnf.ods 16:57:58 mboddu: so the summary is what? 16:58:14 how much work are we looking at and what is the impact? 16:58:26 it may be some scripts we just no longer need 16:59:41 lets take this out of band 16:59:47 #topic open floor 16:59:58 does anyone have anything else they want to cover? 17:00:19 #info s390 builder situation looks to be getting resolved soon 17:01:00 what situation? 17:01:06 the merging into primary koji or something else? 17:02:49 dgilmore: awesome 17:03:14 thanks bowlofeggs 17:03:19 lets wrap up 17:03:33 #endmeeting