16:00:48 <mboddu> #startmeeting RELENG (2019-11-21) 16:00:48 <zodbot> Meeting started Wed Nov 20 16:00:48 2019 UTC. 16:00:48 <zodbot> This meeting is logged and archived in a public location. 16:00:48 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:48 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:48 <zodbot> The meeting name has been set to 'releng_(2019-11-21)' 16:00:48 <mboddu> #meetingname releng 16:00:48 <zodbot> The meeting name has been set to 'releng' 16:00:48 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec 16:00:48 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz 16:00:48 <mboddu> #topic init process 16:00:53 <nirik> morning. 16:01:02 * nirik needs more coffee, back in a min 16:02:16 * mboddu needs something, but dont know what, will go and raid the pantry area, be back in a min as well 16:03:44 <mboddu> And I am back 16:05:35 <nirik> me too 16:05:35 <smooge> hello 16:06:52 <mboddu> Okay, lets get started 16:06:58 <mboddu> #topic Rawhide composes 16:07:25 <mboddu> nirik: Any chance you get a koji build with the patch mentioned in https://pagure.io/releng/failed-composes/issue/562#comment-612417 ? 16:07:31 <mboddu> pungi patch is not required 16:07:42 * nirik has been in a meeting, looks 16:07:50 <mboddu> And I created a pr for pungi config changes - https://pagure.io/pungi-fedora/pull-request/799 16:07:57 <mboddu> nirik: Sure, np 16:08:02 * mboddu waves at smooge 16:08:35 <nirik> yeah, but... we are still on 1.18 in prod right now. 16:08:52 <nirik> I guess I could just schedule the 1.19 upgrade for later today and add this in? 16:09:36 <mboddu> nirik: Whenever it works for you, just let me know when you deployed it and then I can rerun another compose 16:09:59 <mboddu> nirik: Also, is there any doc on how to upgrade koji? Esp when there are db changes? 16:10:03 <nirik> ok. I'll send out a outage announce soon then 16:10:10 <nirik> we have a playbook. ;) 16:10:12 <mboddu> Thanks nirik 16:10:15 <mboddu> nirik++ 16:10:15 <nirik> it has a bug tho... 16:10:41 <nirik> it runs the db migrations as 'postgres' and sometimes leaves new tables as owned by postgres instead of koji 16:10:52 <nirik> but it handles the db and stopping everything, etc. 16:10:54 <mboddu> Haha, my reaction was like - "OH, its awesome, all I need to run a playbook, thats all and then duh, obviously it has bugs"... 16:11:26 <mboddu> Okay, its a minor thing then 16:11:45 <nirik> not too bad. I can try and fix it. 16:12:25 <mboddu> #info nirik will try to upgrade koji to 1.19 with the fix to remove --title option and then we will fire another compose with changes to pungi configs 16:12:36 <mboddu> #info koji fix - https://pagure.io/koji/pull-request/1781 16:12:56 <mboddu> #info pungi configs change - https://pagure.io/pungi-fedora/pull-request/799 16:13:05 <mboddu> nirik: Not a big deal 16:13:21 <mboddu> Dont worry about it if its hard to debug 16:13:28 * dustymabe waves 16:14:06 <nirik> well, it causes issues, but it's easy to fix. 16:15:02 * mboddu waves back at dustymabe 16:15:07 <mboddu> Okay, moving on 16:15:23 <mboddu> Okay, before discussing this, I need to create a ticket for it 16:15:36 * mboddu will be right back 16:15:53 <nirik> ok 16:19:12 <mboddu> #topic #9041 owner-sync-pagure is busted 16:19:17 <mboddu> #link https://pagure.io/releng/issue/9041 16:19:35 <mboddu> So, three problems here 16:19:40 <nirik> huh. 16:19:41 <mboddu> 1. The script is py2 16:19:47 <mboddu> 2. It uses fedmsg 16:19:56 <mboddu> 3. Random auth failures to koji 16:19:58 <nirik> where does this run? 16:20:03 <mboddu> bodhi-backend01 16:20:13 <nirik> 1 and 2 aren't critical 16:21:27 <mboddu> Yup, but 3rd is what creating the problems 16:21:38 <nirik> hum. 16:22:29 <nirik> which ktab does it use? 16:22:45 <nirik> /etc/krb5.bodhi_bodhi.fedoraproject.org.keytab ? 16:22:52 <mboddu> nirik: krb5.bodhi_bodhi.fedoraproject.org.keytab 16:22:54 <mboddu> Yup 16:24:05 <nirik> huh, well, it runs as root, so it's not perms. 16:24:47 <mboddu> Nope, its not perms 16:26:13 <nirik> huh, wild thing to try: update koji? 16:26:39 <mboddu> May be, but I am out of ideas 16:26:46 <mboddu> So, we could give it a shot 16:27:59 <nirik> sure. we can look more after meeting? 16:28:20 <mboddu> Okay 16:29:08 <mboddu> #info We will try to debug it asap and we think that we upgrading koji might help as well 16:30:44 <mboddu> dustymabe: Do you have anything? 16:31:37 <mboddu> I might have to cut this meeting short due to other commitments 16:31:42 <mboddu> While waiting for Dusty 16:31:46 <nirik> ah, I had just 2 things... 16:31:48 <mboddu> #topic Modularity 16:31:57 <nirik> (for open floor) 16:32:03 <mboddu> nirik: I have time, may be in 15 - 20 min 16:32:10 <nirik> ok 16:32:28 <mboddu> #info Module builds are working in epel8 16:32:46 <mboddu> As per tdawson: 16:32:50 <mboddu> "[11:00:27] <tdawson> mboddu: Talking of stage ... my multi step module built great there yesterday. I was imporessed." 16:32:57 <mboddu> So, thats good 16:33:03 <nirik> cool 16:33:31 <mboddu> Currently, we are blocked on https://github.com/fedora-infra/bodhi/issues/3779 16:33:43 <mboddu> For which the fix is merged couple of hours back 16:33:50 <mboddu> I am waiting on cverna to deploy it in stg 16:34:14 <mboddu> Once, that is done and tested, we are ready for prod 16:35:04 <nirik> nice! congrats 16:35:19 <mboddu> #info modules in epel8 are building fine and we got a fix for bodhi issue with long update id's, hoping that we will be able to deploy it in prod by early next week 16:35:35 <mboddu> And coming to ursa prime 16:35:59 <mboddu> nirik: Can you shed us some light on fesco meeting and outcome? 16:36:10 <mboddu> I think everyone is okay with testing it in stg for rawhide, right? 16:37:11 <nirik> yes, and it's ok to test in prod, but only with 2 specific default modules 16:37:36 <mboddu> Okay, so, is there a deadline for it? 16:37:45 <nirik> anything past that is still on hold. 16:37:51 <nirik> not that I know of. 16:38:32 <mboddu> Okay, my plan is to start working on it, once the epel8 module stuff is done, or at least its at a place that I dont need to get involved as much 16:39:05 <mboddu> So, may be this this week sometime, once I get bodhi going or next week (hopefully before thanks giving) 16:39:47 <mboddu> #info ursa prime will be tested in stage before end of next week 16:40:07 <mboddu> Anything else on modularity? 16:40:10 <nirik> sounds good. 16:41:12 <mboddu> Okay, moving on 16:41:17 <mboddu> #topic Open Floor 16:41:20 <mboddu> nirik: You are up 16:41:45 <nirik> ok, first I thought i would share where I am with the builder upgrades... 16:41:54 <nirik> s390x is all moved to f31. 16:42:35 <nirik> ppc64le 01-09 are all f31 (and ppc9-01 host they are on) 16:42:56 <nirik> aarch64-01- 03 and armv7-01 to 03 are f31 and on new hw. 16:43:12 <nirik> I'm trying to move the x86_64 stuff now... 16:43:21 <mboddu> Oh, awesome 16:43:23 <nirik> hopefully I can get everything done by the end of the week. 16:43:30 <nirik> (I am out next week) 16:44:08 <mboddu> #info nirik moved s390x, ppc64le 01-09, aarch64-01- 03 and armv7-01 to 03 to f31, he is working on x86_64 now and hopefully everything will be moved by end of this week 16:44:17 <nirik> and the other item: 16:44:43 <mboddu> #undo 16:44:43 <zodbot> Removing item from minutes: INFO by mboddu at 16:44:08 : nirik moved s390x, ppc64le 01-09, aarch64-01- 03 and armv7-01 to 03 to f31, he is working on x86_64 now and hopefully everything will be moved by end of this week 16:44:48 <mboddu> #info nirik moved s390x, ppc64le 01-09, aarch64-01- 03 and armv7-01 to 03 to f31, he is working on x86_64 now and hopefully everything will be moved to f31 by end of this week 16:44:57 <nirik> any objections to me closing the f31 change tickets in releng ? https://pagure.io/releng/issues?status=Open&tags=change-ack or so 16:45:15 <nirik> and perhaps we should try and schedule another session to go over backlog? like friday ? 16:46:02 <mboddu> Sure, you can close them or I can 16:46:11 <mboddu> +1 on backlog meeting for Fri 16:47:13 <nirik> or week after next I guess... 16:47:22 <nirik> or I could poke at it in my spare time. :) 16:47:25 <nirik> anyhow, thats all I had 16:48:26 <mboddu> I closed the f31 tickets 16:48:45 <mboddu> nirik: Let me know and we can work on it 16:49:04 <nirik> cool. whats your friday schedule like? 16:49:21 <dustymabe> hey mboddu nirik sorry - running another meeting right now 16:49:26 <dustymabe> let me see what I have 16:49:29 <mboddu> nirik: I am free on Fri 16:49:35 <mboddu> Except for 1 meeting 16:49:49 <dustymabe> here is one i'm interested in discussing: https://pagure.io/fedora-infrastructure/issue/8370 16:50:19 <nirik> I'm free after 8:30am... (well, of meetings anyhow, I have about 10000^232 things to do). How about 10am my time 1pm yours? 16:50:27 <dustymabe> let me know when you're ready to discuss 16:51:04 <mboddu> nirik: That will work for me 16:51:22 <mboddu> dustymabe: Thats nirik's playground 16:51:30 * mboddu will sit back and watch 16:51:32 <mboddu> :) 16:51:44 <nirik> I'm not sure what to add aside from what I put there. ;) but happy to discuss... 16:51:50 <dustymabe> nirik: do you know if we can get access to multi-arch VMs (preferrably beefy) \ 16:52:01 <dustymabe> i.e. ones you can hand over to us and we manage? 16:52:28 <nirik> I don;t know that we have any capacity to do that... which aches? 16:53:09 <nirik> our other arch stuff is allocated to koji builders... 16:53:24 <dustymabe> nirik: aarch64 and ppc64le primarily 16:53:32 <dustymabe> but s390x is always knocking on our door 16:53:41 <nirik> we may be able to get some things, but we need to know to plan that... we don't have spare capacity coming out our ears. 16:54:35 <dustymabe> nirik: do you know if centos CI has multi-arch hardware? 16:54:39 <nirik> there's no way you could just do this in koji? 16:54:58 <nirik> I don't think they do, but there might have been plans to add that? 16:55:08 <nirik> would have to talk to bstinson probibly there. 16:55:09 <dustymabe> nirik: I think it would be easier all around if we don't have to re-implement in koji (i.e. less work on us and less work on koji team) 16:55:43 <nirik> can you be any more specific on what resources you need? does 'beefy' have a number of cpus/memory, etc? 16:56:44 <nirik> and fedora? rhel? fcos? for os? 16:57:11 <dustymabe> nirik: we could start with something like 2 cpus and 8G, but would prefer something like 4 cpus/ 16G 16:57:15 <nirik> anyhow, if you could update the ticket with that we can try and find capacity 16:57:29 <mboddu> nirik: I got to run, can you end the meeting when its done? 16:57:31 <dustymabe> also for OS, fedora would work. 16:58:04 <nirik> and you would want to completely manage them? or want us to do os updates, initial ansible install? 16:58:39 <dustymabe> nirik: I think we could manage them 16:58:53 <dustymabe> we don't intend to make your life harder 16:59:15 <dustymabe> i'll add this information to the ticket and we'll see what we can come up with 16:59:17 <dustymabe> sound good? 16:59:41 <smooge> if you are ok with disappointment :) 16:59:41 <nirik> sure. in some ways tho you managing them is harder for us... 16:59:58 <dustymabe> nirik: in that case, i'm sure we could let you run updates for us 17:00:08 <nirik> ie, we need to update the virthost, we have to talk with you to know when you can let us reboot, etc. 17:00:43 <dustymabe> yeah. the goal is that these machines don't have any permanent state 17:00:53 <dustymabe> so hopefully losing them won't be a big deal 17:01:39 <nirik> would aws work for the aarch64/x86_64 versions? but then I guess it would make ppc64le 'different' 17:02:41 <dustymabe> i think we don't need an x86_64 VM (already got that covered) 17:02:49 <dustymabe> AWS won't work because we needed nested virt 17:03:04 <dustymabe> sorry, probably should have mentioned that 17:04:02 <nirik> oh, thats pretty important... 17:04:21 <dustymabe> i'll put all of this in the ticket 17:04:34 <dustymabe> thanks for starting the conversation with me at least :) 17:04:36 <nirik> aarch64 on aarch64? or armv7 on aarch64 ? 17:05:07 <dustymabe> like on like 17:05:20 <nirik> ok. 17:05:28 <dustymabe> so if we have an aarch64 VM we would only run an aarch64 nested VM 17:05:42 <dustymabe> if we could get access to bare metal that would be great too, but I doubt anyone can give us that 17:05:45 <dustymabe> maybe centos CI 17:05:46 <nirik> that narrows down the hw we could use. ;) 17:05:49 <dustymabe> I'll ping brian in the ticket 17:05:55 <nirik> sounds good. 17:06:01 <dustymabe> thanks again! 17:06:09 <dustymabe> that's it from me for now 18:11:58 <mboddu> #endmeeting