16:00:48 <mboddu> #startmeeting RELENG (2019-11-21)
16:00:48 <zodbot> Meeting started Wed Nov 20 16:00:48 2019 UTC.
16:00:48 <zodbot> This meeting is logged and archived in a public location.
16:00:48 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:48 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:48 <zodbot> The meeting name has been set to 'releng_(2019-11-21)'
16:00:48 <mboddu> #meetingname releng
16:00:48 <zodbot> The meeting name has been set to 'releng'
16:00:48 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec
16:00:48 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz
16:00:48 <mboddu> #topic init process
16:00:53 <nirik> morning.
16:01:02 * nirik needs more coffee, back in a min
16:02:16 * mboddu needs something, but dont know what, will go and raid the pantry area, be back in a min as well
16:03:44 <mboddu> And I am back
16:05:35 <nirik> me too
16:05:35 <smooge> hello
16:06:52 <mboddu> Okay, lets get started
16:06:58 <mboddu> #topic Rawhide composes
16:07:25 <mboddu> nirik: Any chance you get a koji build with the patch mentioned in https://pagure.io/releng/failed-composes/issue/562#comment-612417 ?
16:07:31 <mboddu> pungi patch is not required
16:07:42 * nirik has been in a meeting, looks
16:07:50 <mboddu> And I created a pr for pungi config changes - https://pagure.io/pungi-fedora/pull-request/799
16:07:57 <mboddu> nirik: Sure, np
16:08:02 * mboddu waves at smooge
16:08:35 <nirik> yeah, but... we are still on 1.18 in prod right now.
16:08:52 <nirik> I guess I could just schedule the 1.19 upgrade for later today and add this in?
16:09:36 <mboddu> nirik: Whenever it works for you, just let me know when you deployed it and then I can rerun another compose
16:09:59 <mboddu> nirik: Also, is there any doc on how to upgrade koji? Esp when there are db changes?
16:10:03 <nirik> ok. I'll send out a outage announce soon then
16:10:10 <nirik> we have a playbook. ;)
16:10:12 <mboddu> Thanks nirik
16:10:15 <mboddu> nirik++
16:10:15 <nirik> it has a bug tho...
16:10:41 <nirik> it runs the db migrations as 'postgres' and sometimes leaves new tables as owned by postgres instead of koji
16:10:52 <nirik> but it handles the db and stopping everything, etc.
16:10:54 <mboddu> Haha, my reaction was like - "OH, its awesome, all I need to run a playbook, thats all and then duh, obviously it has bugs"...
16:11:26 <mboddu> Okay, its a minor thing then
16:11:45 <nirik> not too bad. I can try and fix it.
16:12:25 <mboddu> #info nirik will try to upgrade koji to 1.19 with the fix to remove --title option and then we will fire another compose with changes to pungi configs
16:12:36 <mboddu> #info koji fix - https://pagure.io/koji/pull-request/1781
16:12:56 <mboddu> #info pungi configs change - https://pagure.io/pungi-fedora/pull-request/799
16:13:05 <mboddu> nirik: Not a big deal
16:13:21 <mboddu> Dont worry about it if its hard to debug
16:13:28 * dustymabe waves
16:14:06 <nirik> well, it causes issues, but it's easy to fix.
16:15:02 * mboddu waves back at dustymabe
16:15:07 <mboddu> Okay, moving on
16:15:23 <mboddu> Okay, before discussing this, I need to create a ticket for it
16:15:36 * mboddu will be right back
16:15:53 <nirik> ok
16:19:12 <mboddu> #topic #9041 owner-sync-pagure is busted
16:19:17 <mboddu> #link https://pagure.io/releng/issue/9041
16:19:35 <mboddu> So, three problems here
16:19:40 <nirik> huh.
16:19:41 <mboddu> 1. The script is py2
16:19:47 <mboddu> 2. It uses fedmsg
16:19:56 <mboddu> 3. Random auth failures to koji
16:19:58 <nirik> where does this run?
16:20:03 <mboddu> bodhi-backend01
16:20:13 <nirik> 1 and 2 aren't critical
16:21:27 <mboddu> Yup, but 3rd is what creating the problems
16:21:38 <nirik> hum.
16:22:29 <nirik> which ktab does it use?
16:22:45 <nirik> /etc/krb5.bodhi_bodhi.fedoraproject.org.keytab ?
16:22:52 <mboddu> nirik: krb5.bodhi_bodhi.fedoraproject.org.keytab
16:22:54 <mboddu> Yup
16:24:05 <nirik> huh, well, it runs as root, so it's not perms.
16:24:47 <mboddu> Nope, its not perms
16:26:13 <nirik> huh, wild thing to try: update koji?
16:26:39 <mboddu> May be, but I am out of ideas
16:26:46 <mboddu> So, we could give it a shot
16:27:59 <nirik> sure. we can look more after meeting?
16:28:20 <mboddu> Okay
16:29:08 <mboddu> #info We will try to debug it asap and we think that we upgrading koji might help as well
16:30:44 <mboddu> dustymabe: Do you have anything?
16:31:37 <mboddu> I might have to cut this meeting short due to other commitments
16:31:42 <mboddu> While waiting for Dusty
16:31:46 <nirik> ah, I had just 2 things...
16:31:48 <mboddu> #topic Modularity
16:31:57 <nirik> (for open floor)
16:32:03 <mboddu> nirik: I have time, may be in 15 - 20 min
16:32:10 <nirik> ok
16:32:28 <mboddu> #info Module builds are working in epel8
16:32:46 <mboddu> As per tdawson:
16:32:50 <mboddu> "[11:00:27] <tdawson> mboddu: Talking of stage ... my multi step module built great there yesterday.  I was imporessed."
16:32:57 <mboddu> So, thats good
16:33:03 <nirik> cool
16:33:31 <mboddu> Currently, we are blocked on https://github.com/fedora-infra/bodhi/issues/3779
16:33:43 <mboddu> For which the fix is merged couple of hours back
16:33:50 <mboddu> I am waiting on cverna to deploy it in stg
16:34:14 <mboddu> Once, that is done and tested, we are ready for prod
16:35:04 <nirik> nice! congrats
16:35:19 <mboddu> #info modules in epel8 are building fine and we got a fix for bodhi issue with long update id's, hoping that we will be able to deploy it in prod by early next week
16:35:35 <mboddu> And coming to ursa prime
16:35:59 <mboddu> nirik: Can you shed us some light on fesco meeting and outcome?
16:36:10 <mboddu> I think everyone is okay with testing it in stg for rawhide, right?
16:37:11 <nirik> yes, and it's ok to test in prod, but only with 2 specific default modules
16:37:36 <mboddu> Okay, so, is there a deadline for it?
16:37:45 <nirik> anything past that is still on hold.
16:37:51 <nirik> not that I know of.
16:38:32 <mboddu> Okay, my plan is to start working on it, once the epel8 module stuff is done, or at least its at a place that I dont need to get involved as much
16:39:05 <mboddu> So, may be this this week sometime, once I get bodhi going or next week (hopefully before thanks giving)
16:39:47 <mboddu> #info ursa prime will be tested in stage before end of next week
16:40:07 <mboddu> Anything else on modularity?
16:40:10 <nirik> sounds good.
16:41:12 <mboddu> Okay, moving on
16:41:17 <mboddu> #topic Open Floor
16:41:20 <mboddu> nirik: You are up
16:41:45 <nirik> ok, first I thought i would share where I am with the builder upgrades...
16:41:54 <nirik> s390x is all moved to f31.
16:42:35 <nirik> ppc64le 01-09 are all f31 (and ppc9-01 host they are on)
16:42:56 <nirik> aarch64-01- 03 and armv7-01 to 03 are f31 and on new hw.
16:43:12 <nirik> I'm trying to move the x86_64 stuff now...
16:43:21 <mboddu> Oh, awesome
16:43:23 <nirik> hopefully I can get everything done by the end of the week.
16:43:30 <nirik> (I am out next week)
16:44:08 <mboddu> #info nirik moved s390x, ppc64le 01-09, aarch64-01- 03 and armv7-01 to 03 to f31, he is working on x86_64 now and hopefully everything will be moved by end of this week
16:44:17 <nirik> and the other item:
16:44:43 <mboddu> #undo
16:44:43 <zodbot> Removing item from minutes: INFO by mboddu at 16:44:08 : nirik moved s390x, ppc64le 01-09, aarch64-01- 03 and armv7-01 to 03 to f31, he is working on x86_64 now and hopefully everything will be moved by end of this week
16:44:48 <mboddu> #info nirik moved s390x, ppc64le 01-09, aarch64-01- 03 and armv7-01 to 03 to f31, he is working on x86_64 now and hopefully everything will be moved to f31 by end of this week
16:44:57 <nirik> any objections to me closing the f31 change tickets in releng ? https://pagure.io/releng/issues?status=Open&tags=change-ack or so
16:45:15 <nirik> and perhaps we should try and schedule another session to go over backlog? like friday ?
16:46:02 <mboddu> Sure, you can close them or I can
16:46:11 <mboddu> +1 on backlog meeting for Fri
16:47:13 <nirik> or week after next I guess...
16:47:22 <nirik> or I could poke at it in my spare time. :)
16:47:25 <nirik> anyhow, thats all I had
16:48:26 <mboddu> I closed the f31 tickets
16:48:45 <mboddu> nirik: Let me know and we can work on it
16:49:04 <nirik> cool. whats your friday schedule like?
16:49:21 <dustymabe> hey mboddu nirik sorry - running another meeting right now
16:49:26 <dustymabe> let me see what I have
16:49:29 <mboddu> nirik: I am free on Fri
16:49:35 <mboddu> Except for 1 meeting
16:49:49 <dustymabe> here is one i'm interested in discussing: https://pagure.io/fedora-infrastructure/issue/8370
16:50:19 <nirik> I'm free after 8:30am... (well, of meetings anyhow, I have about 10000^232 things to do). How about 10am my time 1pm yours?
16:50:27 <dustymabe> let me know when you're ready to discuss
16:51:04 <mboddu> nirik: That will work for me
16:51:22 <mboddu> dustymabe: Thats nirik's playground
16:51:30 * mboddu will sit back and watch
16:51:32 <mboddu> :)
16:51:44 <nirik> I'm not sure what to add aside from what I put there. ;) but happy to discuss...
16:51:50 <dustymabe> nirik: do you know if we can get access to multi-arch VMs (preferrably beefy) \
16:52:01 <dustymabe> i.e. ones you can hand over to us and we manage?
16:52:28 <nirik> I don;t know that we have any capacity to do that... which aches?
16:53:09 <nirik> our other arch stuff is allocated to koji builders...
16:53:24 <dustymabe> nirik: aarch64 and ppc64le primarily
16:53:32 <dustymabe> but s390x is always knocking on our door
16:53:41 <nirik> we may be able to get some things, but we need to know to plan that... we don't have spare capacity coming out our ears.
16:54:35 <dustymabe> nirik: do you know if centos CI has multi-arch hardware?
16:54:39 <nirik> there's no way you could just do this in koji?
16:54:58 <nirik> I don't think they do, but there might have been plans to add that?
16:55:08 <nirik> would have to talk to bstinson probibly there.
16:55:09 <dustymabe> nirik: I think it would be easier all around if we don't have to re-implement in koji (i.e. less work on us and less work on koji team)
16:55:43 <nirik> can you be any more specific on what resources you need? does 'beefy' have a number of cpus/memory, etc?
16:56:44 <nirik> and fedora? rhel? fcos? for os?
16:57:11 <dustymabe> nirik: we could start with something like 2 cpus and 8G, but would prefer something like 4 cpus/ 16G
16:57:15 <nirik> anyhow, if you could update the ticket with that we can try and find capacity
16:57:29 <mboddu> nirik: I got to run, can you end the meeting when its done?
16:57:31 <dustymabe> also for OS, fedora would work.
16:58:04 <nirik> and you would want to completely manage them? or want us to do os updates, initial ansible install?
16:58:39 <dustymabe> nirik: I think we could manage them
16:58:53 <dustymabe> we don't intend to make your life harder
16:59:15 <dustymabe> i'll add this information to the ticket and we'll see what we can come up with
16:59:17 <dustymabe> sound good?
16:59:41 <smooge> if you are ok with disappointment :)
16:59:41 <nirik> sure. in some ways tho you managing them is harder for us...
16:59:58 <dustymabe> nirik: in that case, i'm sure we could let you run updates for us
17:00:08 <nirik> ie, we need to update the virthost, we have to talk with you to know when you can let us reboot, etc.
17:00:43 <dustymabe> yeah. the goal is that these machines don't have any permanent state
17:00:53 <dustymabe> so hopefully losing them won't be a big deal
17:01:39 <nirik> would aws work for the aarch64/x86_64 versions? but then I guess it would make ppc64le 'different'
17:02:41 <dustymabe> i think we don't need an x86_64 VM (already got that covered)
17:02:49 <dustymabe> AWS won't work because we needed nested virt
17:03:04 <dustymabe> sorry, probably should have mentioned that
17:04:02 <nirik> oh, thats pretty important...
17:04:21 <dustymabe> i'll put all of this in the ticket
17:04:34 <dustymabe> thanks for starting the conversation with me at least :)
17:04:36 <nirik> aarch64 on aarch64? or armv7 on aarch64 ?
17:05:07 <dustymabe> like on like
17:05:20 <nirik> ok.
17:05:28 <dustymabe> so if we have an aarch64 VM we would only run an aarch64 nested VM
17:05:42 <dustymabe> if we could get access to bare metal that would be great too, but I doubt anyone can give us that
17:05:45 <dustymabe> maybe centos CI
17:05:46 <nirik> that narrows down the hw we could use. ;)
17:05:49 <dustymabe> I'll ping brian in the ticket
17:05:55 <nirik> sounds good.
17:06:01 <dustymabe> thanks again!
17:06:09 <dustymabe> that's it from me for now
18:11:58 <mboddu> #endmeeting