19:00:01 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
19:00:01 <nirik> #chair mboddu nirik smooge pingou mobrien
19:00:01 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting
19:00:01 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops
19:00:01 <zodbot> Meeting started Mon Nov  9 19:00:01 2020 UTC.
19:00:01 <zodbot> This meeting is logged and archived in a public location.
19:00:01 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:00:01 <zodbot> Current chairs: mboddu mobrien nirik pingou smooge
19:00:01 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:00:01 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily
19:00:02 <nirik> #topic Tickets needing review
19:00:03 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
19:00:42 * nirik is finishing another meeting.
19:02:53 <smooge> here
19:03:15 <nirik> just a min, almost done
19:03:21 <smooge> understood...
19:03:45 <nirik> ok.
19:03:58 <nirik> .ticket 9441
19:03:59 <mboddu> Okay, I am here now as well
19:04:00 <zodbot> nirik: Issue #9441: RFR: fedora-packages-static - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9441
19:04:14 <nirik> med/med/ops I guess...
19:04:29 <nirik> we need to rework our rfr pages, but we can work on this in stg in the mean time
19:04:31 <mboddu> +1
19:04:46 <mboddu> smooge: are you updating the tickets?
19:04:50 * mboddu can do it
19:05:14 <nirik> .ticket 9444
19:05:15 <zodbot> nirik: Issue #9444: EPEL repositories down - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9444
19:05:16 <mboddu> I guess its gonna be me then
19:05:18 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#9441: medium-gain, medium-trouble, and ops https://pagure.io/fedora-infrastructure/issue/9441
19:05:19 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#9441 https://pagure.io/fedora-infrastructure/issue/9441
19:05:47 <smooge> sorry I am.. just was typing in wrong window
19:05:51 <nirik> I don't know what the reporter was seeing here. I guess low/low/ops and keep it open for another day or so, then close
19:06:14 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#9444: low-gain and low-trouble https://pagure.io/fedora-infrastructure/issue/9444
19:06:15 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#9444 https://pagure.io/fedora-infrastructure/issue/9444
19:06:23 <mboddu> nirik: I think its related to https://pagure.io/releng/issue/9845 ?
19:06:59 <nirik> I doubt it?
19:07:03 <mboddu> As in like, the builds are failing to build
19:07:18 <mboddu> With some hash changes, that happened last week?
19:07:25 <mboddu> regen repo should fix it
19:07:27 <nirik> no
19:07:42 <nirik> releng 9845 is about building in koji.
19:08:00 <nirik> 9444 seems to be about epel repos in mirrormanager?
19:08:01 <smooge> mboddu, nirik I thought that issue is that epel has a package in it which is in the RHEL-8 trees and needs to be removed from EPEL
19:08:11 <nirik> and no, regen won't fix that issue.
19:08:18 <nirik> smooge: it has been.
19:08:22 <mboddu> Okay
19:08:22 <nirik> let me summarize.
19:08:53 <nirik> libzstd was in epel. Then they added it to rhel. We blocked it and removed it from epel.
19:09:03 <jschwart> smooge: great thanks again!!
19:09:04 <nirik> koji however didn't correctly start using the external repo version hash.
19:09:09 <nirik> it's wron g in the db.
19:09:22 <nirik> so, this is a koji bug... wanted koji upstream to look at it.
19:09:51 <mboddu> Okay, that makes sense, thanks nirik
19:10:00 <nirik> .ticket 9446
19:10:00 <smooge> thanks nirik
19:10:02 <zodbot> nirik: Issue #9446: openQA prod sometimes fails to respond or 503s, apparently a proxy/load balancer issue - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9446
19:10:18 <smooge> I can't find anything 'broke' or logging 503's for that one
19:10:20 <nirik> no idea on this one. But openqa is using apache as proxy, not haproxy
19:10:33 <nirik> so needs more digging I guess.
19:10:36 <nirik> med/med/ops?
19:10:40 <mboddu> +1
19:11:25 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#9446: "openQA prod sometimes fails to respond or 503s, apparently a proxy/load balancer issue" https://pagure.io/fedora-infrastructure/issue/9446#comment-700771
19:11:48 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#9446: medium-gain, medium-trouble, and ops https://pagure.io/fedora-infrastructure/issue/9446
19:11:49 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#9446 https://pagure.io/fedora-infrastructure/issue/9446
19:11:58 <smooge> ok done
19:11:59 <nirik> thats it for infra ones I guess... any releng ones?
19:12:52 <mboddu> Yup, we just discussed about it
19:12:59 <mboddu> .releng 9845
19:13:00 <zodbot> mboddu: Issue #9845: GenericError: hash changed for external rpm: libzstd-1.4.4-1.el8.s390x@epel8-build - releng - Pagure.io - https://pagure.io/releng/issue/9845
19:13:13 <mboddu> Close it with upstream issue?
19:13:26 <nirik> I'm going to ask upstream to look.
19:13:30 <nirik> done
19:13:35 <mboddu> Thanks
19:13:46 <nirik> I think we can fix it by direct db poking... but it would be nice to fix the error/issue in koji
19:13:59 <mboddu> So, med, med, ops?
19:14:04 <nirik> +1
19:14:04 <mboddu> Just give it something?
19:14:21 <mboddu> Thats all
19:14:40 <nirik> any other blockers, pr's, etc? There are some PR's in infra ansible I was gonna elook at when I get time.
19:14:51 <nirik> but my today has been consumed by meetings. ;(
19:15:00 <mboddu> None from me
19:15:07 <mboddu> I want to clean up some tickets as well
19:15:21 <nirik> also, we need to update staging today?
19:15:24 <mboddu> But I have to work on deploying ODCS in centos infra...
19:15:33 <nirik> and start working on non outage causing updates before wed.
19:15:46 <nirik> and I am out on friday again. ;)
19:15:59 <nirik> #info nirik out friday
19:16:05 <nirik> #info outage for updates on wed
19:16:12 <nirik> #info outage in stg today for updates
19:16:56 <nirik> Anything else? or shall we close out?
19:17:27 <nirik> #endmeeting