18:00:01 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
18:00:01 <zodbot> Meeting started Mon Sep 21 18:00:01 2020 UTC.
18:00:01 <zodbot> This meeting is logged and archived in a public location.
18:00:01 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:01 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:01 <nirik> #chair mboddu nirik smooge pingou mobrien
18:00:01 <zodbot> Current chairs: mboddu mobrien nirik pingou smooge
18:00:01 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting
18:00:01 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops
18:00:01 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:02 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily
18:00:03 <nirik> #topic Tickets needing review
18:00:04 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
18:00:10 <mboddu> Hello
18:00:30 <nirik> hey mboddu
18:00:37 <mboddu> I can update the tickets
18:00:42 <nirik> okey dokey
18:00:50 <nirik> .ticket 9332
18:00:51 <zodbot> nirik: Issue #9332: Stage Wiki login redirects to production ipsilon - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9332
18:00:57 <smooge> hola
18:01:41 <nirik> this seems like a config bug... not sure if it's in the auth path or what tho
18:01:43 <nirik> hey smooge
18:01:58 <smooge> me either..
18:02:17 <mboddu> No idea here as well
18:02:32 <smooge> High Gain, Med Trouble, Ops ?
18:03:04 <mboddu> +1
18:03:26 <nirik> actually...
18:03:29 <fm-admin> pagure.issue.tag.added -- mohanboddu tagged ticket fedora-infrastructure#9332: high-gain, medium-trouble, and ops https://pagure.io/fedora-infrastructure/issue/9332
18:03:30 <fm-admin> pagure.issue.edit -- mohanboddu edited the priority fields of ticket fedora-infrastructure#9332 https://pagure.io/fedora-infrastructure/issue/9332
18:03:46 <nirik> I think it might be due to wiki01.stg not being in stg group. I added it, but possibly didn't re-run the playbook.
18:04:13 <nirik> but +1
18:04:28 <nirik> .ticket 9335
18:04:28 <zodbot> nirik: Issue #9335: sundries01 is overloaded - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9335
18:04:46 <nirik> I think this is just the limits. +1 to applying smooge's patch...
18:05:29 <mboddu> I agree with https://pagure.io/fedora-infrastructure/issue/9335#comment-687487
18:05:30 <nirik> Med gain / Low trouble / Ops?
18:05:39 <mboddu> +1
18:06:02 <smooge> yeah I think this will help. if not, the next is to make the rsync script sleep random up to 120 seconds or so
18:06:58 <nirik> thats all the infra ones... any releng ones?
18:07:04 <mboddu> Yup
18:07:18 <fm-admin> pagure.issue.tag.added -- mohanboddu tagged ticket fedora-infrastructure#9335: low-trouble, medium-gain, and ops https://pagure.io/fedora-infrastructure/issue/9335
18:07:19 <fm-admin> pagure.issue.edit -- mohanboddu edited the priority fields of ticket fedora-infrastructure#9335 https://pagure.io/fedora-infrastructure/issue/9335
18:07:47 <mboddu> .releng 9762
18:07:48 <zodbot> mboddu: Issue #9762: Error creating new scm repo for python-bokeh: "The PDC branch already exists" - releng - Pagure.io - https://pagure.io/releng/issue/9762
18:08:07 <nirik> huh, do we know why it happened?
18:08:09 <mboddu> I need to fix the pdc entries and rerun the fedscm-admin
18:08:56 <mboddu> nirik: Sometimes pdc entries get created but not the git repo and then fedscm-admin thinks the repo already exists
18:09:20 <nirik> is there a way we could just create the git repo in that case?
18:09:57 <mboddu> Yes, pingou can do it or we can remove the pdc entry and process the ticket
18:10:01 * mboddu checks something
18:10:17 <nirik> ok. Just thinking longer term how to avoid having to manually deal with these.
18:11:29 <smooge> make a toddler which does it anyway?
18:11:32 <mboddu> So, med gain, low risk, ops?
18:11:41 <nirik> ack
18:12:11 <mboddu> toddler depends on messages, this is more of processing the ticket
18:12:41 <nirik> I was thinking of changing fedscm-admin to see this case and just make the git repo/finish processing.
18:13:43 <nirik> anyhow, don't need to solve it here now.
18:13:46 <mboddu> Yes, we could do that
18:14:01 <nirik> any other blockers, freeze breaks or tickets to discuss?
18:14:02 <mboddu> .releng 9763
18:14:03 <zodbot> mboddu: Issue #9763: newRepo "outage"/failures earlier 2020-09-21 - releng - Pagure.io - https://pagure.io/releng/issue/9763
18:14:11 <nirik> yeah, this is weird.
18:14:13 <mboddu> nirik: What is the status on the nfs mounts?
18:14:39 <nirik> some of the builders just didn't seem to have their nfs mounts anymore. Nothing dmesg or journal, just wasn't there.
18:14:59 <nirik> I am running playbooks which should bring them back
18:15:36 <nirik> the buildvm one is still running.
18:15:39 <mboddu> Okay
18:16:01 <mboddu> Thanks nirik for the update
18:16:07 <mboddu> So, med,med,ops?
18:16:16 <nirik> mah, let me just mass mount on them...
18:16:19 <nirik> ack, +1
18:17:07 <mboddu> Thats all I got
18:17:23 <nirik> I have a few info items...
18:17:57 <nirik> #info s390x builders i/o should be MUCH better now. cpu is still somewhat constrained, but there's nothing we can do about that. Please report issues to the s390x ticket.
18:18:31 <nirik> #info staging is mostly installed, see list for exceptions.
18:18:59 <nirik> I think thats all I had.
18:19:07 <nirik> anything else? if not, will close in a min
18:19:58 <mboddu> #info No RC request yet, we have a new blocker accepted and couple of proposed blockers (which might be voted as FE's)
18:21:41 <nirik> #info one of the s390x builders sshfs mount has hung, delaying todays f33 compose, should be done soon now.
18:22:32 <nirik> thanks everyone
18:22:37 <nirik> #endmeeting