18:06:36 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
18:06:36 <zodbot> Meeting started Wed Jul  7 18:06:36 2021 UTC.
18:06:36 <zodbot> This meeting is logged and archived in a public location.
18:06:36 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:06:36 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:06:37 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:06:37 <nirik> #chair mboddu nirik pingou mobrien nb
18:06:37 <zodbot> Current chairs: mboddu mobrien nb nirik pingou
18:06:37 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting
18:06:37 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:06:37 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops
18:06:37 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily
18:06:37 <nirik> #info reminder: speak up if you want to work on a ticket!
18:06:39 <nirik> #topic Tickets needing review
18:06:41 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
18:06:43 <nirik> oops. got sidetracked
18:07:15 <darknao> hi
18:07:42 <lenkaseg> hello
18:07:52 <nirik> morning
18:08:48 <nirik> .ticket 10072
18:08:49 <zodbot> nirik: Issue #10072: Builds fail on buildvm-s390x-20.s390.fedoraproject.org: Couldn't resolve host name for http://kojipkgs01.fedoraproject.org/repos/f35-build/3809562/s390x/repodata/repomd.xml - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10072
18:09:04 <nirik> med/med/ops... and I was looking into this one just before this meeting. ;)
18:09:32 <nirik> .ticket 10074
18:09:33 <zodbot> nirik: Issue #10074: eduvpn group account - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10074
18:09:40 <nirik> I can just do this one right now.
18:09:59 <mboddu> Here
18:10:35 <mboddu> Sorry, got a call from my banker
18:11:32 * Southern_Gentlem want to make joke but will refrain
18:11:52 <nirik> done...thats it from infrastructure
18:12:53 <mboddu> One from releng
18:12:58 <mboddu> .releng 10200
18:12:59 <zodbot> mboddu: Issue #10200: Unretire rpms/libad9361 - releng - Pagure.io - https://pagure.io/releng/issue/10200
18:13:01 <mboddu> low, low, ops
18:13:06 <nirik> +1
18:13:25 <mboddu> Southern_Gentlem: What is it?
18:15:01 <nirik> #topic upcoming plans
18:15:10 <nirik> any upcoming items ?
18:15:22 <eddiejennings> None for me.
18:15:28 <darknao> can we take a quick look on one more ticket from infra?
18:15:32 <mboddu> Well, things as they come and help lenkaseg
18:15:44 <nirik> darknao: sure
18:15:49 <nirik> mboddu: +1
18:15:49 <darknao> .ticket 10076
18:15:50 <zodbot> darknao: Issue #10076: Docs staging not building - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10076
18:15:58 * mboddu goes after darknao
18:16:11 <darknao> so i've looked on few namespaces on staging
18:16:27 <darknao> and all of them got pods in ImagePullBackoff
18:16:41 <darknao> looks like the internal registry was purged or something
18:17:05 <darknao> so I had to rebuild all of them
18:17:09 <nirik> yeah, I see some nodes cordoned
18:17:14 * nirik fixes those
18:17:23 <darknao> but i'm most likely not the only one impacted here
18:17:26 <nirik> NAME                                     STATUS                     ROLES           AGE       VERSION
18:17:27 <nirik> os-master01.stg.iad2.fedoraproject.org   Ready                      master          341d      v1.11.0+d4cacc0
18:17:27 <nirik> os-master02.stg.iad2.fedoraproject.org   Ready                      master          341d      v1.11.0+d4cacc0
18:17:27 <nirik> os-master03.stg.iad2.fedoraproject.org   Ready,SchedulingDisabled   master          341d      v1.11.0+d4cacc0
18:17:30 <nirik> os-node01.stg.iad2.fedoraproject.org     Ready                      compute,infra   341d      v1.11.0+d4cacc0
18:17:33 <nirik> os-node02.stg.iad2.fedoraproject.org     Ready,SchedulingDisabled   compute,infra   341d      v1.11.0+d4cacc0
18:17:36 <nirik> os-node03.stg.iad2.fedoraproject.org     Ready,SchedulingDisabled   compute,infra   341d      v1.11.0+d4cacc0
18:17:39 <nirik> os-node04.stg.iad2.fedoraproject.org     Ready                      compute,infra   341d      v1.11.0+d4cacc0
18:17:42 <nirik> os-node05.stg.iad2.fedoraproject.org     Ready                      compute,infra   341d      v1.11.0+d4cacc0
18:17:59 <nirik> darknao: this is likely due to our reboots yesterday... we purge old images, so if something hasn't built in a while...
18:18:07 <darknao> we may need to rebuild everything (or run the openshift-apps playbooks)
18:18:21 <darknao> well, those were not so old
18:18:30 <darknao> like 1 month old maybe
18:18:47 <nirik> yes, we purge... weekly? can't recall
18:19:20 <darknao> i don't remember having the need to rebuild images that often
18:19:29 <nirik> it might be monthly
18:20:22 <darknao> ok weird
18:20:23 <nirik> anyhow, I guess I can go build everything...
18:21:11 <darknao> is there any reason to purge the internal registry that often ?
18:21:20 <nirik> we were running low on space.
18:21:27 <nirik> I am looking for the cron job...
18:24:30 <nirik> ok, its:
18:24:32 <nirik> 0 0 * * 1 docker rmi $(docker images --filter dangling=true -q)
18:25:10 <nirik> but that shouldn't delete images that are in use by existing containers...
18:25:25 <nirik> we could make it run less often...
18:26:14 <darknao> that only purging local docker images
18:26:19 <nirik> I think we hit space issues when a number of apps were in heavy development... like bodhi was making 100 images a day or something.
18:26:23 <darknao> not openshift internal registry
18:26:25 <nirik> yeah, so not sure.
18:27:58 <darknao> can you check if there is a lot of pods currently in ImagePullBackOff state ?
18:28:18 <nirik> yes, there is.
18:29:31 <nirik> well, we are getting out of time here. lets look at that out of meeting.
18:29:42 <nirik> any other topics? mboddu ?
18:30:05 <mboddu> nirik: Yup
18:30:15 <lenkaseg> the issue https://pagure.io/releng/issue/10191
18:30:32 <lenkaseg> I think I'd need sme guidance here :)
18:30:39 <lenkaseg> * I think I'd need some guidance here :)
18:31:44 <lenkaseg> * I think I'd need some guidance here, please :)
18:32:04 <mboddu> nirik: Does apprantice group gives read access to /pub?
18:32:10 <nirik> lenkaseg: oh yeah... so on that one we decided to put in sym links... basically link from SRPMS to source/tree... in all the arch subdirs.
18:32:23 <nirik> mboddu: should yes
18:33:01 <mboddu> lenkaseg: ^ in that case, try ssh'ing into bodhi-backend01.iad2.fedoraproject.org and see if you can access /pub
18:33:33 <mboddu> And I will help lenkaseg after that
18:33:37 <mboddu> That is all
18:34:24 <nirik> pub is also available on batcave01...
18:34:26 <lenkaseg> mmm...I get Connection closed by UNKNOWN port 65535
18:34:53 <nirik> cd /srv/web/pub/
18:35:11 <lenkaseg> yes! thanks
18:35:12 <mboddu> lenkaseg: Not sure if you followed https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/sshaccess.html?
18:35:55 <lenkaseg> I did, but could not recall
18:36:08 <mboddu> Basically, you need to have https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/sshaccess.html#ssh-configuration in your ssh config
18:36:29 <mboddu> But anyway, we can discuss it outside of the meeting
18:36:42 <lenkaseg> Ok!
18:36:59 <nirik> cool. Lets move back to #fedora-noc and/or #fedora-releng and/or #fedora-admin. ;)
18:37:04 <nirik> thanks everyone!
18:37:06 <nirik> #endmeeting