18:06:36 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:06:36 <zodbot> Meeting started Wed Jul 7 18:06:36 2021 UTC. 18:06:36 <zodbot> This meeting is logged and archived in a public location. 18:06:36 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:06:36 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:06:37 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:06:37 <nirik> #chair mboddu nirik pingou mobrien nb 18:06:37 <zodbot> Current chairs: mboddu mobrien nb nirik pingou 18:06:37 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting 18:06:37 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:06:37 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops 18:06:37 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily 18:06:37 <nirik> #info reminder: speak up if you want to work on a ticket! 18:06:39 <nirik> #topic Tickets needing review 18:06:41 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:06:43 <nirik> oops. got sidetracked 18:07:15 <darknao> hi 18:07:42 <lenkaseg> hello 18:07:52 <nirik> morning 18:08:48 <nirik> .ticket 10072 18:08:49 <zodbot> nirik: Issue #10072: Builds fail on buildvm-s390x-20.s390.fedoraproject.org: Couldn't resolve host name for http://kojipkgs01.fedoraproject.org/repos/f35-build/3809562/s390x/repodata/repomd.xml - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10072 18:09:04 <nirik> med/med/ops... and I was looking into this one just before this meeting. ;) 18:09:32 <nirik> .ticket 10074 18:09:33 <zodbot> nirik: Issue #10074: eduvpn group account - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10074 18:09:40 <nirik> I can just do this one right now. 18:09:59 <mboddu> Here 18:10:35 <mboddu> Sorry, got a call from my banker 18:11:32 * Southern_Gentlem want to make joke but will refrain 18:11:52 <nirik> done...thats it from infrastructure 18:12:53 <mboddu> One from releng 18:12:58 <mboddu> .releng 10200 18:12:59 <zodbot> mboddu: Issue #10200: Unretire rpms/libad9361 - releng - Pagure.io - https://pagure.io/releng/issue/10200 18:13:01 <mboddu> low, low, ops 18:13:06 <nirik> +1 18:13:25 <mboddu> Southern_Gentlem: What is it? 18:15:01 <nirik> #topic upcoming plans 18:15:10 <nirik> any upcoming items ? 18:15:22 <eddiejennings> None for me. 18:15:28 <darknao> can we take a quick look on one more ticket from infra? 18:15:32 <mboddu> Well, things as they come and help lenkaseg 18:15:44 <nirik> darknao: sure 18:15:49 <nirik> mboddu: +1 18:15:49 <darknao> .ticket 10076 18:15:50 <zodbot> darknao: Issue #10076: Docs staging not building - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/10076 18:15:58 * mboddu goes after darknao 18:16:11 <darknao> so i've looked on few namespaces on staging 18:16:27 <darknao> and all of them got pods in ImagePullBackoff 18:16:41 <darknao> looks like the internal registry was purged or something 18:17:05 <darknao> so I had to rebuild all of them 18:17:09 <nirik> yeah, I see some nodes cordoned 18:17:14 * nirik fixes those 18:17:23 <darknao> but i'm most likely not the only one impacted here 18:17:26 <nirik> NAME STATUS ROLES AGE VERSION 18:17:27 <nirik> os-master01.stg.iad2.fedoraproject.org Ready master 341d v1.11.0+d4cacc0 18:17:27 <nirik> os-master02.stg.iad2.fedoraproject.org Ready master 341d v1.11.0+d4cacc0 18:17:27 <nirik> os-master03.stg.iad2.fedoraproject.org Ready,SchedulingDisabled master 341d v1.11.0+d4cacc0 18:17:30 <nirik> os-node01.stg.iad2.fedoraproject.org Ready compute,infra 341d v1.11.0+d4cacc0 18:17:33 <nirik> os-node02.stg.iad2.fedoraproject.org Ready,SchedulingDisabled compute,infra 341d v1.11.0+d4cacc0 18:17:36 <nirik> os-node03.stg.iad2.fedoraproject.org Ready,SchedulingDisabled compute,infra 341d v1.11.0+d4cacc0 18:17:39 <nirik> os-node04.stg.iad2.fedoraproject.org Ready compute,infra 341d v1.11.0+d4cacc0 18:17:42 <nirik> os-node05.stg.iad2.fedoraproject.org Ready compute,infra 341d v1.11.0+d4cacc0 18:17:59 <nirik> darknao: this is likely due to our reboots yesterday... we purge old images, so if something hasn't built in a while... 18:18:07 <darknao> we may need to rebuild everything (or run the openshift-apps playbooks) 18:18:21 <darknao> well, those were not so old 18:18:30 <darknao> like 1 month old maybe 18:18:47 <nirik> yes, we purge... weekly? can't recall 18:19:20 <darknao> i don't remember having the need to rebuild images that often 18:19:29 <nirik> it might be monthly 18:20:22 <darknao> ok weird 18:20:23 <nirik> anyhow, I guess I can go build everything... 18:21:11 <darknao> is there any reason to purge the internal registry that often ? 18:21:20 <nirik> we were running low on space. 18:21:27 <nirik> I am looking for the cron job... 18:24:30 <nirik> ok, its: 18:24:32 <nirik> 0 0 * * 1 docker rmi $(docker images --filter dangling=true -q) 18:25:10 <nirik> but that shouldn't delete images that are in use by existing containers... 18:25:25 <nirik> we could make it run less often... 18:26:14 <darknao> that only purging local docker images 18:26:19 <nirik> I think we hit space issues when a number of apps were in heavy development... like bodhi was making 100 images a day or something. 18:26:23 <darknao> not openshift internal registry 18:26:25 <nirik> yeah, so not sure. 18:27:58 <darknao> can you check if there is a lot of pods currently in ImagePullBackOff state ? 18:28:18 <nirik> yes, there is. 18:29:31 <nirik> well, we are getting out of time here. lets look at that out of meeting. 18:29:42 <nirik> any other topics? mboddu ? 18:30:05 <mboddu> nirik: Yup 18:30:15 <lenkaseg> the issue https://pagure.io/releng/issue/10191 18:30:32 <lenkaseg> I think I'd need sme guidance here :) 18:30:39 <lenkaseg> * I think I'd need some guidance here :) 18:31:44 <lenkaseg> * I think I'd need some guidance here, please :) 18:32:04 <mboddu> nirik: Does apprantice group gives read access to /pub? 18:32:10 <nirik> lenkaseg: oh yeah... so on that one we decided to put in sym links... basically link from SRPMS to source/tree... in all the arch subdirs. 18:32:23 <nirik> mboddu: should yes 18:33:01 <mboddu> lenkaseg: ^ in that case, try ssh'ing into bodhi-backend01.iad2.fedoraproject.org and see if you can access /pub 18:33:33 <mboddu> And I will help lenkaseg after that 18:33:37 <mboddu> That is all 18:34:24 <nirik> pub is also available on batcave01... 18:34:26 <lenkaseg> mmm...I get Connection closed by UNKNOWN port 65535 18:34:53 <nirik> cd /srv/web/pub/ 18:35:11 <lenkaseg> yes! thanks 18:35:12 <mboddu> lenkaseg: Not sure if you followed https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/sshaccess.html? 18:35:55 <lenkaseg> I did, but could not recall 18:36:08 <mboddu> Basically, you need to have https://fedora-infra-docs.readthedocs.io/en/latest/sysadmin-guide/sops/sshaccess.html#ssh-configuration in your ssh config 18:36:29 <mboddu> But anyway, we can discuss it outside of the meeting 18:36:42 <lenkaseg> Ok! 18:36:59 <nirik> cool. Lets move back to #fedora-noc and/or #fedora-releng and/or #fedora-admin. ;) 18:37:04 <nirik> thanks everyone! 18:37:06 <nirik> #endmeeting