19:00:00 <smooge> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 19:00:00 <zodbot> Meeting started Thu Mar 5 19:00:00 2020 UTC. 19:00:00 <zodbot> This meeting is logged and archived in a public location. 19:00:00 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:00 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 19:00:00 <smooge> #chair cverna mboddu nirik smooge 19:00:00 <smooge> #info meeting is 30 minutes MAX. At the end of 30, its stops 19:00:00 <smooge> #info agenda is at https://board.net/p/fedora-infra-daily 19:00:00 <zodbot> Current chairs: cverna mboddu nirik smooge 19:00:00 <smooge> #topic Tickets needing review 19:00:01 <smooge> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 19:00:08 <nirik> morning all. 19:00:25 <smooge> hello 19:00:25 <nirik> .ticket 8721 19:00:28 <zodbot> nirik: Issue #8721: Requesting a new compose box - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8721 19:00:30 * cverna is here 19:00:35 * mboddu is sorta here, in another meeting 19:00:38 <nirik> so I said we can do this box until the move to let them test... 19:00:51 <nirik> so I'd say move to waiting on asignee and we will get to it when we do 19:01:05 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8721: request-for-resources https://pagure.io/fedora-infrastructure/issue/8721 19:01:07 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8721 https://pagure.io/fedora-infrastructure/issue/8721 19:01:07 <nirik> .ticket 8726 19:01:08 <zodbot> nirik: Issue #8726: Staging OpenShift failing DNS Config Forming, Docs build stuck - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8726 19:01:22 <nirik> so, the warning in the ticket is harmless, just anoying. 19:01:34 <smooge> so two different things? 19:01:40 <nirik> so it must be stuck for another reason. 19:01:41 <smooge> build is stuck for other reason 19:01:48 <smooge> what you said 19:02:55 <cverna> yes I not sure why but sometimes builds are stuck, a docker service restart seems to fix it but it would be good to understand why that is needed 19:03:01 <nirik> so, I guess edit it to say investigate why build is stuck? 19:03:03 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8726: OpenShift https://pagure.io/fedora-infrastructure/issue/8726 19:03:04 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8726 https://pagure.io/fedora-infrastructure/issue/8726 19:03:48 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8726: "Staging OpenShift failing DNS Config Forming, Docs build stuck" https://pagure.io/fedora-infrastructure/issue/8726#comment-631108 19:04:38 <nirik> one thing I hate is that most pods don't have any debugging tools. ;( 19:04:39 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8726: "Staging OpenShift failing DNS Config Forming, Docs build stuck" https://pagure.io/fedora-infrastructure/issue/8726#comment-631109 19:04:49 <smooge> pods never fail 19:04:55 <smooge> we fail them 19:05:31 <smooge> cverna, where does one restart the docker? 19:05:49 <cverna> smooge: on the node where the build was schedule 19:05:55 <cverna> scheduled* 19:06:07 <nirik> but this should have nothing to do with docker 19:06:23 <cverna> the builds are still done using docker 19:06:30 <nirik> it's not a build, it's a cron job, so it's running in a cri-o container 19:06:42 <cverna> oh I thought it was a build 19:06:55 <nirik> well, it's building the website... 19:07:04 <nirik> but its in a openshift cronjob 19:07:35 <cverna> ok then it is just running a pod 19:07:47 <cverna> that needs more investigation I guess 19:07:50 <nirik> possibly related: 19:08:22 <nirik> netapp alerted eariler that the openshift_stg_docs volume was full... 19:08:28 <nirik> then it said it was ok again. 19:08:33 <nirik> could it be it runs out of space? 19:08:41 <nirik> 000320+ 88074 8.1 0.0 3980 1092 ? D 18:57 0:50 cp -r /tmp/tmpg017wh_1/docs_repo/public /antora/output/fr.building 19:08:44 <nirik> thats where it's stuck 19:09:24 <cverna> ha that could be it, then the cronjob log should display an error 19:09:35 <nirik> thats all the needs review ones. 19:09:53 <nirik> should we debug this one more? (happy to share my debugging) 19:09:59 <nirik> or any other tickets we want to look at? 19:10:11 <smooge> thanks for debugging 19:10:29 <smooge> I don't have any tickets to go over at the moment 19:10:31 <cverna> a quick one on the new review_stats, it is python3 and there is no python3-bugzilla in epel7 :( 19:10:52 <smooge> where does it need to run (aka what systems?) 19:10:57 <cverna> sundries 19:11:04 <nirik> in short: ssh os-master01.stg, oc project docsbuilding, oc get all to look at whats there, then oc get pods -o wide to see where the running one is, then login to os-node04 and look in ps 19:11:07 <smooge> so it is probably time to make el8 workable 19:11:31 <cverna> so far I just tried in stg, we could try to make a el8 in stg 19:11:42 <nirik> so, we have a sundries02, which we never use for anything... 19:11:54 <nirik> perhaps we should repave it as rhel8 and move things one by one? 19:12:10 <smooge> we will still probably need a python3-bugzilla for el8 19:12:11 <cverna> nirik: that is in prod right ? 19:12:14 <nirik> or... could we stick this in openshift? then just sync off the built thing? 19:12:23 <nirik> cverna: yeah 19:13:02 <nirik> epel8 does have python3-bugzilla 19:14:09 <cverna> I can put a ticket to rebuild the sundries with rhel8, how safe would it be to do it in stg ? 19:15:07 <nirik> so, the main things that does is... websites build... does it also still do geoip? 19:15:21 <smooge> ah ok 19:15:54 <nirik> does easyfix 19:16:13 <nirik> membership-map (which has been broken for years, so we should just remove it) 19:16:23 <cverna> hum easyfix migh not be python3 friendly :S 19:16:29 <nirik> reg-server (for flatpaks/registry index) 19:18:06 <nirik> so I dunno. 19:18:11 <cverna> nirik: we could move review_stats completely to OpenShift and have a http pod serving the content 19:18:19 <cverna> that might be easier :P 19:18:54 <nirik> yeah. openshift++ 19:19:09 <nirik> proxies could just sync from there instead. 19:19:24 <cverna> ok I ll update the upstream ticket with these details 19:19:40 <smooge> ok for me to exit meeting? 19:19:40 <cverna> that's all from me 19:19:47 <nirik> hang on 19:19:50 <nirik> I had one thing 19:19:50 <smooge> np 19:20:22 <nirik> I know we have a number of trackers... but wanted to note the 'infra-sig' bugzilla bugs. 19:20:28 <nirik> (this will look ugly): 19:20:30 <nirik> https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&classification=Fedora&email1=infra-sig&emailassigned_to1=1&emailcc1=1&emailtype1=substring&list_id=10893303&order=bug_id&product=Fedora&query_format=advanced 19:20:52 <nirik> there's only 38 of them now... but we might want to start keeping track of them. 19:21:03 <nirik> and of course we need to poke at the releng ones someday too. ;) 19:21:17 <smooge> oof I didn't know about those 19:21:40 <nirik> they are basically packages we have in the past needed for our applications. 19:21:45 <nirik> likely there's some we no longer need 19:22:01 <cverna> yeah we could use Friday to look at these ? 19:22:15 <cverna> or another day ? 19:22:17 <nirik> https://src.fedoraproject.org/group/infra-sig 19:22:35 <nirik> yeah, some other time... just wanted to bring it up. 19:23:00 <nirik> also I have been trying to come up with a better workflow for my bugs, might be able to leverage that here too. 19:23:19 <nirik> (ie, bugs that are needinfo first, then cves, then ... ) 19:23:36 <cverna> sounds good 19:23:38 <nirik> thats all I had, just wanted to bring it up. 19:23:53 <smooge> ok thanks 19:24:02 <smooge> #endmeeting