19:00:00 <smooge> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
19:00:00 <zodbot> Meeting started Thu Mar  5 19:00:00 2020 UTC.
19:00:00 <zodbot> This meeting is logged and archived in a public location.
19:00:00 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:00 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:00:00 <smooge> #chair cverna mboddu nirik smooge
19:00:00 <smooge> #info meeting is 30 minutes MAX. At the end of 30, its stops
19:00:00 <smooge> #info agenda is at https://board.net/p/fedora-infra-daily
19:00:00 <zodbot> Current chairs: cverna mboddu nirik smooge
19:00:00 <smooge> #topic Tickets needing review
19:00:01 <smooge> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
19:00:08 <nirik> morning all.
19:00:25 <smooge> hello
19:00:25 <nirik> .ticket 8721
19:00:28 <zodbot> nirik: Issue #8721: Requesting a new compose box - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8721
19:00:30 * cverna is here
19:00:35 * mboddu is sorta here, in another meeting
19:00:38 <nirik> so I said we can do this box until the move to let them test...
19:00:51 <nirik> so I'd say move to waiting on asignee and we will get to it when we do
19:01:05 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8721: request-for-resources https://pagure.io/fedora-infrastructure/issue/8721
19:01:07 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8721 https://pagure.io/fedora-infrastructure/issue/8721
19:01:07 <nirik> .ticket 8726
19:01:08 <zodbot> nirik: Issue #8726: Staging OpenShift failing DNS Config Forming, Docs build stuck - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8726
19:01:22 <nirik> so, the warning in the ticket is harmless, just anoying.
19:01:34 <smooge> so two different things?
19:01:40 <nirik> so it must be stuck for another reason.
19:01:41 <smooge> build is stuck for other reason
19:01:48 <smooge> what you said
19:02:55 <cverna> yes I not sure why but sometimes builds are stuck, a docker service restart seems to fix it but it would be good to understand why that is needed
19:03:01 <nirik> so, I guess edit it to say investigate why build is stuck?
19:03:03 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8726: OpenShift https://pagure.io/fedora-infrastructure/issue/8726
19:03:04 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8726 https://pagure.io/fedora-infrastructure/issue/8726
19:03:48 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8726: "Staging OpenShift failing DNS Config Forming, Docs build stuck" https://pagure.io/fedora-infrastructure/issue/8726#comment-631108
19:04:38 <nirik> one thing I hate is that most pods don't have any debugging tools. ;(
19:04:39 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8726: "Staging OpenShift failing DNS Config Forming, Docs build stuck" https://pagure.io/fedora-infrastructure/issue/8726#comment-631109
19:04:49 <smooge> pods never fail
19:04:55 <smooge> we fail them
19:05:31 <smooge> cverna, where does one restart the docker?
19:05:49 <cverna> smooge: on the node where the build was schedule
19:05:55 <cverna> scheduled*
19:06:07 <nirik> but this should have nothing to do with docker
19:06:23 <cverna> the builds are still done using docker
19:06:30 <nirik> it's not a build, it's a cron job, so it's running in a cri-o container
19:06:42 <cverna> oh I thought it was a build
19:06:55 <nirik> well, it's building the website...
19:07:04 <nirik> but its in a openshift cronjob
19:07:35 <cverna> ok then it is just running a pod
19:07:47 <cverna> that needs more investigation I guess
19:07:50 <nirik> possibly related:
19:08:22 <nirik> netapp alerted eariler that the openshift_stg_docs volume was full...
19:08:28 <nirik> then it said it was ok again.
19:08:33 <nirik> could it be it runs out of space?
19:08:41 <nirik> 000320+  88074  8.1  0.0   3980  1092 ?        D    18:57   0:50 cp -r /tmp/tmpg017wh_1/docs_repo/public /antora/output/fr.building
19:08:44 <nirik> thats where it's stuck
19:09:24 <cverna> ha that could be it, then the cronjob log should display an error
19:09:35 <nirik> thats all the needs review ones.
19:09:53 <nirik> should we debug this one more? (happy to share my debugging)
19:09:59 <nirik> or any other tickets we want to look at?
19:10:11 <smooge> thanks for debugging
19:10:29 <smooge> I don't have any tickets to go over at the moment
19:10:31 <cverna> a quick one on the new review_stats, it is python3 and there is no python3-bugzilla in epel7 :(
19:10:52 <smooge> where does it need to run (aka what systems?)
19:10:57 <cverna> sundries
19:11:04 <nirik> in short: ssh os-master01.stg, oc project docsbuilding, oc get all to look at whats there, then oc get pods -o wide to see where the running one is, then login to os-node04 and look in ps
19:11:07 <smooge> so it is probably time to make el8 workable
19:11:31 <cverna> so far I just tried in stg, we could try to make a el8 in stg
19:11:42 <nirik> so, we have a sundries02, which we never use for anything...
19:11:54 <nirik> perhaps we should repave it as rhel8 and move things one by one?
19:12:10 <smooge> we will still probably need a python3-bugzilla for el8
19:12:11 <cverna> nirik: that is in prod right ?
19:12:14 <nirik> or... could we stick this in openshift? then just sync off the built thing?
19:12:23 <nirik> cverna: yeah
19:13:02 <nirik> epel8 does have python3-bugzilla
19:14:09 <cverna> I can put a ticket to rebuild the sundries with rhel8, how safe would it be to do it in stg ?
19:15:07 <nirik> so, the main things that does is... websites build... does it also still do geoip?
19:15:21 <smooge> ah ok
19:15:54 <nirik> does easyfix
19:16:13 <nirik> membership-map (which has been broken for years, so we should just remove it)
19:16:23 <cverna> hum easyfix migh not be python3 friendly :S
19:16:29 <nirik> reg-server (for flatpaks/registry index)
19:18:06 <nirik> so I dunno.
19:18:11 <cverna> nirik: we could move review_stats completely to OpenShift and have a http pod serving the content
19:18:19 <cverna> that might be easier :P
19:18:54 <nirik> yeah. openshift++
19:19:09 <nirik> proxies could just sync from there instead.
19:19:24 <cverna> ok I ll update the upstream ticket with these details
19:19:40 <smooge> ok for me to exit meeting?
19:19:40 <cverna> that's all from me
19:19:47 <nirik> hang on
19:19:50 <nirik> I had one thing
19:19:50 <smooge> np
19:20:22 <nirik> I know we have a number of trackers... but wanted to note the 'infra-sig' bugzilla bugs.
19:20:28 <nirik> (this will look ugly):
19:20:30 <nirik> https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&classification=Fedora&email1=infra-sig&emailassigned_to1=1&emailcc1=1&emailtype1=substring&list_id=10893303&order=bug_id&product=Fedora&query_format=advanced
19:20:52 <nirik> there's only 38 of them now... but we might want to start keeping track of them.
19:21:03 <nirik> and of course we need to poke at the releng ones someday too. ;)
19:21:17 <smooge> oof I didn't know about those
19:21:40 <nirik> they are basically packages we have in the past needed for our applications.
19:21:45 <nirik> likely there's some we no longer need
19:22:01 <cverna> yeah we could use Friday to look at these ?
19:22:15 <cverna> or another day ?
19:22:17 <nirik> https://src.fedoraproject.org/group/infra-sig
19:22:35 <nirik> yeah, some other time... just wanted to bring it up.
19:23:00 <nirik> also I have been trying to come up with a better workflow for my bugs, might be able to leverage that here too.
19:23:19 <nirik> (ie, bugs that are needinfo first, then cves, then ... )
19:23:36 <cverna> sounds good
19:23:38 <nirik> thats all I had, just wanted to bring it up.
19:23:53 <smooge> ok thanks
19:24:02 <smooge> #endmeeting