18:00:37 <phsmoura> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
18:00:37 <zodbot> Meeting started Tue May 23 18:00:37 2023 UTC.
18:00:37 <zodbot> This meeting is logged and archived in a public location.
18:00:37 <zodbot> The chair is phsmoura. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
18:00:37 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:37 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:37 <phsmoura> #chair nirik smooge phsmoura
18:00:37 <phsmoura> #meetingname fedora_infrastructure_ops_daily_standup_meeting
18:00:37 <zodbot> Current chairs: nirik phsmoura smooge
18:00:37 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:37 <phsmoura> #info meeting is 30 minutes MAX. At the end of 30, its stops
18:00:37 <phsmoura> #info agenda is at https://board.net/p/fedora-infra-daily
18:01:10 <nirik> good morning everyone.
18:01:24 <phsmoura> morning
18:02:06 <smooge> hello
18:02:30 <phsmoura> #info reminder: speak up if you want to work on a ticket!
18:02:30 <phsmoura> #topic Tickets needing review
18:02:37 <phsmoura> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
18:03:18 <phsmoura> .ticket 11332
18:03:19 <zodbot> phsmoura: Issue #11332: Outage: Upgrade of Copr servers - 2023-05-25 10:00 UTC - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11332
18:03:50 <nirik> med med ops... and it's on the copr team. ;)
18:04:28 <phsmoura> +1
18:05:05 <phsmoura> #info https://pagure.io/releng/issues?status=Open
18:05:19 <phsmoura> no new releng issues
18:05:31 <phsmoura> #topic Planning, Upcoming work and Open floor
18:05:39 <nirik> 😤
18:06:21 <nirik> So, for me: I'm reinstalling some builders now. After that I want to go back to rhel9 stuff... starting with batcave02 and possibly ipa upgrades.
18:06:59 <nirik> Also need to ponder on scheduling some more outages... we need to update the wiki in prod and batcave will take an outage and etc... but will come up with a plan.
18:07:38 <phsmoura> Im reading this dnf countme log issue, smooge are you working on 11331?
18:07:45 <phsmoura> .ticket 11331
18:07:46 <zodbot> phsmoura: Issue #11331: Syncing of mirror log files to log01 broke around May 5th-10th, bad DNF countme data - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11331
18:08:12 <nirik> yeah, I am not sure what we can do there... I mean, better erroring would be nice. Perhaps a nagios check to make sure it didn't fail?
18:09:52 <nirik> I looked for output from the script and couldn't really find it either.
18:10:44 <smooge> so what happened was that the rsync didn't get anything but the IAD2 proxies that week
18:11:06 <nirik> I am a bit confused on what script even does this.
18:11:13 <smooge> one sec
18:11:48 <nirik> in the playbooks there's one installed, but the cron for it is state=absent
18:12:47 <nirik> we should probibly go thru all of ansible and find everything with state=absent and possibly remove it. (after figuring out if it's still needed for some obscure reason)
18:13:36 <smooge> roles/web-data-analysis/files/sync-http-logs-and-merge.sh
18:13:59 <smooge> is the script which syncs down the http logs to log01
18:14:42 <nirik> wow.. nice. the matrix bridge didn't include the filename you had there. ;)
18:14:42 <smooge> which got replaced with roles/web-data-analysis/files/sync-http-logs.py
18:14:54 <smooge> of course not
18:15:36 <phsmoura> is it ok if I work on that too?
18:15:56 <smooge> phsmoura: sure i am not doing anything on it
18:16:04 <phsmoura> ok :)
18:16:49 <smooge> basically what needs to be done is make the script output that it had problems connecting and/or downloading logs from hosts. Then make sure the cron job which runs that script doesn't /dev/null those errors
18:16:53 <nirik> phsmoura: I'd suggest coordinating with the rest of the iniative team working on it. ;) I think there's a standup and such... just to avoid duplicate work
18:17:22 <nirik> ok, I found the output, it's unhelpfully "Subject: Cron <root@log01> [ ! -f /etc/cron.hourly/0anacron ] && run-parts /etc/cron.daily"
18:17:34 <smooge> and give it a better subject
18:18:31 <nirik> notice when it fails somehow would be nice... then someone could look/fix it while we still have the logs.
18:18:59 <smooge> phsmoura: nirik I am basically doing a 'I think I know where the bodies were buried. you will need a backhoe and some dynamite'
18:19:07 <nirik> anternately, we could try again to just rsyslog logs to log01
18:20:21 <nirik> anyhow, yeah, it needs more work for sure.
18:20:56 <phsmoura> we were talking about that one yesterday, Ill take a look
18:21:05 <phsmoura> have nothing else to share
18:21:59 <nirik> Yeah, I don't have anything else I don't think...
18:23:12 <phsmoura> ok, thank you nirik and smooge :)
18:23:25 <phsmoura> #endmeeting