18:02:00 <phsmoura_> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
18:02:00 <zodbot> Meeting started Wed Jul  5 18:02:00 2023 UTC.
18:02:00 <zodbot> This meeting is logged and archived in a public location.
18:02:00 <zodbot> The chair is phsmoura_. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
18:02:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:02:00 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:02:00 <phsmoura_> #chair nirik smooge phsmoura
18:02:00 <zodbot> Current chairs: nirik phsmoura phsmoura_ smooge
18:02:00 <phsmoura_> #meetingname fedora_infrastructure_ops_daily_standup_meeting
18:02:00 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:02:00 <phsmoura_> #info meeting is 30 minutes MAX. At the end of 30, its stops
18:02:00 <phsmoura_> #info agenda is at https://board.net/p/fedora-infra-daily
18:02:28 <phsmoura_> hello
18:02:54 <nirik> morning
18:03:47 <phsmoura_> #info reminder: speak up if you want to work on a ticket!
18:03:47 <phsmoura_> #topic Tickets needing review
18:03:55 <phsmoura_> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
18:04:08 <phsmoura_> .ticket 11397
18:04:09 <zodbot> phsmoura_: Issue #11397: Koji seem to be repeatdaly unreachable (timeouts) and builds are stuck - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11397
18:04:43 <nirik> lets close it. ;)
18:04:44 <nirik> shall I?
18:04:58 <phsmoura_> sure :)
18:05:08 <phsmoura_> go ahead
18:05:59 <phsmoura_> .ticket 11401
18:06:00 <zodbot> phsmoura_: Issue #11401: Badges website is very slow / unresponsive - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11401
18:06:40 <nirik> I guess we can investigate this, everytime I look it's fast. ;(
18:06:43 <nirik> low low ops?
18:06:49 <phsmoura_> +1
18:07:34 <phsmoura_> #info https://pagure.io/releng/issues?status=Open
18:07:51 <phsmoura_> .releng 11513
18:07:52 <zodbot> phsmoura_: Issue #11513: buildvm-s390x-20.s390.fedoraproject.org might be slow on IO - releng - Pagure.io - https://pagure.io/releng/issue/11513
18:08:07 <nirik> needs investigation. med med ops?
18:08:51 <phsmoura_> .releng 11514
18:08:52 <zodbot> phsmoura_: Issue #11514: Releng impact analysis for telemetry change proposal - releng - Pagure.io - https://pagure.io/releng/issue/11514
18:09:45 <smooge> i am going to bet that the badges issue is proxy related
18:10:09 <smooge> the 11514 issue looked more like an infrastructure issue
18:10:12 <nirik> I  don't think this has any releng impact... but perhaps? I guess med med ops and we can review? (I haven't looked at it yet)
18:10:35 <nirik> smooge: you might think so, but look in -admin... the same proxies were used and fast for one person and slow for the reporter.
18:11:02 <smooge> i looked the telemetry request over earlier today, and I don't think it will have a rleeng impact either
18:11:18 <smooge> nirik: ah ok. I dropped admin for some reason a while ago
18:11:50 <smooge> so looks like I lose another blame the proxies bet
18:12:11 * smooge puts 5 cents in the jar
18:12:19 <nirik> it could be still... no idea.
18:13:43 <smooge> I didn't get a chance to look at the s390x issue this morning.. the rdu-cc cage issue came up and I focused on it
18:14:19 <phsmoura_> let me change to open floor
18:14:24 <smooge> oops
18:14:28 <phsmoura_> #topic Planning, Upcoming work and Open floor
18:14:38 <phsmoura_> np :)
18:15:32 <nirik> I've been in meetings or fighting fires all day so far. ;) My plan for this week is to just try and catch up on emails I have to reply too, tickets, etc...
18:15:59 <nirik> I have a lunch on friday, so I will be out much of the day friday doing that and my regular grocery shopping. ;)
18:16:11 <smooge> good luck with that.
18:16:21 <smooge> both the emails and the lunch
18:16:37 <phsmoura_> Im working on a script to compress logs in log01
18:16:44 <nirik> I'm sure the lunch will be lovely. :) The emails... probibly not so much
18:17:05 <nirik> phsmoura: oh? what logs... we are already compressing some...
18:18:28 <phsmoura_> that relates to dnf count me
18:19:17 <nirik> ok. We are compressing host logs... dunno if thats the same thing, but I guess you can check the logs. :) If they are already compressed then...
18:19:19 <nirik> ¯\_(ツ)_/¯
18:19:51 <phsmoura_> they are in combined-http
18:21:26 <phsmoura_> cant see them, but nils was explaining earlier
18:21:30 <nirik> ok, then yeah, thats likely other ones.
18:22:21 <smooge> those are the ones which are created daily on the nfs partition for awstats or any other tools to be used on
18:22:32 <nils> nirik, the uncompressed combined log files take up so much space, and about ~30G more per day 😬
18:22:48 <phsmoura_> yep those
18:23:28 <smooge> nils: I think one of the issues is that I originally combined ALL the httpd logs because we were looking at awstats or other tools to do stuff
18:23:29 <phsmoura_> btw do you know the path they are? Im working on the script locally for now but will do a pr in ansible repo when its finished
18:23:56 <nils> phsmoura_, /mnt/fedora_stats/combined-http from the top of my head
18:24:02 <nils> don’t have it open anymore 😉
18:24:04 <smooge> no one looked at those so I really think it would be ok to delete everything but the 2 mirrors from the combined logs and cut down 27GB of space
18:24:55 <smooge> nils: your memory is impeccable /mnt/fedora_stats/combined-http
18:24:58 <nils> smooge, compressing old ones is still worthwhile I think, mirrors.fpo accounts for ~10G daily uncompressed
18:24:58 <phsmoura_> ah ok, so they were in in combined-http :)
18:26:07 <nils> smooge, sometimes, sometimes not. Sometimes I forget that we should document the stuff we work on 😁
18:26:22 <nils> Forest, trees, something something
18:27:28 <nils> smooge, but I agree, if we don’t need all the rest, let’s do away with it, best not generate it in the first place (which should be only a small change to the scripts that do that I hope)
18:28:12 <smooge> nils: I think it would just be a small change in ansible and a script
18:28:41 <smooge> then a 'clean' out would reclaim space
18:29:19 <smooge> just in time for mattdm to say he wants to resurrect log analysis and do past history
18:29:20 <nils> yeah agreed, a great (and easy) candidate for the maintenance phase
18:29:36 <nils> haha
18:29:37 <smooge> and with that... I exit stage left
18:30:01 <nils> .hello nphilipp
18:30:02 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
18:30:09 <nils> having entered stage from wherever
18:31:04 <nirik> 🎵all the worlds a stage, and we are merely players 🎵
18:32:19 <phsmoura_> ok, lets close this meeting we can continue in fedora-noc :)
18:32:39 <smooge> okie dokie
18:32:40 <nils> Before someone calls us all puppets? 😂
18:32:54 * smooge starts playing master of puppets in his head
18:33:11 <phsmoura_> thanks everyone for joining :)
18:33:25 <phsmoura_> #endmeeting