18:02:00 <phsmoura_> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:02:00 <zodbot> Meeting started Wed Jul 5 18:02:00 2023 UTC. 18:02:00 <zodbot> This meeting is logged and archived in a public location. 18:02:00 <zodbot> The chair is phsmoura_. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 18:02:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:02:00 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:02:00 <phsmoura_> #chair nirik smooge phsmoura 18:02:00 <zodbot> Current chairs: nirik phsmoura phsmoura_ smooge 18:02:00 <phsmoura_> #meetingname fedora_infrastructure_ops_daily_standup_meeting 18:02:00 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:02:00 <phsmoura_> #info meeting is 30 minutes MAX. At the end of 30, its stops 18:02:00 <phsmoura_> #info agenda is at https://board.net/p/fedora-infra-daily 18:02:28 <phsmoura_> hello 18:02:54 <nirik> morning 18:03:47 <phsmoura_> #info reminder: speak up if you want to work on a ticket! 18:03:47 <phsmoura_> #topic Tickets needing review 18:03:55 <phsmoura_> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:04:08 <phsmoura_> .ticket 11397 18:04:09 <zodbot> phsmoura_: Issue #11397: Koji seem to be repeatdaly unreachable (timeouts) and builds are stuck - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11397 18:04:43 <nirik> lets close it. ;) 18:04:44 <nirik> shall I? 18:04:58 <phsmoura_> sure :) 18:05:08 <phsmoura_> go ahead 18:05:59 <phsmoura_> .ticket 11401 18:06:00 <zodbot> phsmoura_: Issue #11401: Badges website is very slow / unresponsive - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11401 18:06:40 <nirik> I guess we can investigate this, everytime I look it's fast. ;( 18:06:43 <nirik> low low ops? 18:06:49 <phsmoura_> +1 18:07:34 <phsmoura_> #info https://pagure.io/releng/issues?status=Open 18:07:51 <phsmoura_> .releng 11513 18:07:52 <zodbot> phsmoura_: Issue #11513: buildvm-s390x-20.s390.fedoraproject.org might be slow on IO - releng - Pagure.io - https://pagure.io/releng/issue/11513 18:08:07 <nirik> needs investigation. med med ops? 18:08:51 <phsmoura_> .releng 11514 18:08:52 <zodbot> phsmoura_: Issue #11514: Releng impact analysis for telemetry change proposal - releng - Pagure.io - https://pagure.io/releng/issue/11514 18:09:45 <smooge> i am going to bet that the badges issue is proxy related 18:10:09 <smooge> the 11514 issue looked more like an infrastructure issue 18:10:12 <nirik> I don't think this has any releng impact... but perhaps? I guess med med ops and we can review? (I haven't looked at it yet) 18:10:35 <nirik> smooge: you might think so, but look in -admin... the same proxies were used and fast for one person and slow for the reporter. 18:11:02 <smooge> i looked the telemetry request over earlier today, and I don't think it will have a rleeng impact either 18:11:18 <smooge> nirik: ah ok. I dropped admin for some reason a while ago 18:11:50 <smooge> so looks like I lose another blame the proxies bet 18:12:11 * smooge puts 5 cents in the jar 18:12:19 <nirik> it could be still... no idea. 18:13:43 <smooge> I didn't get a chance to look at the s390x issue this morning.. the rdu-cc cage issue came up and I focused on it 18:14:19 <phsmoura_> let me change to open floor 18:14:24 <smooge> oops 18:14:28 <phsmoura_> #topic Planning, Upcoming work and Open floor 18:14:38 <phsmoura_> np :) 18:15:32 <nirik> I've been in meetings or fighting fires all day so far. ;) My plan for this week is to just try and catch up on emails I have to reply too, tickets, etc... 18:15:59 <nirik> I have a lunch on friday, so I will be out much of the day friday doing that and my regular grocery shopping. ;) 18:16:11 <smooge> good luck with that. 18:16:21 <smooge> both the emails and the lunch 18:16:37 <phsmoura_> Im working on a script to compress logs in log01 18:16:44 <nirik> I'm sure the lunch will be lovely. :) The emails... probibly not so much 18:17:05 <nirik> phsmoura: oh? what logs... we are already compressing some... 18:18:28 <phsmoura_> that relates to dnf count me 18:19:17 <nirik> ok. We are compressing host logs... dunno if thats the same thing, but I guess you can check the logs. :) If they are already compressed then... 18:19:19 <nirik> ¯\_(ツ)_/¯ 18:19:51 <phsmoura_> they are in combined-http 18:21:26 <phsmoura_> cant see them, but nils was explaining earlier 18:21:30 <nirik> ok, then yeah, thats likely other ones. 18:22:21 <smooge> those are the ones which are created daily on the nfs partition for awstats or any other tools to be used on 18:22:32 <nils> nirik, the uncompressed combined log files take up so much space, and about ~30G more per day 😬 18:22:48 <phsmoura_> yep those 18:23:28 <smooge> nils: I think one of the issues is that I originally combined ALL the httpd logs because we were looking at awstats or other tools to do stuff 18:23:29 <phsmoura_> btw do you know the path they are? Im working on the script locally for now but will do a pr in ansible repo when its finished 18:23:56 <nils> phsmoura_, /mnt/fedora_stats/combined-http from the top of my head 18:24:02 <nils> don’t have it open anymore 😉 18:24:04 <smooge> no one looked at those so I really think it would be ok to delete everything but the 2 mirrors from the combined logs and cut down 27GB of space 18:24:55 <smooge> nils: your memory is impeccable /mnt/fedora_stats/combined-http 18:24:58 <nils> smooge, compressing old ones is still worthwhile I think, mirrors.fpo accounts for ~10G daily uncompressed 18:24:58 <phsmoura_> ah ok, so they were in in combined-http :) 18:26:07 <nils> smooge, sometimes, sometimes not. Sometimes I forget that we should document the stuff we work on 😁 18:26:22 <nils> Forest, trees, something something 18:27:28 <nils> smooge, but I agree, if we don’t need all the rest, let’s do away with it, best not generate it in the first place (which should be only a small change to the scripts that do that I hope) 18:28:12 <smooge> nils: I think it would just be a small change in ansible and a script 18:28:41 <smooge> then a 'clean' out would reclaim space 18:29:19 <smooge> just in time for mattdm to say he wants to resurrect log analysis and do past history 18:29:20 <nils> yeah agreed, a great (and easy) candidate for the maintenance phase 18:29:36 <nils> haha 18:29:37 <smooge> and with that... I exit stage left 18:30:01 <nils> .hello nphilipp 18:30:02 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com> 18:30:09 <nils> having entered stage from wherever 18:31:04 <nirik> 🎵all the worlds a stage, and we are merely players 🎵 18:32:19 <phsmoura_> ok, lets close this meeting we can continue in fedora-noc :) 18:32:39 <smooge> okie dokie 18:32:40 <nils> Before someone calls us all puppets? 😂 18:32:54 * smooge starts playing master of puppets in his head 18:33:11 <phsmoura_> thanks everyone for joining :) 18:33:25 <phsmoura_> #endmeeting