18:00:59 <smooge> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
18:00:59 <zodbot> Meeting started Thu Aug 20 18:00:59 2020 UTC.
18:00:59 <zodbot> This meeting is logged and archived in a public location.
18:00:59 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:59 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:59 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:59 <smooge> #chair mboddu nirik smooge pingou
18:00:59 <zodbot> Current chairs: mboddu nirik pingou smooge
18:00:59 <smooge> #meetingname fedora_infrastructure_ops_daily_standup_meeting
18:00:59 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:59 <smooge> #info meeting is 30 minutes MAX. At the end of 30, its stops
18:00:59 <smooge> #info agenda is at https://board.net/p/fedora-infra-daily
18:01:21 <smooge> There are no waiting issues for infrastructure
18:01:30 <mboddu> Hello
18:01:41 <mboddu> There are a couple on releng
18:01:47 <mboddu> Well, one is about the storage
18:01:48 <smooge> #info /mnt/koji ran out of disk space today due to snapshot churn eating 30 TB of disk.
18:02:33 <mboddu> Which I am closing now
18:02:39 <smooge> ok mboddu what are the links for releng tickets?
18:02:48 <mboddu> .releng 9701
18:02:49 <zodbot> mboddu: Issue #9701: Koji: : [Errno 28] No space left on device - releng - Pagure.io - https://pagure.io/releng/issue/9701
18:03:08 <mboddu> smooge: What is the fix in https://pagure.io/releng/issue/9701#comment-672336 ?
18:03:24 <smooge> There is no fix
18:03:32 <smooge> STOP USING UP MY DISK SPACE
18:03:58 <mboddu> Okay, I thought I was missing something
18:04:10 <smooge> a slow down is going to be that we create some more trees for places that get a lot of churn
18:04:24 <mboddu> I asked Peter to clean up some IoT stuff in https://kojipkgs.fedoraproject.org/compose/iot/
18:04:38 <smooge> those get 0 snapshots on them and that will slow down the churn in /mnt/koji
18:05:00 <smooge> basically deleting files does not delete them completely for about a month.
18:05:30 <mboddu> Okay, maybe we should reduce that time delta, set it to 2weeks?
18:05:55 <smooge> there is an hourly snapshot, there is a daily snapshot and there is a month snapshot..
18:06:12 <mboddu> Okay
18:06:42 <mboddu> Well, we only need $30K to get to 200T :D
18:06:44 <smooge> up until today it kept N hourly ones, M daily and Y monthly ones
18:06:52 <smooge> mboddu, we also need space to put it
18:06:59 <smooge> and power and ...
18:07:15 <smooge> 30k gets you a tray which sits in a corner not being able to be used
18:07:16 <mboddu> Those are all logistics, we can throw them on the floor :P
18:07:44 <mboddu> Anyway, we are going off topic
18:07:47 <smooge> anyway.. we worked with storage to cut down the number of snapshots left
18:07:55 <mboddu> +1
18:08:16 <mboddu> Closing the ticket then
18:08:29 <smooge> that is about all we can do until various trees like /mnt/fedora_koji/koji/scratch /mnt/fedora_koji/koji/compose etc are put onto their own volumes
18:08:58 <smooge> [which will depend on what parts of koji do hardlinks and which ones do copies]
18:09:09 <smooge> next ticket
18:09:20 <fm-admin> pagure.issue.edit -- mohanboddu edited the close_status and status fields of ticket releng#9701 https://pagure.io/releng/issue/9701
18:09:21 <fm-admin> pagure.issue.comment.added -- mohanboddu commented on ticket releng#9701: "Koji: <class 'OSError'>: [Errno 28] No space left on device" https://pagure.io/releng/issue/9701#comment-672348
18:09:35 <mboddu> .releng 9702
18:09:36 <zodbot> mboddu: Issue #9702: Unretire rpms/nodejs-tape - releng - Pagure.io - https://pagure.io/releng/issue/9702
18:09:40 <mboddu> This just needs doing
18:09:46 <mboddu> low,low,groomed,ops
18:09:55 <smooge> med/med/groomed/ops
18:10:00 <smooge> sorry what you siad
18:10:03 <smooge> +1
18:10:16 <mboddu> Thats all I got
18:10:23 <smooge> okie dokie
18:10:32 <smooge> lets call this meeting done then
18:10:41 <mboddu> Okay
18:10:50 <mboddu> Lets go back to the fires
18:10:53 <mboddu> And meetings
18:11:00 <smooge> I am waiting for allergy medicine to kick in
18:11:12 <smooge> #endmeeting