19:00:41 <phsmoura> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
19:00:41 <zodbot> Meeting started Wed Nov 16 19:00:41 2022 UTC.
19:00:41 <zodbot> This meeting is logged and archived in a public location.
19:00:41 <zodbot> The chair is phsmoura. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
19:00:41 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:41 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:00:42 <phsmoura> #chair nirik aheath1992 smooge
19:00:42 <zodbot> Current chairs: aheath1992 nirik phsmoura smooge
19:00:42 <phsmoura> #meetingname fedora_infrastructure_ops_daily_standup_meeting
19:00:42 <phsmoura> #info meeting is 30 minutes MAX. At the end of 30, its stops
19:00:42 <phsmoura> #info agenda is at https://board.net/p/fedora-infra-daily
19:00:42 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:01:06 <nirik> hello. good morning.
19:01:07 <smooge> here
19:01:16 <jednorozec> good evening
19:01:32 <phsmoura> hello
19:01:45 <phsmoura> #info reminder: speak up if you want to work on a ticket!
19:01:45 <phsmoura> #topic Tickets needing review
19:01:58 <phsmoura> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
19:02:10 <phsmoura> we dont have infra tickets today :)
19:02:35 <smooge> we don't? oh they must have been dealt with earlier
19:02:49 <phsmoura> yep
19:02:53 <phsmoura> #info https://pagure.io/releng/issues?status=Open
19:02:58 <nirik> I closed the outage ones, they seemed fixed to me.
19:03:01 <phsmoura> neither releng tickets
19:03:30 <phsmoura> #topic Planning, Upcoming work and Open floor
19:03:31 <smooge> okie dokie..
19:03:41 <nirik> #info freeze is over!
19:03:42 <jednorozec> nirik, any idea on how to mitigate the DB issue?
19:04:21 <nirik> jednorozec: not sure. We may be able to look at throttling connections per ip? or perhaps koji developers have some ideas.
19:04:52 <jednorozec> throttling connections seems like quick fix
19:04:57 <nirik> they were doing rpminfo requests... so that takes a lot of queries.
19:05:18 <jednorozec> yeah the selects were pretty complex
19:05:26 <jednorozec> and a lot of them...
19:05:54 <nirik> so basically things like: https://koji.fedoraproject.org/koji/rpminfo?rpmID=32489050
19:06:10 <nirik> so it has to get all the files in the rpm, all the requires/provides, sizes, etc
19:07:01 <nirik> anyhow, I am not sure apache can limit requests, we will need to look into it. There used to be a module, but I think it went away
19:08:02 <nirik> mod_qos perhaps
19:08:34 <nirik> anyhow, for plans: I have more meetings today, but aside that I want to start merging pr's soon... tomorrow I will be nuking the old openshift 3 clusters
19:10:15 <jednorozec> hm, mod_qos-11.70-2.fc37.x86_64.rpm
19:10:16 <phsmoura> about the outage tadoy. I tryed to help, but didnt had perms to ssh koji machines. Also found a sop that could help but seems its outdated. So, I could update that sop if you all agree. what do you all say?
19:12:06 <nirik> sure! which sop?
19:12:24 <phsmoura> it was under sysadmin guide
19:12:38 <phsmoura> let me see if I can find again
19:13:02 <phsmoura> https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/koji/#_troubleshooting_and_resolution
19:13:34 <smooge> nirik, I started to read through that and saw mentions of xen_builders and plague and did not think any of it could be trusted
19:13:52 <nirik> yeah, that is ancient/out of date for sure.
19:14:03 <nirik> yep. old old old
19:14:34 <smooge> and phsmoura my apologies again for not giving you more play by play of what I was trying to help you learn what was going on
19:15:25 <smooge> this actually brought up something I have been wanting to do for a paper.. mapping how the build system works backwards from a user downloading something to how a developer commits source code
19:15:55 <nirik> I did a flock talk on that long ago... 'package life cycle'
19:16:21 <phsmoura> np, couldnt ssh to them so not much I could do I guess :/
19:16:22 <smooge> it was my continual blind spot when I was in CPE because by the time I felt I had learned what was being done, nirik would have to clarify whatever i was saying because I forgot several subparts or still had something which wasn't there anymore
19:17:09 <nirik> it would be nice to have a thing we keep up to date...
19:17:19 <nirik> but we keep adding things... ostrees, containers, etc
19:17:25 <smooge> nirik, yeah I remembered that but wasn't sure how it fit in with things like containers, flatpaks, ostrees, etc
19:17:52 <smooge> so I was thinking of 'drawing' a picture for each 'flow' and seeing what things affected it and why
19:18:57 <nirik> sounds like a great project
19:21:55 <nirik> I don't have much else
19:22:26 <phsmoura> ok, so maybe next week I could to one of you to know how to fix those outages and update the sop?
19:23:49 <nirik> phsmoura: sure, happy to share what we learned in this case, and help you update it.
19:24:00 <phsmoura> cool :)
19:24:31 <phsmoura> is there anything else?
19:24:55 <nirik> not from me.
19:24:58 <smooge> not from me
19:25:49 <phsmoura> thanks for joining :)
19:25:57 <phsmoura> #endmeeting