19:00:42 <smooge> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
19:00:42 <zodbot> Meeting started Mon Dec  9 19:00:42 2019 UTC.
19:00:42 <zodbot> This meeting is logged and archived in a public location.
19:00:42 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:42 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:42 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:00:42 <smooge> #chair smooge nirik relrod
19:00:42 <zodbot> Current chairs: nirik relrod smooge
19:00:42 <smooge> #info meeting is 30 minutes MAX. At the end of 30, its stops
19:00:42 <smooge> #info agenda is at https://board.net/p/fedora-infra-daily
19:01:53 <nirik> hey
19:01:58 <smooge> #topic Tickets needing review
19:01:58 <smooge> #info https://pagure.io/fedora-infrastructure/issues
19:02:45 <nirik> .ticket 8425
19:02:46 <zodbot> nirik: Issue #8425: ppc64le mirror problems - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8425
19:03:13 <nirik> this might be due to mirrorlists this morning?
19:03:16 <nirik> not updating for a while?
19:03:38 <smooge> it is probably due to this but I don't know why only ppc would see it
19:03:47 <nirik> yeah, thats odd.
19:04:30 <nirik> we can ask for more info? or just close it?
19:05:14 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8425: mirrorlists https://pagure.io/fedora-infrastructure/issue/8425
19:05:15 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8425 https://pagure.io/fedora-infrastructure/issue/8425
19:06:08 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8425: "ppc64le mirror problems" https://pagure.io/fedora-infrastructure/issue/8425#comment-615723
19:06:26 <smooge> going to ask for more info.
19:06:58 <smooge> there were 2 other tickets i reviewed earlier today and put for waiting on assignee
19:06:59 <nirik> ok, sounds fine.
19:09:14 <nirik> so I see we have one ticket marked urgent...
19:09:57 <nirik> .ticket 8426
19:09:58 <zodbot> nirik: Issue #8426: bodhi's page to create a new update returns a 504 error - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8426
19:10:17 <nirik> I am not really sure what the status is here... pingou cverna ? either of you around?
19:10:59 <smooge> I don't think it is urgent. It seems to occur and then fix itself
19:11:15 <nirik> I do note that koji's db is getting the crap pounded out of it again.
19:12:38 <cverna> yes
19:13:08 <cverna> That should not block people from creating update anymore
19:13:41 <nirik> I think it's autovacuming the big tables... which is causing a lot of load
19:13:47 <cverna> but we need to check why we have so many 504, if this is too heavy on koji maybe we can drop that feature
19:14:20 <cverna> I could try to look at the logs and see when it happens
19:14:26 <nirik> I am leaning toward partitioning the tables... that would solve lots of our problems... it's makes me a bit nervous tho as it's a delecate change
19:14:49 <cverna> you can maybe play with stg first
19:14:51 <nirik> I suspect it's just koji load... but would be good to know more
19:15:41 <cverna> yeah bodhi itself is not super fast also
19:16:15 <nirik> so should we close this one? or leave it for tracking and change the priority?
19:16:52 <smooge> I will change the priority
19:17:00 <cverna> let's leave it and change the priority for now, I ll chat with pingou to see what he want to do
19:17:26 <cverna> maybe the community fire fighting team can look at this :)
19:18:00 <nirik> ok. I think it can mostly be solved by partitioning, (which we have instructions on).
19:18:19 <nirik> perhaps if everyone is done using stg for testing things, I can do a prod->stg sync and try and partition there?
19:18:40 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8426 https://pagure.io/fedora-infrastructure/issue/8426
19:18:41 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8426: "bodhi's page to create a new update returns a 504 error" https://pagure.io/fedora-infrastructure/issue/8426#comment-615729
19:18:45 <cverna> +1
19:19:14 <cverna> currently we don't test stuff related to koji in stg (from the bodhi point of view) so this would be safe
19:19:40 <cverna> I think mboddu was doing some compose test for epel8 modular so might want to check with him
19:19:44 <pingou> cverna: well the new update page returning 504 is pretty big
19:19:54 <pingou> the ticket about bodhi not talking to koji is no longer a priority
19:20:10 <pingou> but the new update form not showing, I'm not sure if it's related to koji or not (I'd think not)
19:20:15 <pingou> (maybe wrongly)
19:20:20 <nirik> ah, so this is perhaps actually fixed then?
19:20:34 <pingou> it off and on
19:20:42 <pingou> let me try right now
19:21:01 <cverna> the 504 is on and off correct ?
19:21:19 <pingou> yes
19:21:28 <nirik> do note that partitioning in prod will probibly involve a day or so downtime... need to take everything off line, get a clean db dump, do the partitioning and bring things up. But we could do that... just need to schedule it I think
19:21:30 <pingou> the 504 on koji, is currently almost always on afaict
19:21:41 <pingou> ok new update page loaded fine
19:21:46 <nirik> 504 for koji, or the update page?
19:21:57 <pingou> its calls to the recent builds in koji -> 504
19:22:13 <pingou> but I've closed the infra ticket on that issue with a ticket upstream ^
19:23:36 <nirik> so, this ticket should be closed.
19:23:43 <nirik> it's for the updates form 504
19:23:51 <pingou> currently yes
19:23:55 <nirik> The koji thing we need to investigate more...
19:24:01 <nirik> ok, it's down sometimes too?
19:24:14 <pingou> I'm wondering if this could be related to a) 5.1 fixing it, or b) load on the db server?
19:24:20 <pingou> the b) would be odd though
19:24:50 <pingou> nirik: I'll take the ticket and try to reproduce it a few times tomorrow
19:24:53 <pingou> if I can't, I'll close it
19:24:59 <pingou> if I can, we can re-evaluate
19:25:08 <fm-admin> pagure.issue.assigned.added -- pingou assigned ticket fedora-infrastructure#8426 to pingou https://pagure.io/fedora-infrastructure/issue/8426
19:25:34 <cverna> we can also look at kibana and search for the 504 there
19:25:38 <nirik> well, for the bodhi form, that sounds like openshift app isn't responding
19:25:49 <cverna> see if we see a pattern or a particular time
19:25:49 <nirik> which would be weird, but... possibly
19:25:57 <fm-admin> pagure.issue.comment.added -- pingou commented on ticket fedora-infrastructure#8426: "bodhi's page to create a new update returns a 504 error" https://pagure.io/fedora-infrastructure/issue/8426#comment-615733
19:26:17 <pingou> cverna: your kibana skills are I think the best in the team :-p
19:26:40 <cverna> lol that's not a good thing :P
19:26:52 <pingou> that tells a lot about everyone else :D
19:26:57 <nirik> so, which pod is doing the form? bodhi-base?
19:27:00 <smooge> ok we have come to the end of the 25 minutes for this meeting in my calendar. Do we have any other items for today?
19:27:10 <nirik> bodhi-web...
19:27:10 <cverna> we could check bodhi's events in prod too see if we don't crash
19:27:18 <cverna> bodhi-web yes
19:27:19 <nirik> smooge: endit
19:27:23 <smooge> #endmeeting