19:00:42 #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 19:00:42 Meeting started Mon Dec 9 19:00:42 2019 UTC. 19:00:42 This meeting is logged and archived in a public location. 19:00:42 The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:42 Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:42 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 19:00:42 #chair smooge nirik relrod 19:00:42 Current chairs: nirik relrod smooge 19:00:42 #info meeting is 30 minutes MAX. At the end of 30, its stops 19:00:42 #info agenda is at https://board.net/p/fedora-infra-daily 19:01:53 hey 19:01:58 #topic Tickets needing review 19:01:58 #info https://pagure.io/fedora-infrastructure/issues 19:02:45 .ticket 8425 19:02:46 nirik: Issue #8425: ppc64le mirror problems - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8425 19:03:13 this might be due to mirrorlists this morning? 19:03:16 not updating for a while? 19:03:38 it is probably due to this but I don't know why only ppc would see it 19:03:47 yeah, thats odd. 19:04:30 we can ask for more info? or just close it? 19:05:14 pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8425: mirrorlists https://pagure.io/fedora-infrastructure/issue/8425 19:05:15 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8425 https://pagure.io/fedora-infrastructure/issue/8425 19:06:08 pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8425: "ppc64le mirror problems" https://pagure.io/fedora-infrastructure/issue/8425#comment-615723 19:06:26 going to ask for more info. 19:06:58 there were 2 other tickets i reviewed earlier today and put for waiting on assignee 19:06:59 ok, sounds fine. 19:09:14 so I see we have one ticket marked urgent... 19:09:57 .ticket 8426 19:09:58 nirik: Issue #8426: bodhi's page to create a new update returns a 504 error - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8426 19:10:17 I am not really sure what the status is here... pingou cverna ? either of you around? 19:10:59 I don't think it is urgent. It seems to occur and then fix itself 19:11:15 I do note that koji's db is getting the crap pounded out of it again. 19:12:38 yes 19:13:08 That should not block people from creating update anymore 19:13:41 I think it's autovacuming the big tables... which is causing a lot of load 19:13:47 but we need to check why we have so many 504, if this is too heavy on koji maybe we can drop that feature 19:14:20 I could try to look at the logs and see when it happens 19:14:26 I am leaning toward partitioning the tables... that would solve lots of our problems... it's makes me a bit nervous tho as it's a delecate change 19:14:49 you can maybe play with stg first 19:14:51 I suspect it's just koji load... but would be good to know more 19:15:41 yeah bodhi itself is not super fast also 19:16:15 so should we close this one? or leave it for tracking and change the priority? 19:16:52 I will change the priority 19:17:00 let's leave it and change the priority for now, I ll chat with pingou to see what he want to do 19:17:26 maybe the community fire fighting team can look at this :) 19:18:00 ok. I think it can mostly be solved by partitioning, (which we have instructions on). 19:18:19 perhaps if everyone is done using stg for testing things, I can do a prod->stg sync and try and partition there? 19:18:40 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8426 https://pagure.io/fedora-infrastructure/issue/8426 19:18:41 pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8426: "bodhi's page to create a new update returns a 504 error" https://pagure.io/fedora-infrastructure/issue/8426#comment-615729 19:18:45 +1 19:19:14 currently we don't test stuff related to koji in stg (from the bodhi point of view) so this would be safe 19:19:40 I think mboddu was doing some compose test for epel8 modular so might want to check with him 19:19:44 cverna: well the new update page returning 504 is pretty big 19:19:54 the ticket about bodhi not talking to koji is no longer a priority 19:20:10 but the new update form not showing, I'm not sure if it's related to koji or not (I'd think not) 19:20:15 (maybe wrongly) 19:20:20 ah, so this is perhaps actually fixed then? 19:20:34 it off and on 19:20:42 let me try right now 19:21:01 the 504 is on and off correct ? 19:21:19 yes 19:21:28 do note that partitioning in prod will probibly involve a day or so downtime... need to take everything off line, get a clean db dump, do the partitioning and bring things up. But we could do that... just need to schedule it I think 19:21:30 the 504 on koji, is currently almost always on afaict 19:21:41 ok new update page loaded fine 19:21:46 504 for koji, or the update page? 19:21:57 its calls to the recent builds in koji -> 504 19:22:13 but I've closed the infra ticket on that issue with a ticket upstream ^ 19:23:36 so, this ticket should be closed. 19:23:43 it's for the updates form 504 19:23:51 currently yes 19:23:55 The koji thing we need to investigate more... 19:24:01 ok, it's down sometimes too? 19:24:14 I'm wondering if this could be related to a) 5.1 fixing it, or b) load on the db server? 19:24:20 the b) would be odd though 19:24:50 nirik: I'll take the ticket and try to reproduce it a few times tomorrow 19:24:53 if I can't, I'll close it 19:24:59 if I can, we can re-evaluate 19:25:08 pagure.issue.assigned.added -- pingou assigned ticket fedora-infrastructure#8426 to pingou https://pagure.io/fedora-infrastructure/issue/8426 19:25:34 we can also look at kibana and search for the 504 there 19:25:38 well, for the bodhi form, that sounds like openshift app isn't responding 19:25:49 see if we see a pattern or a particular time 19:25:49 which would be weird, but... possibly 19:25:57 pagure.issue.comment.added -- pingou commented on ticket fedora-infrastructure#8426: "bodhi's page to create a new update returns a 504 error" https://pagure.io/fedora-infrastructure/issue/8426#comment-615733 19:26:17 cverna: your kibana skills are I think the best in the team :-p 19:26:40 lol that's not a good thing :P 19:26:52 that tells a lot about everyone else :D 19:26:57 so, which pod is doing the form? bodhi-base? 19:27:00 ok we have come to the end of the 25 minutes for this meeting in my calendar. Do we have any other items for today? 19:27:10 bodhi-web... 19:27:10 we could check bodhi's events in prod too see if we don't crash 19:27:18 bodhi-web yes 19:27:19 smooge: endit 19:27:23 #endmeeting