19:00:42 <smooge> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 19:00:42 <zodbot> Meeting started Mon Dec 9 19:00:42 2019 UTC. 19:00:42 <zodbot> This meeting is logged and archived in a public location. 19:00:42 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:42 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:42 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 19:00:42 <smooge> #chair smooge nirik relrod 19:00:42 <zodbot> Current chairs: nirik relrod smooge 19:00:42 <smooge> #info meeting is 30 minutes MAX. At the end of 30, its stops 19:00:42 <smooge> #info agenda is at https://board.net/p/fedora-infra-daily 19:01:53 <nirik> hey 19:01:58 <smooge> #topic Tickets needing review 19:01:58 <smooge> #info https://pagure.io/fedora-infrastructure/issues 19:02:45 <nirik> .ticket 8425 19:02:46 <zodbot> nirik: Issue #8425: ppc64le mirror problems - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8425 19:03:13 <nirik> this might be due to mirrorlists this morning? 19:03:16 <nirik> not updating for a while? 19:03:38 <smooge> it is probably due to this but I don't know why only ppc would see it 19:03:47 <nirik> yeah, thats odd. 19:04:30 <nirik> we can ask for more info? or just close it? 19:05:14 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8425: mirrorlists https://pagure.io/fedora-infrastructure/issue/8425 19:05:15 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8425 https://pagure.io/fedora-infrastructure/issue/8425 19:06:08 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8425: "ppc64le mirror problems" https://pagure.io/fedora-infrastructure/issue/8425#comment-615723 19:06:26 <smooge> going to ask for more info. 19:06:58 <smooge> there were 2 other tickets i reviewed earlier today and put for waiting on assignee 19:06:59 <nirik> ok, sounds fine. 19:09:14 <nirik> so I see we have one ticket marked urgent... 19:09:57 <nirik> .ticket 8426 19:09:58 <zodbot> nirik: Issue #8426: bodhi's page to create a new update returns a 504 error - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8426 19:10:17 <nirik> I am not really sure what the status is here... pingou cverna ? either of you around? 19:10:59 <smooge> I don't think it is urgent. It seems to occur and then fix itself 19:11:15 <nirik> I do note that koji's db is getting the crap pounded out of it again. 19:12:38 <cverna> yes 19:13:08 <cverna> That should not block people from creating update anymore 19:13:41 <nirik> I think it's autovacuming the big tables... which is causing a lot of load 19:13:47 <cverna> but we need to check why we have so many 504, if this is too heavy on koji maybe we can drop that feature 19:14:20 <cverna> I could try to look at the logs and see when it happens 19:14:26 <nirik> I am leaning toward partitioning the tables... that would solve lots of our problems... it's makes me a bit nervous tho as it's a delecate change 19:14:49 <cverna> you can maybe play with stg first 19:14:51 <nirik> I suspect it's just koji load... but would be good to know more 19:15:41 <cverna> yeah bodhi itself is not super fast also 19:16:15 <nirik> so should we close this one? or leave it for tracking and change the priority? 19:16:52 <smooge> I will change the priority 19:17:00 <cverna> let's leave it and change the priority for now, I ll chat with pingou to see what he want to do 19:17:26 <cverna> maybe the community fire fighting team can look at this :) 19:18:00 <nirik> ok. I think it can mostly be solved by partitioning, (which we have instructions on). 19:18:19 <nirik> perhaps if everyone is done using stg for testing things, I can do a prod->stg sync and try and partition there? 19:18:40 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8426 https://pagure.io/fedora-infrastructure/issue/8426 19:18:41 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8426: "bodhi's page to create a new update returns a 504 error" https://pagure.io/fedora-infrastructure/issue/8426#comment-615729 19:18:45 <cverna> +1 19:19:14 <cverna> currently we don't test stuff related to koji in stg (from the bodhi point of view) so this would be safe 19:19:40 <cverna> I think mboddu was doing some compose test for epel8 modular so might want to check with him 19:19:44 <pingou> cverna: well the new update page returning 504 is pretty big 19:19:54 <pingou> the ticket about bodhi not talking to koji is no longer a priority 19:20:10 <pingou> but the new update form not showing, I'm not sure if it's related to koji or not (I'd think not) 19:20:15 <pingou> (maybe wrongly) 19:20:20 <nirik> ah, so this is perhaps actually fixed then? 19:20:34 <pingou> it off and on 19:20:42 <pingou> let me try right now 19:21:01 <cverna> the 504 is on and off correct ? 19:21:19 <pingou> yes 19:21:28 <nirik> do note that partitioning in prod will probibly involve a day or so downtime... need to take everything off line, get a clean db dump, do the partitioning and bring things up. But we could do that... just need to schedule it I think 19:21:30 <pingou> the 504 on koji, is currently almost always on afaict 19:21:41 <pingou> ok new update page loaded fine 19:21:46 <nirik> 504 for koji, or the update page? 19:21:57 <pingou> its calls to the recent builds in koji -> 504 19:22:13 <pingou> but I've closed the infra ticket on that issue with a ticket upstream ^ 19:23:36 <nirik> so, this ticket should be closed. 19:23:43 <nirik> it's for the updates form 504 19:23:51 <pingou> currently yes 19:23:55 <nirik> The koji thing we need to investigate more... 19:24:01 <nirik> ok, it's down sometimes too? 19:24:14 <pingou> I'm wondering if this could be related to a) 5.1 fixing it, or b) load on the db server? 19:24:20 <pingou> the b) would be odd though 19:24:50 <pingou> nirik: I'll take the ticket and try to reproduce it a few times tomorrow 19:24:53 <pingou> if I can't, I'll close it 19:24:59 <pingou> if I can, we can re-evaluate 19:25:08 <fm-admin> pagure.issue.assigned.added -- pingou assigned ticket fedora-infrastructure#8426 to pingou https://pagure.io/fedora-infrastructure/issue/8426 19:25:34 <cverna> we can also look at kibana and search for the 504 there 19:25:38 <nirik> well, for the bodhi form, that sounds like openshift app isn't responding 19:25:49 <cverna> see if we see a pattern or a particular time 19:25:49 <nirik> which would be weird, but... possibly 19:25:57 <fm-admin> pagure.issue.comment.added -- pingou commented on ticket fedora-infrastructure#8426: "bodhi's page to create a new update returns a 504 error" https://pagure.io/fedora-infrastructure/issue/8426#comment-615733 19:26:17 <pingou> cverna: your kibana skills are I think the best in the team :-p 19:26:40 <cverna> lol that's not a good thing :P 19:26:52 <pingou> that tells a lot about everyone else :D 19:26:57 <nirik> so, which pod is doing the form? bodhi-base? 19:27:00 <smooge> ok we have come to the end of the 25 minutes for this meeting in my calendar. Do we have any other items for today? 19:27:10 <nirik> bodhi-web... 19:27:10 <cverna> we could check bodhi's events in prod too see if we don't crash 19:27:18 <cverna> bodhi-web yes 19:27:19 <nirik> smooge: endit 19:27:23 <smooge> #endmeeting