18:00:21 #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:00:21 Meeting started Mon Apr 6 18:00:21 2020 UTC. 18:00:21 This meeting is logged and archived in a public location. 18:00:21 The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:21 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:21 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:21 #chair cverna mboddu nirik smooge 18:00:21 Current chairs: cverna mboddu nirik smooge 18:00:21 #info meeting is 30 minutes MAX. At the end of 30, its stops 18:00:21 #info agenda is at https://board.net/p/fedora-infra-daily 18:00:21 #topic Tickets needing review 18:00:22 #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:00:52 .ticket 8815 18:00:53 smooge: Issue #8815: buildroot override "active" according to bodhi, but not active according to koji - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8815 18:01:16 hello 18:01:21 hello cverna 18:01:41 is this just kojira being slow ? we could just close I think 18:01:58 Hello 18:01:59 buildroot override one 18:02:32 yeah I think kojira rebuilds the buildroot and it has been quite slow recently our overloaded 18:02:38 s/our/or 18:02:42 ok will close 18:03:03 pagure.issue.edit -- smooge edited the close_status and status fields of ticket fedora-infrastructure#8815 https://pagure.io/fedora-infrastructure/issue/8815 18:03:04 pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8815: "buildroot override "active" according to bodhi, but not active according to koji" https://pagure.io/fedora-infrastructure/issue/8815#comment-640495 18:03:10 .ticket 8816 18:03:11 smooge: Issue #8816: Corrupted package in Fedora 31 compose: perf-debuginfo-5.5.15-200.fc31.x86_64 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8816 18:03:54 yeah it seems that the reporter also found a way to have it work so I think that's ok 18:04:11 oops 18:04:19 sorry, was heads down ignoring irc for once 18:04:37 ok this ticket needs more info I have asked for it and will put it as waiting on reporter 18:04:38 8815 is not just kojira 18:05:02 I think it's the bodhi bug where it untags things when someone edits? 18:05:52 fine on 8816, but we should not wait forever on them. :) 18:05:54 oh ok I can try to look at it closer, but I am trying to focus on finding why the web pods are hanging sometimes :( 18:07:09 hum, it's odd. 18:07:14 ok those are the only ones in review 18:07:19 Mon Apr 6 03:03:35 2020: gala-3.3.0-1.fc32 tagged into f32-override by bodhi 18:07:20 Mon Apr 6 05:24:40 2020: gala-3.3.0-1.fc32 untagged from f32-updates-testing by bodhi 18:07:20 Mon Apr 6 05:24:41 2020: gala-3.3.0-1.fc32 untagged from f32-override by bodhi 18:07:37 I don't see it tagged back in... so it should be broken right now 18:08:05 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8816 https://pagure.io/fedora-infrastructure/issue/8816 18:08:12 so build was removed during the edit ? 18:08:36 ah, I didn't look back enough 18:08:38 Fri Apr 3 14:13:44 2020: gala-3.3.0-1.fc32 tagged into f32-override by bodhi 18:08:38 Fri Apr 3 15:53:18 2020: gala-3.3.0-1.fc32 untagged from f32-override by bodhi 18:08:38 Mon Apr 6 03:03:35 2020: gala-3.3.0-1.fc32 tagged into f32-override by bodhi 18:08:38 Mon Apr 6 05:24:41 2020: gala-3.3.0-1.fc32 untagged from f32-override by bodhi 18:10:02 dunno. 18:10:12 so I had one thing: 18:10:16 ok I ll file an upstream issue for that and put the link in the infra ticket 18:10:39 since final freeze is now thursday morning, I will also make the infra freeze thursday... any objections? 18:11:18 nope 18:11:30 nope 18:11:36 we can do it during the meeting 18:11:57 cverna: ooooh... 18:12:08 oh, nevermind. 18:12:15 I thought I saw something with the update. 18:12:25 Anyhow, I am also working to get kojira working again. 18:13:00 cverna: I think when the update is edited, it lost the override as well 18:13:01 did the openshift node restart improve the situation ? 18:13:17 cverna: I rebooted all the nodes... 18:13:22 they were still acting weird. 18:13:31 mboddu: actually, there's no update for it yet. 18:13:47 nirik: There is https://bodhi.fedoraproject.org/updates/FEDORA-2020-369ab3db3d 18:13:58 yeah I tried to do some cleanup and rebooting services this morning 18:14:20 cverna: yeah, and I did some the other day too... but I took out the big hammer this time. 18:14:32 what is the koji command to get the tag history of a build 18:14:38 nirik: :) 18:14:42 mboddu: ah ha. yeah, missed it. ;) thanks 18:15:00 koji list-tag-history --all --package gala | grep 3.3.0-1.fc32 18:15:20 thanks 18:15:29 ok, if nothing else shall we close out? 18:15:41 But I am not 100% sure its the edits, since that update is edited 3 times but override is lost only 2 times, but the other two dates are matching 18:16:03 pagure.issue.comment.added -- cverna commented on ticket fedora-infrastructure#8815: "buildroot override "active" according to bodhi, but not active according to koji" https://pagure.io/fedora-infrastructure/issue/8815#comment-640499 18:16:20 * mboddu has 1 ticket for nirik 18:16:26 .releng 9364 18:16:27 mboddu: Issue #9364: drop fedora-release and fedora-repos from the secure-boot channel - releng - Pagure.io - https://pagure.io/releng/issue/9364 18:16:46 nirik: ^ you have removed them from secure-boot right? 18:17:05 mboddu: yeah, it's done. I was gonna see if dgilmore could chime in with some more history (if it mattered) 18:17:21 Okay, then I will leave it as is 18:17:36 we can close it if you want and he can comment if he likes 18:18:15 Lets leave it for now, if we dont hear anything in a week, I will close it 18:18:31 ok 18:19:00 We never actually implemented the thing we were going to do that required the packages be in secure-boot 18:19:31 oh one quick thing, review-stats now runs in openshift and everything seems to work well :) 18:19:52 we were going to tie the release keys to the secure boot key 18:19:58 for end to end verification 18:20:27 dgilmore: ah ha. I knew there was some further reason 18:20:51 we can add em back if we ever do that 18:20:57 dgilmore: Ohh, thats a good idea 18:22:07 cverna: cool! 18:22:13 ok going to end this meeting if that is ok 18:22:36 cverna: I don't know if the bodhi web pods hangs are fixed now or not... they hang entirely? or just in a db transaction? 18:23:22 Nothing from me 18:24:00 nirik: yeah so I hope to be able to look at it more seriously in the next couple days, It looks like a request that leaves a db session open and eventually that will block other requests 18:24:21 nirik: it is just not clear why the request hangs like that 18:24:44 I have added a bit more logs to try to find out 18:24:57 cool. nothing more from me now. 18:25:44 #endmeeting