18:00:21 <smooge> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:00:21 <zodbot> Meeting started Mon Apr 6 18:00:21 2020 UTC. 18:00:21 <zodbot> This meeting is logged and archived in a public location. 18:00:21 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:21 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:21 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:21 <smooge> #chair cverna mboddu nirik smooge 18:00:21 <zodbot> Current chairs: cverna mboddu nirik smooge 18:00:21 <smooge> #info meeting is 30 minutes MAX. At the end of 30, its stops 18:00:21 <smooge> #info agenda is at https://board.net/p/fedora-infra-daily 18:00:21 <smooge> #topic Tickets needing review 18:00:22 <smooge> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:00:52 <smooge> .ticket 8815 18:00:53 <zodbot> smooge: Issue #8815: buildroot override "active" according to bodhi, but not active according to koji - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8815 18:01:16 <cverna> hello 18:01:21 <smooge> hello cverna 18:01:41 <cverna> is this just kojira being slow ? we could just close I think 18:01:58 <mboddu> Hello 18:01:59 <smooge> buildroot override one 18:02:32 <cverna> yeah I think kojira rebuilds the buildroot and it has been quite slow recently our overloaded 18:02:38 <cverna> s/our/or 18:02:42 <smooge> ok will close 18:03:03 <fm-admin> pagure.issue.edit -- smooge edited the close_status and status fields of ticket fedora-infrastructure#8815 https://pagure.io/fedora-infrastructure/issue/8815 18:03:04 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8815: "buildroot override "active" according to bodhi, but not active according to koji" https://pagure.io/fedora-infrastructure/issue/8815#comment-640495 18:03:10 <smooge> .ticket 8816 18:03:11 <zodbot> smooge: Issue #8816: Corrupted package in Fedora 31 compose: perf-debuginfo-5.5.15-200.fc31.x86_64 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8816 18:03:54 <cverna> yeah it seems that the reporter also found a way to have it work so I think that's ok 18:04:11 <nirik> oops 18:04:19 <nirik> sorry, was heads down ignoring irc for once 18:04:37 <smooge> ok this ticket needs more info I have asked for it and will put it as waiting on reporter 18:04:38 <nirik> 8815 is not just kojira 18:05:02 <nirik> I think it's the bodhi bug where it untags things when someone edits? 18:05:52 <nirik> fine on 8816, but we should not wait forever on them. :) 18:05:54 <cverna> oh ok I can try to look at it closer, but I am trying to focus on finding why the web pods are hanging sometimes :( 18:07:09 <nirik> hum, it's odd. 18:07:14 <smooge> ok those are the only ones in review 18:07:19 <nirik> Mon Apr 6 03:03:35 2020: gala-3.3.0-1.fc32 tagged into f32-override by bodhi 18:07:20 <nirik> Mon Apr 6 05:24:40 2020: gala-3.3.0-1.fc32 untagged from f32-updates-testing by bodhi 18:07:20 <nirik> Mon Apr 6 05:24:41 2020: gala-3.3.0-1.fc32 untagged from f32-override by bodhi 18:07:37 <nirik> I don't see it tagged back in... so it should be broken right now 18:08:05 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8816 https://pagure.io/fedora-infrastructure/issue/8816 18:08:12 <cverna> so build was removed during the edit ? 18:08:36 <nirik> ah, I didn't look back enough 18:08:38 <nirik> Fri Apr 3 14:13:44 2020: gala-3.3.0-1.fc32 tagged into f32-override by bodhi 18:08:38 <nirik> Fri Apr 3 15:53:18 2020: gala-3.3.0-1.fc32 untagged from f32-override by bodhi 18:08:38 <nirik> Mon Apr 6 03:03:35 2020: gala-3.3.0-1.fc32 tagged into f32-override by bodhi 18:08:38 <nirik> Mon Apr 6 05:24:41 2020: gala-3.3.0-1.fc32 untagged from f32-override by bodhi 18:10:02 <nirik> dunno. 18:10:12 <nirik> so I had one thing: 18:10:16 <cverna> ok I ll file an upstream issue for that and put the link in the infra ticket 18:10:39 <nirik> since final freeze is now thursday morning, I will also make the infra freeze thursday... any objections? 18:11:18 <cverna> nope 18:11:30 <smooge> nope 18:11:36 <smooge> we can do it during the meeting 18:11:57 <nirik> cverna: ooooh... 18:12:08 <nirik> oh, nevermind. 18:12:15 <nirik> I thought I saw something with the update. 18:12:25 <nirik> Anyhow, I am also working to get kojira working again. 18:13:00 <mboddu> cverna: I think when the update is edited, it lost the override as well 18:13:01 <cverna> did the openshift node restart improve the situation ? 18:13:17 <nirik> cverna: I rebooted all the nodes... 18:13:22 <nirik> they were still acting weird. 18:13:31 <nirik> mboddu: actually, there's no update for it yet. 18:13:47 <mboddu> nirik: There is https://bodhi.fedoraproject.org/updates/FEDORA-2020-369ab3db3d 18:13:58 <cverna> yeah I tried to do some cleanup and rebooting services this morning 18:14:20 <nirik> cverna: yeah, and I did some the other day too... but I took out the big hammer this time. 18:14:32 <cverna> what is the koji command to get the tag history of a build 18:14:38 <cverna> nirik: :) 18:14:42 <nirik> mboddu: ah ha. yeah, missed it. ;) thanks 18:15:00 <nirik> koji list-tag-history --all --package gala | grep 3.3.0-1.fc32 18:15:20 <cverna> thanks 18:15:29 <nirik> ok, if nothing else shall we close out? 18:15:41 <mboddu> But I am not 100% sure its the edits, since that update is edited 3 times but override is lost only 2 times, but the other two dates are matching 18:16:03 <fm-admin> pagure.issue.comment.added -- cverna commented on ticket fedora-infrastructure#8815: "buildroot override "active" according to bodhi, but not active according to koji" https://pagure.io/fedora-infrastructure/issue/8815#comment-640499 18:16:20 * mboddu has 1 ticket for nirik 18:16:26 <mboddu> .releng 9364 18:16:27 <zodbot> mboddu: Issue #9364: drop fedora-release and fedora-repos from the secure-boot channel - releng - Pagure.io - https://pagure.io/releng/issue/9364 18:16:46 <mboddu> nirik: ^ you have removed them from secure-boot right? 18:17:05 <nirik> mboddu: yeah, it's done. I was gonna see if dgilmore could chime in with some more history (if it mattered) 18:17:21 <mboddu> Okay, then I will leave it as is 18:17:36 <nirik> we can close it if you want and he can comment if he likes 18:18:15 <mboddu> Lets leave it for now, if we dont hear anything in a week, I will close it 18:18:31 <nirik> ok 18:19:00 <dgilmore> We never actually implemented the thing we were going to do that required the packages be in secure-boot 18:19:31 <cverna> oh one quick thing, review-stats now runs in openshift and everything seems to work well :) 18:19:52 <dgilmore> we were going to tie the release keys to the secure boot key 18:19:58 <dgilmore> for end to end verification 18:20:27 <nirik> dgilmore: ah ha. I knew there was some further reason 18:20:51 <nirik> we can add em back if we ever do that 18:20:57 <mboddu> dgilmore: Ohh, thats a good idea 18:22:07 <nirik> cverna: cool! 18:22:13 <smooge> ok going to end this meeting if that is ok 18:22:36 <nirik> cverna: I don't know if the bodhi web pods hangs are fixed now or not... they hang entirely? or just in a db transaction? 18:23:22 <mboddu> Nothing from me 18:24:00 <cverna> nirik: yeah so I hope to be able to look at it more seriously in the next couple days, It looks like a request that leaves a db session open and eventually that will block other requests 18:24:21 <cverna> nirik: it is just not clear why the request hangs like that 18:24:44 <cverna> I have added a bit more logs to try to find out 18:24:57 <nirik> cool. nothing more from me now. 18:25:44 <smooge> #endmeeting