19:00:10 #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 19:00:10 #chair cverna mboddu nirik relrod smooge 19:00:10 Meeting started Mon Feb 24 19:00:10 2020 UTC. 19:00:10 This meeting is logged and archived in a public location. 19:00:10 The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:10 Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:10 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 19:00:10 Current chairs: cverna mboddu nirik relrod smooge 19:00:11 #info meeting is 30 minutes MAX. At the end of 30, its stops 19:00:11 #info agenda is at https://board.net/p/fedora-infra-daily 19:00:11 #topic Tickets needing review 19:00:12 #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 19:00:16 huh, perhaps my clock is off. 19:00:23 zodbot: localtime 19:00:23 nirik: (localtime ) -- Returns the current time of the user. The timezone is queried from FAS. 19:00:31 zodbot: localtime kevin 19:00:32 nirik: The current local time of "kevin" is: "11:00" (timezone: US/Pacific) 19:00:37 whatever. 19:00:38 * cverna waves 19:00:39 Hello 19:00:54 nice pile of needs review today 19:01:05 .ticket 8676 19:01:06 nirik: Issue #8676: Issues with grokmirror - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8676 19:01:27 move to waiting on asignee... I can look at it, or several of us can and I can explain the setup 19:01:36 yes the weekend was productive :P 19:01:52 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8676 https://pagure.io/fedora-infrastructure/issue/8676 19:01:54 it just cant handle the number of repos we have... :( so not sure how easy it will be to fix this. 19:01:56 +1 to explain the setup :) 19:02:04 .ticket 8677 19:02:05 nirik: Issue #8677: Update aarch64 Koji builders kernel to 5.5.5 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8677 19:02:18 this is waiting on external... they need to determine the fixed kernel. 19:02:29 and if it's after tomorrow we will need a freeze break. ;) 19:02:43 .ticket 8679 19:02:44 nirik: Issue #8679: F32 mirrors problems? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8679 19:02:49 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8677 https://pagure.io/fedora-infrastructure/issue/8677 19:02:50 pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8677: "Update aarch64 Koji builders kernel to 5.5.5" https://pagure.io/fedora-infrastructure/issue/8677#comment-628323 19:03:24 this one may be because of branching. 19:03:46 nirik, ah ok. I don't know how we put stuff into cloudfront 19:03:47 we basically had to sync up all of development/32 and then all of development/rawhide again. 19:04:00 there's a cron job on mm-backend01 I think it is... 19:04:04 nirik: Is there a doc explaining this? 19:04:29 If its documented somewhere or if you can show it to me, I can run it next time after branching 19:05:16 no, that won't help... 19:05:33 the sync runs 3 times a day. 19:06:38 the problem is that we are syncing ~500GB of content when 32 appears. 19:06:45 That takes a bit of time 19:07:04 at least thats my theory... 19:07:38 also I guess it depends on when branched composes finish 19:07:56 so should we close this ticket ? 19:08:01 so, how about we keep this open and investigate tweaking the crons times? 19:08:09 and also ask them if it's still happening? 19:08:18 ha :) got my answer 19:08:20 +1 19:08:30 right now, it's run at 2, 7, 10... 19:08:44 but I think we should run it later. 19:09:00 nirik: The ticket was created today 19:09:15 That means, they might be still facing the issue as of yesterday 19:09:18 the one today finished at 14:37 19:09:32 so, it won't sync again until 2 utc 19:09:33 Okay, I understood now 19:09:55 so perhaps 15, 17, 19 ? 19:10:01 We should definitely tweak the cron times 19:10:08 of course if we fire another one... it will be totally different 19:10:22 so I guess in an ideal world we could trigger after a compose finishes. 19:10:47 anyhow, mboddu you want to look into this more 19:10:48 ? 19:10:50 May be 13, 17, 23? 19:10:55 Sure 19:11:08 Since rawhide finishes before branched 19:11:08 cverna: Do you have exactly what debugging code you added into koji? 19:11:30 cverna: I was asking on #koji, but then realized that I didn't have a good sense of what the debug output represents 19:11:31 mboddu: sure, worth a try 19:11:53 .ticket 8680 19:11:53 nirik, do you want me to put this into the ticket or are you doing so? 19:11:54 nirik: Issue #8680: My app in Communishift needs to send emails - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8680 19:11:58 smooge: go ahead 19:12:15 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8679 https://pagure.io/fedora-infrastructure/issue/8679 19:12:16 pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8679: "F32 mirrors problems?" https://pagure.io/fedora-infrastructure/issue/8679#comment-628357 19:12:18 so I can answer this one and close it: we have no smtp server for communishift, so I guess roll your own. 19:12:56 otaylor: yes I just logged chunk before that line (https://pagure.io/koji/blob/master/f/hub/kojixmlrpc.py) 19:13:07 nirik: +1 19:13:11 ok nirik will let you do that 19:13:11 pagure.issue.edit -- kevin edited the close_status and status fields of ticket fedora-infrastructure#8680 https://pagure.io/fedora-infrastructure/issue/8680 19:13:12 pagure.issue.comment.added -- kevin commented on ticket fedora-infrastructure#8680: "My app in Communishift needs to send emails" https://pagure.io/fedora-infrastructure/issue/8680#comment-628360 19:13:16 cverna: but how did you log it? 19:13:18 'let' 19:13:27 .ticket 8681 19:13:28 nirik: Issue #8681: How to request branch for modules ? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8681 19:13:40 otaylor: self.logger.error(chunk) 19:13:48 mboddu: do you knwo the process here? it's just a regular scm request isn't it? 19:14:00 cverna: do you know if we're running koji as python2 or python3? 19:14:08 otaylor: the hub is python2 19:14:09 otaylor: python2 19:14:50 pagure.issue.edit -- mohanboddu edited the priority fields of ticket fedora-infrastructure#8679 https://pagure.io/fedora-infrastructure/issue/8679 19:14:51 pagure.issue.comment.added -- mohanboddu commented on ticket fedora-infrastructure#8679: "F32 mirrors problems?" https://pagure.io/fedora-infrastructure/issue/8679#comment-628363 19:15:19 ok on 8681.. I don't know 19:15:41 cverna: Yes, to request a branch in modules, they need to provide --namespace modules, I can comment on the ticket 19:15:45 it looks like it is and they have been processed. 19:15:50 so I think we can close this one 19:16:09 .ticket 8682 19:16:13 nirik: Issue #8682: DNS update: testdays.fedorainfracloud.org - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8682 19:16:20 close? 19:16:39 we can't do much about this ? 19:16:42 cverna: The question is when the logger shows 'Bo\xeete' - what actually was the input bytes 19:17:02 otaylor: does --debug-xmlrpc on the client get the info ? 19:17:06 otaylor, could you come back in 15 minutes? We are trying to get a standup meeting done 19:17:42 pagure.issue.comment.added -- mohanboddu commented on ticket fedora-infrastructure#8681: "How to request branch for modules ?" https://pagure.io/fedora-infrastructure/issue/8681#comment-628370 19:17:53 smooge: so, we can't do the fedoraproject.org one currently. I guess lets list their options to them and see what they want to do? 19:17:55 smooge: sure 19:18:05 nirik, so we can't do either of them 19:18:06 yes sorry otaylor, I ll ping you when we are done 19:18:08 * otaylor steps back 19:18:31 pagure.issue.edit -- mohanboddu edited the close_status and status fields of ticket fedora-infrastructure#8681 https://pagure.io/fedora-infrastructure/issue/8681 19:18:45 I closed #8681 19:18:57 what I think needs to be done is a proxy which redirects 19:18:57 ah, I see... yes, we can, but it's a bit more work... we need to have them get a letsencrypt cert there. 19:19:54 or that... since they will need to answer for testdays.fedorainfracloud.org and the cert coming back is *.apps.os.fedorainfracloud.org 19:20:40 right, so we have cert-manager setup in communishift. Should be easily possible for them to ask it for a cert/etc 19:20:45 I'd have to look up the details. 19:21:03 so, leave open, move to waiting on asignee... 19:21:05 ? 19:21:07 the fedoraproject.org one might be easier cause I can just have the proxies put in a redirect to whatever url works in *.apps.os.fedorainfracloiug 19:21:17 there was also talk about moving this app into our stg/prod clusters. 19:21:24 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8682 https://pagure.io/fedora-infrastructure/issue/8682 19:21:31 yeah, we could go via proxies too... sure. 19:21:47 that would also work if we ever move it into the internal clusters. 19:22:13 ok I was working on that before the meeting so I will try and finish that 19:22:26 .ticket 8683 19:22:27 nirik: Issue #8683: Fedmsg sometimes inserts unhelpful / unrelated messages into #fedora-design - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8683 19:22:43 yeah, so this is the fedmsg irc bot filtering... it's... not very flexable. 19:23:03 also reminds me: we need to replace it with a fedora-messaging version. :) 19:23:18 so, I guess waiting on asignee... and we can look at it someday. 19:23:29 .ticket 8684 19:23:30 nirik: Issue #8684: docs.stg.fp.o doesn't build, openshift cronjob confused - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8684 19:23:54 I commented on that I think they should be able to delete the object using ansible 19:23:55 cverna already suggested a workaround/fix for this 19:23:57 yeah 19:24:36 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8683 https://pagure.io/fedora-infrastructure/issue/8683 19:24:43 we can move to waiting on external I guess 19:24:48 yeah. 19:24:54 thats all the needs reviews... 19:24:55 pagure.issue.edit -- cverna edited the priority fields of ticket fedora-infrastructure#8684 https://pagure.io/fedora-infrastructure/issue/8684 19:25:01 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8684 https://pagure.io/fedora-infrastructure/issue/8684 19:25:03 pagure.issue.tag.added -- cverna tagged ticket fedora-infrastructure#8684: OpenShift https://pagure.io/fedora-infrastructure/issue/8684 19:25:05 pagure.issue.edit -- cverna edited the priority fields of ticket fedora-infrastructure#8684 https://pagure.io/fedora-infrastructure/issue/8684 19:25:14 ahah sorry smooge 19:25:47 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8679 https://pagure.io/fedora-infrastructure/issue/8679 19:26:18 any others we want to discuss today? 19:26:33 do we have anything on releng ? 19:27:19 Yes 19:27:33 .releng 9275 19:27:34 mboddu: Issue #9275: tag Haskell_Stackage_LTS_14 builds into f33 - releng - Pagure.io - https://pagure.io/releng/issue/9275 19:27:52 I tagged the builds into f33-updates-candidate, but nothing is happening 19:29:12 Based on https://docs.fedoraproject.org/en-US/rawhide-gating/_images/Simplified_single_build_update.png - bodhi should pick them up and tag into f33-signing-pending and sign the builds and move on... 19:29:25 Did I miss something? 19:29:37 ok I can look at this with you mboddu 19:29:53 cverna: Okay 19:30:37 might just be taking a bit 19:30:43 mboddu: I am not sure bodhi will accept a build with .fc32 in a f33 release 19:31:10 there are some validators that checks that if remember correctly 19:31:20 thats... wrong 19:31:47 nirik: Well, its kinda wrong 19:32:06 well, I guess bodhi never hits it, but other dist tag packages are in any release. 19:32:14 I might be wrong too :P 19:32:18 After branching, all the builds will have .fc33 dist tag, so its fine, until we hit a edge case like this 19:32:19 if i podman pull registry.fedoraproject.org/fedora i get a bad ssl cert 19:32:26 also, some packages like 'fedora-repos' have no dist tag 19:32:45 i also seem to get it if i hit it in a web-browser.. (but not registry.fp.o itself, just the cdn) 19:32:54 https://cdn.registry.fedoraproject.org/v2/fedora/blobs/sha256:294cdb5bd8ee428966c9b23fae954f56d5ddd22c972ebf394d5dacce16759646: x509: certificate has expired or is not yet valid 19:32:55 langdon, we are in a meeting please ask in 5 minutes 19:32:56 just me? 19:33:03 smooge: ohh sure 19:33:11 so anything else, or shall we close out? 19:33:17 speaking of which we are over.. so I wil close out 19:33:21 #endmeeting