19:00:10 <smooge> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
19:00:10 <smooge> #chair cverna mboddu nirik relrod smooge
19:00:10 <zodbot> Meeting started Mon Feb 24 19:00:10 2020 UTC.
19:00:10 <zodbot> This meeting is logged and archived in a public location.
19:00:10 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:10 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:10 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:00:10 <zodbot> Current chairs: cverna mboddu nirik relrod smooge
19:00:11 <smooge> #info meeting is 30 minutes MAX. At the end of 30, its stops
19:00:11 <smooge> #info agenda is at https://board.net/p/fedora-infra-daily
19:00:11 <smooge> #topic Tickets needing review
19:00:12 <smooge> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
19:00:16 <nirik> huh, perhaps my clock is off.
19:00:23 <nirik> zodbot: localtime
19:00:23 <zodbot> nirik: (localtime <username>) -- Returns the current time of the user. The timezone is queried from FAS.
19:00:31 <nirik> zodbot: localtime kevin
19:00:32 <zodbot> nirik: The current local time of "kevin" is: "11:00" (timezone: US/Pacific)
19:00:37 <nirik> whatever.
19:00:38 * cverna waves
19:00:39 <mboddu> Hello
19:00:54 <nirik> nice pile of needs review today
19:01:05 <nirik> .ticket 8676
19:01:06 <zodbot> nirik: Issue #8676: Issues with grokmirror - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8676
19:01:27 <nirik> move to waiting on asignee... I can look at it, or several of us can and I can explain the setup
19:01:36 <cverna> yes the weekend was productive :P
19:01:52 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8676 https://pagure.io/fedora-infrastructure/issue/8676
19:01:54 <nirik> it just cant handle the number of repos we have... :( so not sure how easy it will be to fix this.
19:01:56 <cverna> +1 to explain the setup :)
19:02:04 <nirik> .ticket 8677
19:02:05 <zodbot> nirik: Issue #8677: Update aarch64 Koji builders kernel to 5.5.5 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8677
19:02:18 <nirik> this is waiting on external... they need to determine the fixed kernel.
19:02:29 <nirik> and if it's after tomorrow we will need a freeze break. ;)
19:02:43 <nirik> .ticket 8679
19:02:44 <zodbot> nirik: Issue #8679: F32 mirrors problems? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8679
19:02:49 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8677 https://pagure.io/fedora-infrastructure/issue/8677
19:02:50 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8677: "Update aarch64 Koji builders kernel to 5.5.5" https://pagure.io/fedora-infrastructure/issue/8677#comment-628323
19:03:24 <nirik> this one may be because of branching.
19:03:46 <smooge> nirik, ah ok. I don't know how we put stuff into cloudfront
19:03:47 <nirik> we basically had to sync up all of development/32 and then all of development/rawhide again.
19:04:00 <nirik> there's a cron job on mm-backend01 I think it is...
19:04:04 <mboddu> nirik: Is there a doc explaining this?
19:04:29 <mboddu> If its documented somewhere or if you can show it to me, I can run it next time after branching
19:05:16 <nirik> no, that won't help...
19:05:33 <nirik> the sync runs 3 times a day.
19:06:38 <nirik> the problem is that we are syncing ~500GB of content when 32 appears.
19:06:45 <nirik> That takes a bit of time
19:07:04 <nirik> at least thats my theory...
19:07:38 <nirik> also I guess it depends on when branched composes finish
19:07:56 <cverna> so should we close this ticket ?
19:08:01 <nirik> so, how about we keep this open and investigate tweaking the crons times?
19:08:09 <nirik> and also ask them if it's still happening?
19:08:18 <cverna> ha :) got my answer
19:08:20 <mboddu> +1
19:08:30 <nirik> right now, it's run at 2, 7, 10...
19:08:44 <nirik> but I think we should run it later.
19:09:00 <mboddu> nirik: The ticket was created today
19:09:15 <mboddu> That means, they might be still facing the issue as of yesterday
19:09:18 <nirik> the one today finished at 14:37
19:09:32 <nirik> so, it won't sync again until 2 utc
19:09:33 <mboddu> Okay, I understood now
19:09:55 <nirik> so perhaps 15, 17, 19 ?
19:10:01 <mboddu> We should definitely tweak the cron times
19:10:08 <nirik> of course if we fire another one... it will be totally different
19:10:22 <nirik> so I guess in an ideal world we could trigger after a compose finishes.
19:10:47 <nirik> anyhow, mboddu you want to look into this more
19:10:48 <nirik> ?
19:10:50 <mboddu> May be 13, 17, 23?
19:10:55 <mboddu> Sure
19:11:08 <mboddu> Since rawhide finishes before branched
19:11:08 <otaylor> cverna: Do you have exactly what debugging code you added into koji?
19:11:30 <otaylor> cverna: I was asking on #koji, but then realized that I didn't have a good sense of what the debug output represents
19:11:31 <nirik> mboddu: sure, worth a try
19:11:53 <nirik> .ticket 8680
19:11:53 <smooge> nirik, do you want me to put this into the ticket or are you doing so?
19:11:54 <zodbot> nirik: Issue #8680: My app in Communishift needs to send emails - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8680
19:11:58 <nirik> smooge: go ahead
19:12:15 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8679 https://pagure.io/fedora-infrastructure/issue/8679
19:12:16 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#8679: "F32 mirrors problems?" https://pagure.io/fedora-infrastructure/issue/8679#comment-628357
19:12:18 <nirik> so I can answer this one and close it: we have no smtp server for communishift, so I guess roll your own.
19:12:56 <cverna> otaylor: yes I just logged chunk before that line (https://pagure.io/koji/blob/master/f/hub/kojixmlrpc.py)
19:13:07 <cverna> nirik: +1
19:13:11 <smooge> ok nirik will let you do that
19:13:11 <fm-admin> pagure.issue.edit -- kevin edited the close_status and status fields of ticket fedora-infrastructure#8680 https://pagure.io/fedora-infrastructure/issue/8680
19:13:12 <fm-admin> pagure.issue.comment.added -- kevin commented on ticket fedora-infrastructure#8680: "My app in Communishift needs to send emails" https://pagure.io/fedora-infrastructure/issue/8680#comment-628360
19:13:16 <otaylor> cverna: but how did you log it?
19:13:18 <smooge> 'let'
19:13:27 <nirik> .ticket 8681
19:13:28 <zodbot> nirik: Issue #8681: How to request branch for modules ? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8681
19:13:40 <cverna> otaylor: self.logger.error(chunk)
19:13:48 <nirik> mboddu: do you knwo the process here? it's just a regular scm request isn't it?
19:14:00 <otaylor> cverna: do you know if we're running koji as python2 or python3?
19:14:08 <nirik> otaylor: the hub is python2
19:14:09 <cverna> otaylor: python2
19:14:50 <fm-admin> pagure.issue.edit -- mohanboddu edited the priority fields of ticket fedora-infrastructure#8679 https://pagure.io/fedora-infrastructure/issue/8679
19:14:51 <fm-admin> pagure.issue.comment.added -- mohanboddu commented on ticket fedora-infrastructure#8679: "F32 mirrors problems?" https://pagure.io/fedora-infrastructure/issue/8679#comment-628363
19:15:19 <smooge> ok on 8681.. I don't know
19:15:41 <mboddu> cverna: Yes, to request a branch in modules, they need to provide --namespace modules, I can comment on the ticket
19:15:45 <nirik> it looks like it is and they have been processed.
19:15:50 <nirik> so I think we can close this one
19:16:09 <nirik> .ticket 8682
19:16:13 <zodbot> nirik: Issue #8682: DNS update: testdays.fedorainfracloud.org - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8682
19:16:20 <nirik> close?
19:16:39 <cverna> we can't do much about this ?
19:16:42 <otaylor> cverna: The question is when the logger shows 'Bo\xeete' - what actually was the input bytes
19:17:02 <nirik> otaylor: does --debug-xmlrpc on the client get the info ?
19:17:06 <smooge> otaylor, could you come back in 15 minutes? We are trying to get a standup meeting done
19:17:42 <fm-admin> pagure.issue.comment.added -- mohanboddu commented on ticket fedora-infrastructure#8681: "How to request branch for modules ?" https://pagure.io/fedora-infrastructure/issue/8681#comment-628370
19:17:53 <nirik> smooge: so, we can't do the fedoraproject.org one currently. I guess lets list their options to them and see what they want to do?
19:17:55 <otaylor> smooge: sure
19:18:05 <smooge> nirik, so we can't do either of them
19:18:06 <cverna> yes sorry otaylor, I ll ping you when we are done
19:18:08 * otaylor steps back
19:18:31 <fm-admin> pagure.issue.edit -- mohanboddu edited the close_status and status fields of ticket fedora-infrastructure#8681 https://pagure.io/fedora-infrastructure/issue/8681
19:18:45 <mboddu> I closed #8681
19:18:57 <smooge> what I think needs to be done is a proxy which redirects
19:18:57 <nirik> ah, I see... yes, we can, but it's a bit more work... we need to have them get a letsencrypt cert there.
19:19:54 <smooge> or that... since they will need to answer for testdays.fedorainfracloud.org and the cert coming back is *.apps.os.fedorainfracloud.org
19:20:40 <nirik> right, so we have cert-manager setup in communishift. Should be easily possible for them to ask it for a cert/etc
19:20:45 <nirik> I'd have to look up the details.
19:21:03 <nirik> so, leave open, move to waiting on asignee...
19:21:05 <nirik> ?
19:21:07 <smooge> the fedoraproject.org one might be easier cause I can just have the proxies put in a redirect to whatever url works in *.apps.os.fedorainfracloiug
19:21:17 <nirik> there was also talk about moving this app into our stg/prod clusters.
19:21:24 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8682 https://pagure.io/fedora-infrastructure/issue/8682
19:21:31 <nirik> yeah, we could go via proxies too... sure.
19:21:47 <nirik> that would also work if we ever move it into the internal clusters.
19:22:13 <smooge> ok I was working on that before the meeting so I will try and finish that
19:22:26 <nirik> .ticket 8683
19:22:27 <zodbot> nirik: Issue #8683: Fedmsg sometimes inserts unhelpful / unrelated messages into #fedora-design - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8683
19:22:43 <nirik> yeah, so this is the fedmsg irc bot filtering... it's... not very flexable.
19:23:03 <nirik> also reminds me: we need to replace it with a fedora-messaging version. :)
19:23:18 <nirik> so, I guess waiting on asignee... and we can look at it someday.
19:23:29 <nirik> .ticket 8684
19:23:30 <zodbot> nirik: Issue #8684: docs.stg.fp.o doesn't build, openshift cronjob confused - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8684
19:23:54 <cverna> I commented on that I think they should be able to delete the object using ansible
19:23:55 <nirik> cverna already suggested a workaround/fix for this
19:23:57 <nirik> yeah
19:24:36 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8683 https://pagure.io/fedora-infrastructure/issue/8683
19:24:43 <cverna> we can move to waiting on external I guess
19:24:48 <nirik> yeah.
19:24:54 <nirik> thats all the needs reviews...
19:24:55 <fm-admin> pagure.issue.edit -- cverna edited the priority fields of ticket fedora-infrastructure#8684 https://pagure.io/fedora-infrastructure/issue/8684
19:25:01 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8684 https://pagure.io/fedora-infrastructure/issue/8684
19:25:03 <fm-admin> pagure.issue.tag.added -- cverna tagged ticket fedora-infrastructure#8684: OpenShift https://pagure.io/fedora-infrastructure/issue/8684
19:25:05 <fm-admin> pagure.issue.edit -- cverna edited the priority fields of ticket fedora-infrastructure#8684 https://pagure.io/fedora-infrastructure/issue/8684
19:25:14 <cverna> ahah sorry smooge
19:25:47 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8679 https://pagure.io/fedora-infrastructure/issue/8679
19:26:18 <nirik> any others we want to discuss today?
19:26:33 <cverna> do we have anything on releng ?
19:27:19 <mboddu> Yes
19:27:33 <mboddu> .releng 9275
19:27:34 <zodbot> mboddu: Issue #9275: tag Haskell_Stackage_LTS_14 builds into f33 - releng - Pagure.io - https://pagure.io/releng/issue/9275
19:27:52 <mboddu> I tagged the builds into f33-updates-candidate, but nothing is happening
19:29:12 <mboddu> Based on https://docs.fedoraproject.org/en-US/rawhide-gating/_images/Simplified_single_build_update.png - bodhi should pick them up and tag into f33-signing-pending and sign the builds and move on...
19:29:25 <mboddu> Did I miss something?
19:29:37 <cverna> ok I can look at this with you mboddu
19:29:53 <mboddu> cverna: Okay
19:30:37 <nirik> might just be taking a bit
19:30:43 <cverna> mboddu: I am not sure bodhi will accept a build with .fc32 in a f33 release
19:31:10 <cverna> there are some validators that checks that if remember correctly
19:31:20 <nirik> thats... wrong
19:31:47 <mboddu> nirik: Well, its kinda wrong
19:32:06 <nirik> well, I guess bodhi never hits it, but other dist tag packages are in any release.
19:32:14 <cverna> I might be wrong too :P
19:32:18 <mboddu> After branching, all the builds will have .fc33 dist tag, so its fine, until we hit a edge case like this
19:32:19 <langdon> if i podman pull registry.fedoraproject.org/fedora i get a bad ssl cert
19:32:26 <nirik> also, some packages like 'fedora-repos' have no dist tag
19:32:45 <langdon> i also seem to get it if i hit it in a web-browser.. (but not registry.fp.o itself, just the cdn)
19:32:54 <langdon> https://cdn.registry.fedoraproject.org/v2/fedora/blobs/sha256:294cdb5bd8ee428966c9b23fae954f56d5ddd22c972ebf394d5dacce16759646: x509: certificate has expired or is not yet valid
19:32:55 <smooge> langdon, we are in a meeting please ask in 5 minutes
19:32:56 <langdon> just me?
19:33:03 <langdon> smooge: ohh sure
19:33:11 <nirik> so anything else, or shall we close out?
19:33:17 <smooge> speaking of which we are over.. so I wil close out
19:33:21 <smooge> #endmeeting