18:00:43 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
18:00:43 <zodbot> Meeting started Thu Jun 18 18:00:43 2020 UTC.
18:00:43 <zodbot> This meeting is logged and archived in a public location.
18:00:43 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:43 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:43 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:43 <nirik> #chair cverna mboddu nirik smooge
18:00:43 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting
18:00:43 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops
18:00:43 <zodbot> Current chairs: cverna mboddu nirik smooge
18:00:43 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
18:00:43 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily
18:00:45 <nirik> #topic Tickets needing review
18:00:46 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
18:00:50 * mboddu is here
18:01:10 <nirik> .ticket 9045
18:01:11 <zodbot> nirik: Issue #9045: Access to FAS for AAA migration - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9045
18:01:18 * mboddu can update the tickets
18:01:45 <smooge> hello
18:01:46 <smooge> sorry
18:01:50 <smooge> was fixing something
18:01:53 <nirik> I'm not sure what to do here. It's a reasonable request, but it makes my teeth itch. I guess move it to waiting on asignee, high-trouble/high-gain/groomed
18:02:08 <nirik> and I can ponder on how to do it safely
18:02:09 <cverna> Hello
18:02:37 <smooge> hi cverna
18:02:39 <mboddu> ack
18:02:42 <mboddu> +1
18:02:54 <fm-admin_> pagure.issue.tag.added -- mohanboddu tagged ticket fedora-infrastructure#9045: groomed, high-gain, and high-trouble https://pagure.io/fedora-infrastructure/issue/9045
18:02:55 <fm-admin_> pagure.issue.edit -- mohanboddu edited the priority fields of ticket fedora-infrastructure#9045 https://pagure.io/fedora-infrastructure/issue/9045
18:03:10 <cverna> hi smooge
18:03:12 <nirik> perhaps we can anonomize it, not sure.
18:03:20 <fm-admin_> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#9047: "set koji builder hostnames in reverse dns" https://pagure.io/fedora-infrastructure/issue/9047#comment-659132
18:03:22 <nirik> .ticket 9047
18:03:23 <zodbot> nirik: Issue #9047: set koji builder hostnames in reverse dns - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9047
18:03:56 <nirik> is this one done then smooge ?
18:04:07 <nirik> seems to be
18:04:08 <smooge> ansible is still running and I have not been able to test
18:04:46 <nirik> yeah, it's working
18:04:55 <nirik> [root@buildvm-s390x-11 ~][PROD]# host 10.16.0.11
18:04:55 <nirik> 11.0.16.10.in-addr.arpa domain name pointer buildvm-s390x-01.s390.fedoraproject.org.
18:04:56 <fm-admin_> pagure.issue.edit -- smooge edited the close_status and status fields of ticket fedora-infrastructure#9047 https://pagure.io/fedora-infrastructure/issue/9047
18:04:57 <fm-admin_> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#9047: "set koji builder hostnames in reverse dns" https://pagure.io/fedora-infrastructure/issue/9047#comment-659135
18:05:18 <nirik> .ticket 9048
18:05:19 <zodbot> nirik: Issue #9048: Set sidetag_debuginfo_allowed/sidetag_rpm_macros_allowed on fXX-build - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9048
18:05:31 <nirik> I have some concerns here... I need to add them to the ticket.
18:05:56 <nirik> I am not sure who best to decide this actually... perhaps FPC? fesco? I am not sure we should by ourselves
18:06:14 <nirik> proposed: waiting on external, low-trouble,medium-gain,groomed?
18:06:18 <mboddu> I think we should punt it to FESCo
18:06:19 <pingou> ask FESCo and see if they send us to FPC?
18:06:39 <mboddu> +1 to pingou suggestion and the proposed
18:06:39 <nirik> perhaps we should form a new comittee to decide which comittee to send it to. ;)
18:06:47 <mboddu> Haha :)
18:06:53 <nb> +1 more committees
18:06:54 <pingou> know that the idea of side tags is that they end up being merged
18:07:01 <cverna> I like that idea
18:07:02 <nirik> right.
18:07:03 <cverna> :p
18:07:15 <fm-admin_> pagure.issue.tag.added -- mohanboddu tagged ticket fedora-infrastructure#9048: groomed, low-trouble, and medium-gain https://pagure.io/fedora-infrastructure/issue/9048
18:07:16 <fm-admin_> pagure.issue.edit -- mohanboddu edited the priority fields of ticket fedora-infrastructure#9048 https://pagure.io/fedora-infrastructure/issue/9048
18:07:17 <fm-admin_> pagure.issue.comment.added -- mohanboddu commented on ticket fedora-infrastructure#9048: "Set sidetag_debuginfo_allowed/sidetag_rpm_macros_allowed on fXX-build" https://pagure.io/fedora-infrastructure/issue/9048#comment-659137
18:07:27 <nirik> the problem here is if you change macros, it could be that it wouldn't build outside a side tag anymore. Or would build very differently
18:07:41 <pingou> nirik: we should involve the council in the creation of the new comittee
18:07:59 <pingou> nirik: +1, I've the same concerns there
18:08:13 <mboddu> yup, thats aproblem
18:08:19 <nirik> the debuginfo part I don't care about. thats fine. ;)
18:08:31 <nirik> anyhow, onward
18:08:39 <nirik> .ticket 9049
18:08:40 <zodbot> nirik: Issue #9049: Please update ipsilon in prod. - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9049
18:09:11 <smooge> ...
18:09:24 <nirik> waiting on asignee, med/med/groomed?
18:09:29 <smooge> I have a lot of swearing involved
18:09:32 <nirik> we aren't going to install from copr.
18:09:36 <mboddu> +1
18:09:39 <smooge> 1. its not going to be installed from copr
18:09:48 <smooge> 2. its not puiterwijk's job
18:09:54 <fm-admin_> pagure.issue.tag.added -- mohanboddu tagged ticket fedora-infrastructure#9049: groomed, medium-gain, and medium-trouble https://pagure.io/fedora-infrastructure/issue/9049
18:09:55 <fm-admin_> pagure.issue.edit -- mohanboddu edited the priority fields of ticket fedora-infrastructure#9049 https://pagure.io/fedora-infrastructure/issue/9049
18:10:05 <nirik> that also
18:10:22 <pingou> smooge: Patrick replied on irc to him earlier so I think this was "tag as an FYI"
18:10:31 <nirik> I can walk someone thru the update tho... it's basically just run playbook, start-build in openshift
18:10:45 <nirik> (once it's in infra tags or f32)
18:10:59 <pingou> something for Mark ?
18:11:07 <smooge> and me
18:11:08 <nirik> ok, thats all the needs-review. anything from releng?
18:11:12 <pingou> or Vipul?
18:11:21 <smooge> I would like to learn on building something in openshift etc
18:11:30 <mboddu> Completed the new tickets in releng, nothing for needs-review
18:11:49 * mboddu can also tag along to update ipsilon
18:12:23 <nirik> I'm happy to schedule a group time to do it? everyone interested can join?
18:12:30 <mboddu> +1
18:12:30 <nirik> of course finding time is always fun. ;(
18:12:51 <nirik> I had a few things...
18:12:55 <fm-admin_> pagure.issue.comment.added -- ignatenkobrain commented on ticket fedora-infrastructure#9048: "Set sidetag_debuginfo_allowed/sidetag_rpm_macros_allowed on fXX-build" https://pagure.io/fedora-infrastructure/issue/9048#comment-659142
18:13:15 <nirik> cverna: so, whats the current status of the side tags in bodhi? are we deleting them as soon as the update is created?
18:14:25 <cverna> I pushed the change so Koji should be deleting them once the last build was untagged from it
18:14:31 <cverna> I have not check if this is happening tho
18:14:45 <nirik> so, doesn't that break the user flow that churchyard wanted?
18:15:09 <nirik> so if you need to do another build, you need to submit overrides and just do a new side tag?
18:15:26 <fm-admin_> pagure.issue.comment.added -- ngompa commented on ticket fedora-infrastructure#9049: "Please update ipsilon in prod." https://pagure.io/fedora-infrastructure/issue/9049#comment-659145
18:15:48 <cverna> That workflow needs more work in bodhi and not really trivial work
18:16:04 <nirik> ok.
18:16:19 <nirik> I just wasn't sure what the current state/plan is...
18:16:55 <cverna> Currently for normal release we delete the tag when the update is created
18:16:57 <nirik> so the testing-pending and signing-pending tags should only be there when it's merging and removed by koji after? along with the side-tag?
18:17:56 <cverna> Testing-pending and signing-pending are still deleted by bodhi
18:18:12 <nirik> ok, we have 105 of those currently. :) Should it be safe to remove them?
18:18:14 <cverna> These tags only exists in rawhide
18:18:20 <cverna> And we delete them when the update reach stable
18:18:22 <fm-admin_> pagure.issue.comment.added -- tflink commented on ticket fedora-infrastructure#9047: "set koji builder hostnames in reverse dns" https://pagure.io/fedora-infrastructure/issue/9047#comment-659147
18:18:34 <cverna> Yeah
18:19:01 <nirik> so, they could exist for 5-10min or something normally?
18:19:06 <cverna> They were not deleted because of the Koji issue with admin not being able to delete sidetags
18:19:27 <cverna> Yes that's it
18:19:44 <nirik> ok, I will go delete them after this meeting.
18:19:59 <cverna> We can safely delete them since bodhi check if they exist before using them
18:20:05 <pingou> it could be also updates that were never pushed to stable
18:20:09 <cverna> If they don't we create them
18:20:29 <pingou> never passed gating before branching for example
18:20:36 <cverna> Yeah we have a few podman updates that are stuck there
18:20:39 <nirik> on the kojira bug where it wasn't deleting expired repos... we have deleted I think about 40TB in the last 2 days now.
18:20:46 <cverna> In testing
18:20:55 <cverna> But they don't seems to care about it
18:20:56 <nirik> ah, so I shouldn't delete them?
18:21:24 <nirik> happily we have 80 side tags and 80 side tag targets right now, so that matches up ok.
18:22:10 <cverna> Also I think this is mostly the monitor gating updatea
18:22:11 <nirik> I could delete all the ones that don't have a side-tag ? ie, f33-build-side-xxx doesn't exist, and f33-build-side-xxx-signing-pending and testing-pending exist
18:22:28 <cverna> I don't think we have other side tags update currently
18:22:51 <cverna> nirik: yeah sounds safe
18:23:14 <nirik> ok.
18:23:38 <nirik> I had one other thing, but having trouble remembering it.
18:23:53 <nirik> oh, how is osbs? did you see my note about those secrets?
18:24:08 <cverna> mobrien and I have been working on osbs
18:24:26 <cverna> We fixed a couple issue but got block on another one
18:24:41 <cverna> I think mark was going to try something
18:25:04 <nirik> ok. Cool. I couldn't figure out where those secrets were supposed to be.
18:25:18 <cverna> So the secret bit is fixed, but osbs is still not running
18:25:26 <smooge> filed in a cabinet in Alpha Centauri?
18:25:41 <cverna> No this was something that could be removed. It was removed in staging bit not in prod
18:26:12 <nirik> ah, ok.
18:26:25 <nirik> alright, thats all I had...
18:26:29 <nirik> anything else from anyone?
18:26:34 <smooge> not from me
18:26:35 <pingou> fasClient?
18:26:52 <nirik> pingou: oh yeah, you want to try your hack/fix?
18:26:57 <pingou> ansible, not ansible, its own repo, its own rpm, keeping the current situation? :)
18:27:29 <nirik> note that (hopefully) fas client will be going away in favor of noggin before too long
18:27:45 <nirik> so I think it's own repo is too much.
18:28:07 <nirik> we arent sure this will fix it either are we?
18:28:29 <pingou> well it no longer spew the error
18:28:43 <pingou> and it's running restorecon as root, which I think is pretty safe to do
18:28:56 <pingou> so I'm not too worried
18:29:40 <nirik> I think we should just add the patch to the src.rpm and rebuild it.
18:30:04 <nirik> I'm worried if we put it in ansible that we aren't accounting for all the different os versions/releases...
18:30:28 <pingou> and it looks to be only F32, rhel8 (testing only in pagure-stg01 though) seemed fine
18:30:29 <nirik> where with the src.rpm we can just rebuild for el8and f32 rpms
18:30:43 <nirik> ah, ok. I thought I saw one, but might be mistaken
18:31:31 <pingou> only tested pagure-stg01 but it didn't complain :)
18:31:51 <nirik> ok. the complaints come in a flurry of emails for the daily crons
18:33:00 <pingou> let's see if we can reduce this for tomorrow :)
18:33:30 <nirik> sounds good.
18:33:46 <nirik> oh, if you didn't see, the fedora-messaging for pungi worked on bodhi-backend01.
18:33:57 <nirik> we need a config.toml for compose-rawhide01
18:34:08 <pingou> col
18:34:08 <nirik> and I guess the new-updates-sync still needs moving
18:34:11 <pingou> cool
18:34:22 <pingou> new-updates-sync?
18:35:17 <nirik> the script that syncs updates from bodhi-backend01 to master mirrors... it was emiting fedmsgs saying 'I synced f32-updates it was X changed files, Y size'
18:35:32 <nirik> not urgent, but kinda nice to see it syncing
18:35:35 <nirik> anyhow, we are over.
18:35:38 <nirik> #endmeeting