18:00:43 #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:00:43 Meeting started Thu Jun 18 18:00:43 2020 UTC. 18:00:43 This meeting is logged and archived in a public location. 18:00:43 The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:43 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:43 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:43 #chair cverna mboddu nirik smooge 18:00:43 #meetingname fedora_infrastructure_ops_daily_standup_meeting 18:00:43 #info meeting is 30 minutes MAX. At the end of 30, its stops 18:00:43 Current chairs: cverna mboddu nirik smooge 18:00:43 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:43 #info agenda is at https://board.net/p/fedora-infra-daily 18:00:45 #topic Tickets needing review 18:00:46 #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:00:50 * mboddu is here 18:01:10 .ticket 9045 18:01:11 nirik: Issue #9045: Access to FAS for AAA migration - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9045 18:01:18 * mboddu can update the tickets 18:01:45 hello 18:01:46 sorry 18:01:50 was fixing something 18:01:53 I'm not sure what to do here. It's a reasonable request, but it makes my teeth itch. I guess move it to waiting on asignee, high-trouble/high-gain/groomed 18:02:08 and I can ponder on how to do it safely 18:02:09 Hello 18:02:37 hi cverna 18:02:39 ack 18:02:42 +1 18:02:54 pagure.issue.tag.added -- mohanboddu tagged ticket fedora-infrastructure#9045: groomed, high-gain, and high-trouble https://pagure.io/fedora-infrastructure/issue/9045 18:02:55 pagure.issue.edit -- mohanboddu edited the priority fields of ticket fedora-infrastructure#9045 https://pagure.io/fedora-infrastructure/issue/9045 18:03:10 hi smooge 18:03:12 perhaps we can anonomize it, not sure. 18:03:20 pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#9047: "set koji builder hostnames in reverse dns" https://pagure.io/fedora-infrastructure/issue/9047#comment-659132 18:03:22 .ticket 9047 18:03:23 nirik: Issue #9047: set koji builder hostnames in reverse dns - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9047 18:03:56 is this one done then smooge ? 18:04:07 seems to be 18:04:08 ansible is still running and I have not been able to test 18:04:46 yeah, it's working 18:04:55 [root@buildvm-s390x-11 ~][PROD]# host 10.16.0.11 18:04:55 11.0.16.10.in-addr.arpa domain name pointer buildvm-s390x-01.s390.fedoraproject.org. 18:04:56 pagure.issue.edit -- smooge edited the close_status and status fields of ticket fedora-infrastructure#9047 https://pagure.io/fedora-infrastructure/issue/9047 18:04:57 pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#9047: "set koji builder hostnames in reverse dns" https://pagure.io/fedora-infrastructure/issue/9047#comment-659135 18:05:18 .ticket 9048 18:05:19 nirik: Issue #9048: Set sidetag_debuginfo_allowed/sidetag_rpm_macros_allowed on fXX-build - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9048 18:05:31 I have some concerns here... I need to add them to the ticket. 18:05:56 I am not sure who best to decide this actually... perhaps FPC? fesco? I am not sure we should by ourselves 18:06:14 proposed: waiting on external, low-trouble,medium-gain,groomed? 18:06:18 I think we should punt it to FESCo 18:06:19 ask FESCo and see if they send us to FPC? 18:06:39 +1 to pingou suggestion and the proposed 18:06:39 perhaps we should form a new comittee to decide which comittee to send it to. ;) 18:06:47 Haha :) 18:06:53 +1 more committees 18:06:54 know that the idea of side tags is that they end up being merged 18:07:01 I like that idea 18:07:02 right. 18:07:03 :p 18:07:15 pagure.issue.tag.added -- mohanboddu tagged ticket fedora-infrastructure#9048: groomed, low-trouble, and medium-gain https://pagure.io/fedora-infrastructure/issue/9048 18:07:16 pagure.issue.edit -- mohanboddu edited the priority fields of ticket fedora-infrastructure#9048 https://pagure.io/fedora-infrastructure/issue/9048 18:07:17 pagure.issue.comment.added -- mohanboddu commented on ticket fedora-infrastructure#9048: "Set sidetag_debuginfo_allowed/sidetag_rpm_macros_allowed on fXX-build" https://pagure.io/fedora-infrastructure/issue/9048#comment-659137 18:07:27 the problem here is if you change macros, it could be that it wouldn't build outside a side tag anymore. Or would build very differently 18:07:41 nirik: we should involve the council in the creation of the new comittee 18:07:59 nirik: +1, I've the same concerns there 18:08:13 yup, thats aproblem 18:08:19 the debuginfo part I don't care about. thats fine. ;) 18:08:31 anyhow, onward 18:08:39 .ticket 9049 18:08:40 nirik: Issue #9049: Please update ipsilon in prod. - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9049 18:09:11 ... 18:09:24 waiting on asignee, med/med/groomed? 18:09:29 I have a lot of swearing involved 18:09:32 we aren't going to install from copr. 18:09:36 +1 18:09:39 1. its not going to be installed from copr 18:09:48 2. its not puiterwijk's job 18:09:54 pagure.issue.tag.added -- mohanboddu tagged ticket fedora-infrastructure#9049: groomed, medium-gain, and medium-trouble https://pagure.io/fedora-infrastructure/issue/9049 18:09:55 pagure.issue.edit -- mohanboddu edited the priority fields of ticket fedora-infrastructure#9049 https://pagure.io/fedora-infrastructure/issue/9049 18:10:05 that also 18:10:22 smooge: Patrick replied on irc to him earlier so I think this was "tag as an FYI" 18:10:31 I can walk someone thru the update tho... it's basically just run playbook, start-build in openshift 18:10:45 (once it's in infra tags or f32) 18:10:59 something for Mark ? 18:11:07 and me 18:11:08 ok, thats all the needs-review. anything from releng? 18:11:12 or Vipul? 18:11:21 I would like to learn on building something in openshift etc 18:11:30 Completed the new tickets in releng, nothing for needs-review 18:11:49 * mboddu can also tag along to update ipsilon 18:12:23 I'm happy to schedule a group time to do it? everyone interested can join? 18:12:30 +1 18:12:30 of course finding time is always fun. ;( 18:12:51 I had a few things... 18:12:55 pagure.issue.comment.added -- ignatenkobrain commented on ticket fedora-infrastructure#9048: "Set sidetag_debuginfo_allowed/sidetag_rpm_macros_allowed on fXX-build" https://pagure.io/fedora-infrastructure/issue/9048#comment-659142 18:13:15 cverna: so, whats the current status of the side tags in bodhi? are we deleting them as soon as the update is created? 18:14:25 I pushed the change so Koji should be deleting them once the last build was untagged from it 18:14:31 I have not check if this is happening tho 18:14:45 so, doesn't that break the user flow that churchyard wanted? 18:15:09 so if you need to do another build, you need to submit overrides and just do a new side tag? 18:15:26 pagure.issue.comment.added -- ngompa commented on ticket fedora-infrastructure#9049: "Please update ipsilon in prod." https://pagure.io/fedora-infrastructure/issue/9049#comment-659145 18:15:48 That workflow needs more work in bodhi and not really trivial work 18:16:04 ok. 18:16:19 I just wasn't sure what the current state/plan is... 18:16:55 Currently for normal release we delete the tag when the update is created 18:16:57 so the testing-pending and signing-pending tags should only be there when it's merging and removed by koji after? along with the side-tag? 18:17:56 Testing-pending and signing-pending are still deleted by bodhi 18:18:12 ok, we have 105 of those currently. :) Should it be safe to remove them? 18:18:14 These tags only exists in rawhide 18:18:20 And we delete them when the update reach stable 18:18:22 pagure.issue.comment.added -- tflink commented on ticket fedora-infrastructure#9047: "set koji builder hostnames in reverse dns" https://pagure.io/fedora-infrastructure/issue/9047#comment-659147 18:18:34 Yeah 18:19:01 so, they could exist for 5-10min or something normally? 18:19:06 They were not deleted because of the Koji issue with admin not being able to delete sidetags 18:19:27 Yes that's it 18:19:44 ok, I will go delete them after this meeting. 18:19:59 We can safely delete them since bodhi check if they exist before using them 18:20:05 it could be also updates that were never pushed to stable 18:20:09 If they don't we create them 18:20:29 never passed gating before branching for example 18:20:36 Yeah we have a few podman updates that are stuck there 18:20:39 on the kojira bug where it wasn't deleting expired repos... we have deleted I think about 40TB in the last 2 days now. 18:20:46 In testing 18:20:55 But they don't seems to care about it 18:20:56 ah, so I shouldn't delete them? 18:21:24 happily we have 80 side tags and 80 side tag targets right now, so that matches up ok. 18:22:10 Also I think this is mostly the monitor gating updatea 18:22:11 I could delete all the ones that don't have a side-tag ? ie, f33-build-side-xxx doesn't exist, and f33-build-side-xxx-signing-pending and testing-pending exist 18:22:28 I don't think we have other side tags update currently 18:22:51 nirik: yeah sounds safe 18:23:14 ok. 18:23:38 I had one other thing, but having trouble remembering it. 18:23:53 oh, how is osbs? did you see my note about those secrets? 18:24:08 mobrien and I have been working on osbs 18:24:26 We fixed a couple issue but got block on another one 18:24:41 I think mark was going to try something 18:25:04 ok. Cool. I couldn't figure out where those secrets were supposed to be. 18:25:18 So the secret bit is fixed, but osbs is still not running 18:25:26 filed in a cabinet in Alpha Centauri? 18:25:41 No this was something that could be removed. It was removed in staging bit not in prod 18:26:12 ah, ok. 18:26:25 alright, thats all I had... 18:26:29 anything else from anyone? 18:26:34 not from me 18:26:35 fasClient? 18:26:52 pingou: oh yeah, you want to try your hack/fix? 18:26:57 ansible, not ansible, its own repo, its own rpm, keeping the current situation? :) 18:27:29 note that (hopefully) fas client will be going away in favor of noggin before too long 18:27:45 so I think it's own repo is too much. 18:28:07 we arent sure this will fix it either are we? 18:28:29 well it no longer spew the error 18:28:43 and it's running restorecon as root, which I think is pretty safe to do 18:28:56 so I'm not too worried 18:29:40 I think we should just add the patch to the src.rpm and rebuild it. 18:30:04 I'm worried if we put it in ansible that we aren't accounting for all the different os versions/releases... 18:30:28 and it looks to be only F32, rhel8 (testing only in pagure-stg01 though) seemed fine 18:30:29 where with the src.rpm we can just rebuild for el8and f32 rpms 18:30:43 ah, ok. I thought I saw one, but might be mistaken 18:31:31 only tested pagure-stg01 but it didn't complain :) 18:31:51 ok. the complaints come in a flurry of emails for the daily crons 18:33:00 let's see if we can reduce this for tomorrow :) 18:33:30 sounds good. 18:33:46 oh, if you didn't see, the fedora-messaging for pungi worked on bodhi-backend01. 18:33:57 we need a config.toml for compose-rawhide01 18:34:08 col 18:34:08 and I guess the new-updates-sync still needs moving 18:34:11 cool 18:34:22 new-updates-sync? 18:35:17 the script that syncs updates from bodhi-backend01 to master mirrors... it was emiting fedmsgs saying 'I synced f32-updates it was X changed files, Y size' 18:35:32 not urgent, but kinda nice to see it syncing 18:35:35 anyhow, we are over. 18:35:38 #endmeeting