2024-01-16 17:00:38 <@humaton:fedora.im> !startmeeting RELENG (2024-01-16)
2024-01-16 17:00:39 <@meetbot:fedora.im> Meeting started at 2024-01-16 17:00:38 UTC
2024-01-16 17:00:39 <@meetbot:fedora.im> The Meeting name is 'RELENG (2024-01-16)'
2024-01-16 17:00:46 <@humaton:fedora.im> !meetingname releng
2024-01-16 17:00:51 <@humaton:fedora.im> !chair nirik sharkcz pbrobinson phsmoura dustymabe jednorozec
2024-01-16 17:00:53 <@humaton:fedora.im> !topic init process
2024-01-16 17:01:27 <@nirik:matrix.scrye.com> !hi
2024-01-16 17:01:29 <@zodbot:fedora.im> Kevin Fenzi (kevin) - he / him / his
2024-01-16 17:01:50 <@humaton:fedora.im> !hi
2024-01-16 17:01:51 <@zodbot:fedora.im> Tomáš Hrčka (humaton) - he / him / his
2024-01-16 17:01:53 <@lenkaseg:fedora.im> !hi
2024-01-16 17:01:54 <@zodbot:fedora.im> Lenka Segura (lenkaseg)
2024-01-16 17:01:57 <@leo:fedora.im> !hi
2024-01-16 17:01:59 <@zodbot:fedora.im> Leo Puvilland (leo) - he / him / his
2024-01-16 17:02:06 <@leo:fedora.im> just watching :)
2024-01-16 17:02:19 <@patrikp:matrix.org> Good evening to all.
2024-01-16 17:02:30 <@pcreech:matrix.org> !hello pcreech17
2024-01-16 17:02:31 <@zodbot:fedora.im> Patrick Creech (pcreech17)
2024-01-16 17:04:19 <@humaton:fedora.im> hello everybody, I hope you have a lovely day. Lets start with the agenda.
2024-01-16 17:04:22 <@humaton:fedora.im> .releng 11888
2024-01-16 17:04:26 <@humaton:fedora.im> !releng 11888
2024-01-16 17:04:27 <@zodbot:fedora.im> **releng #11888** (https://pagure.io/releng/issue/11888):**Fedora 40 Mass Rebuild Tracker**

● **Opened:** a day ago by jnsamyak
● **Last Updated:** 6 hours ago
● **Assignee:** Not Assigned
2024-01-16 17:04:52 <@humaton:fedora.im> So mass rebuild is scheduled to happen in about 24 hours
2024-01-16 17:05:15 <@humaton:fedora.im> changes to releng scripts and ansible should have appropriate PRs open
2024-01-16 17:05:27 <@humaton:fedora.im> But 
2024-01-16 17:05:31 <@humaton:fedora.im> !fesco 3145
2024-01-16 17:05:32 <@zodbot:fedora.im> **fesco #3145** (https://pagure.io/fesco/issue/3145):**Delay mass rebuild for Fedora 40 until f40-build-side-81394 side-tag is merged.**

● **Opened:** 3 hours ago by codonell
● **Last Updated:** 22 minutes ago
● **Assignee:** Not Assigned
2024-01-16 17:06:26 <@nirik:matrix.scrye.com> i've updated the builders (but still f38)... and we finally got the last bits to enable dnf5 for builds this morning, so thats enabled.
2024-01-16 17:06:50 <@nirik:matrix.scrye.com> and also applied a cve fix to them. so hopefully they are all ready
2024-01-16 17:06:58 <@jnsamyak:matrix.org> !hello jnsamyak
2024-01-16 17:06:59 <@zodbot:fedora.im> Samyak Jain (jnsamyak) - he / him / his
2024-01-16 17:07:03 <@jnsamyak:matrix.org> sorry I'm kate
2024-01-16 17:07:09 <@jnsamyak:matrix.org> s/kate/late
2024-01-16 17:07:41 <@nirik:matrix.scrye.com> hey Samyak...
2024-01-16 17:07:48 <@humaton:fedora.im> hey jnsamyak 
2024-01-16 17:08:04 <@humaton:fedora.im> SO I counted 4 +1 on the fesco ticket
2024-01-16 17:08:52 <@nirik:matrix.scrye.com> i'm not sure it really needed a vote. ;)
2024-01-16 17:08:55 <@humaton:fedora.im> if it gets through we will have to wait until the merge gets through
2024-01-16 17:09:05 <@jnsamyak:matrix.org> oh there's gonna be obe more +1 :)
2024-01-16 17:10:09 <@jnsamyak:matrix.org> oh there's gonna be one more +1 :)e
2024-01-16 17:10:24 <@nirik:matrix.scrye.com> sure
2024-01-16 17:12:01 <@humaton:fedora.im> ok so the bodhi update is not yet created
2024-01-16 17:12:16 <@humaton:fedora.im> should I go ahead and create it?
2024-01-16 17:12:20 <@nirik:matrix.scrye.com> no
2024-01-16 17:12:33 <@nirik:matrix.scrye.com> leave that to them when they are ready.
2024-01-16 17:12:35 <@nirik:matrix.scrye.com> imho
2024-01-16 17:13:16 <@humaton:fedora.im> ok so lets postpone the mass-rebuild for 1 day. Because this can easily endup merged in EU evening/night  
2024-01-16 17:13:23 <@humaton:fedora.im> and that leads to errors
2024-01-16 17:13:45 <@jnsamyak:matrix.org> +1
2024-01-16 17:14:24 <@nirik:matrix.scrye.com> yeah... agreed
2024-01-16 17:14:32 <@jnsamyak:matrix.org> also, I heard late and fast work is devil works so waiting a day will be best, I can update the epoch date in the prs if we all agree
2024-01-16 17:15:03 <@humaton:fedora.im> jnsamyak: Can you also send announcement or should I do that?
2024-01-16 17:15:34 <@jnsamyak:matrix.org> I can send one at devel-announce, in an hour post meeting :D
2024-01-16 17:16:14 <@humaton:fedora.im> oh I just checked and its 1045pm on your side 
2024-01-16 17:16:16 <@humaton:fedora.im> ouch
2024-01-16 17:17:33 <@humaton:fedora.im> !agreed Mass rebuild for the Fedora Linux 40 will be done on 2024-01-18
2024-01-16 17:18:22 <@nirik:matrix.scrye.com> hopefully theres no more delays. ;)
2024-01-16 17:18:56 <@jnsamyak:matrix.org> 🤞
2024-01-16 17:19:21 <@humaton:fedora.im> hope not but its much faster when we dont have to wait for aarch64 builds :)
2024-01-16 17:19:58 <@nirik:matrix.scrye.com> you mean armv7? ;)
2024-01-16 17:20:32 <@humaton:fedora.im> they should ahve named it leg64 and leave arm to be 32bit forever
2024-01-16 17:20:57 <@humaton:fedora.im> but also the new builders we got some time ago
2024-01-16 17:21:35 <@nirik:matrix.scrye.com> the slow ones these days seem to be ppc64le sadly
2024-01-16 17:22:22 <@humaton:fedora.im> Ok so lets move on.
2024-01-16 17:22:30 <@humaton:fedora.im> !releng 11870
2024-01-16 17:22:31 <@zodbot:fedora.im> **releng #11870** (https://pagure.io/releng/issue/11870):**Disable HTTP caches for repomd.xml files for Koji buildroots**

● **Opened:** a week ago by praiskup
● **Last Updated:** a week ago
● **Assignee:** Not Assigned
2024-01-16 17:23:15 <@nirik:matrix.scrye.com> saw it, too far down the pile for me to have loked into yet
2024-01-16 17:24:36 <@nirik:matrix.scrye.com> needs investigation. sadly theres lots of layers
2024-01-16 17:25:16 <@humaton:fedora.im> the proposed hotfix can be done by adjusting httpd config or?
2024-01-16 17:25:18 <@nirik:matrix.scrye.com> proxies / httpd  -> haproxy -> varnish -> httpd
2024-01-16 17:25:31 <@nirik:matrix.scrye.com> unknown. needs digging
2024-01-16 17:25:38 <@humaton:fedora.im> I see
2024-01-16 17:26:53 <@jnsamyak:matrix.org> should we change tag to investigation, and medium trouble? I think it's tagged wrong
2024-01-16 17:28:31 <@nirik:matrix.scrye.com> if you like... didn't know we had an investigation tag... need to investigate that
2024-01-16 17:28:42 <@humaton:fedora.im> :D
2024-01-16 17:28:50 <@humaton:fedora.im> I dont know about investigation tag
2024-01-16 17:29:05 <@jnsamyak:matrix.org> there is and i added it to the issue
2024-01-16 17:29:22 <@humaton:fedora.im> but what we agreed on previously was that medium trouble menas we need to investigate becouse we are not sure what is the root cause
2024-01-16 17:29:42 <@humaton:fedora.im> but what we agreed on previously was that medium trouble means we need to investigate because we are not sure what is the root cause
2024-01-16 17:29:50 <@jnsamyak:matrix.org> gotcha
2024-01-16 17:30:58 <@humaton:fedora.im> !releng 11800
2024-01-16 17:30:58 <@zodbot:fedora.im> **releng #11800** (https://pagure.io/releng/issue/11800):**Please add both obs-studio-libs and obs-studio-plugin-vkcapture to the multilib whitelist **

● **Opened:** a month ago by gloriouseggroll
● **Last Updated:** a month ago
● **Assignee:** Not Assigned
2024-01-16 17:31:09 <@humaton:fedora.im> I think jnsamyak did some PRs for this
2024-01-16 17:32:16 <@jnsamyak:matrix.org> yeah, I can work on this, should be a quick addition to pungi-config multilib once I read go through the whole issue
2024-01-16 17:34:27 <@humaton:fedora.im> Didnt you already do that?
2024-01-16 17:34:32 <@humaton:fedora.im> or it was another package?
2024-01-16 17:35:04 <@humaton:fedora.im> here it is https://pagure.io/pungi-fedora/pull-request/1224
2024-01-16 17:35:39 <@humaton:fedora.im> you just need to do the same for that config in ansible repo in the bodhi role
2024-01-16 17:38:12 <@humaton:fedora.im> once the ansible PR is there I can merge both and redeploy 
2024-01-16 17:38:22 <@jnsamyak:matrix.org> oh, I don't remember if it was the same package, but sure I can do that tomorrow
2024-01-16 17:39:10 <@jnsamyak:matrix.org> ok isee the pr and your comment there, sorry for missing it
2024-01-16 17:39:36 <@humaton:fedora.im> no worries :) I bet you I have bunch of abandoned PRs somewhere
2024-01-16 17:39:50 <@nirik:matrix.scrye.com> we all do ;)
2024-01-16 17:40:14 <@humaton:fedora.im> I just closed this one
2024-01-16 17:40:20 <@humaton:fedora.im> !releng 11826
2024-01-16 17:40:20 <@zodbot:fedora.im> **releng #11826** (https://pagure.io/releng/issue/11826):**Untretire EPEL branches for cxxopts**

● **Closed: Fixed with Explanation** 3 minutes ago by humaton
● **Opened:** a month ago by orion
● **Last Updated:** 2 minutes ago
● **Assignee:** Not Assigned
2024-01-16 17:40:32 <@humaton:fedora.im> this was a struggle with timings and automation :D
2024-01-16 17:40:35 <@nirik:matrix.scrye.com> yeah, how did you update pdc?
2024-01-16 17:40:46 <@humaton:fedora.im> I put it in the comment
2024-01-16 17:41:06 <@humaton:fedora.im> there is a script we have that uses client to update all slas for branch at once
2024-01-16 17:41:19 <@nirik:matrix.scrye.com> weird. I am pretry sure I used the same thing and it was telling me there was no epel8 branch.
2024-01-16 17:41:27 <@nirik:matrix.scrye.com> yeah, I know...
2024-01-16 17:41:33 <@humaton:fedora.im> weird
2024-01-16 17:41:46 <@humaton:fedora.im> but yeah I have seen some strange errors from PDC
2024-01-16 17:42:07 <@nirik:matrix.scrye.com> I sure hope we can kill it finally very soon.
2024-01-16 17:42:09 <@humaton:fedora.im> branch in api call in the web "UI" but nothing from client
2024-01-16 17:42:35 <@nirik:matrix.scrye.com> fun fact: pdc is the 2rd largest db we have on db01... (just behind hyperkitty)
2024-01-16 17:42:54 <@humaton:fedora.im> we will ceremonially burn the VM with release of 41
2024-01-16 17:43:07 <@humaton:fedora.im> composes composes
2024-01-16 17:43:15 <@humaton:fedora.im> there is a LOT of data in there
2024-01-16 17:43:29 <@nirik:matrix.scrye.com> yes, it used to store metadata on rpms too...
2024-01-16 17:43:34 <@nirik:matrix.scrye.com> anyhow, sidetrack
2024-01-16 17:44:05 <@nirik:matrix.scrye.com> although... how far are we from retiring pdc?
2024-01-16 17:44:35 <@humaton:fedora.im> from releng workflows I would say we have 50% covered
2024-01-16 17:45:07 <@humaton:fedora.im> but things will run in parallel for the 40 release cycle
2024-01-16 17:45:21 <@nirik:matrix.scrye.com> reminder: rhel7 goes eol on 2024-06-30... thats the hard deadline IMHO
2024-01-16 17:45:38 <@humaton:fedora.im> ok so we have to kill it at that date
2024-01-16 17:46:40 <@jnsamyak:matrix.org> sorry for interrupting folks, I drafted this email quickly, can someone read it through in case i missed any important details late night: https://hackmd.io/@2se5GAvIQGSWX9IGCuz2Tw/r1QukrNY6/edit
2024-01-16 17:47:27 <@humaton:fedora.im> go for it
2024-01-16 17:47:35 <@jnsamyak:matrix.org> shooot at us :D
2024-01-16 17:48:43 <@nirik:matrix.scrye.com> jnsamyak: looks fine to me, although I would change suppose to 'supposed' (which I know looks weird, but reads better). Or perhaps just change it to 'planned' ;)
2024-01-16 17:49:01 <@nirik:matrix.scrye.com> !releng 11823
2024-01-16 17:49:02 <@zodbot:fedora.im> **releng #11823** (https://pagure.io/releng/issue/11823):**please send openh264-2.3.1-4.fc40 to cisco**

● **Opened:** a month ago by kevin
● **Last Updated:** 2 days ago
● **Assignee:** patrikp
2024-01-16 17:49:14 <@nirik:matrix.scrye.com> whats the status here? no word from cisco?
2024-01-16 17:49:42 <@humaton:fedora.im> Weird thing happened here, they recieved all the parts of compose except for the first one
2024-01-16 17:49:59 <@humaton:fedora.im> So i resended it agan with response to theyr email before holidays
2024-01-16 17:50:02 <@nirik:matrix.scrye.com> :frowning
2024-01-16 17:50:13 <@humaton:fedora.im> and ofc it didnt go through the filter again
2024-01-16 17:50:14 <@nirik:matrix.scrye.com> ok, great. Hopefully they act on it soon.
2024-01-16 17:50:20 <@humaton:fedora.im> so they contacted me Yeasterday 
2024-01-16 17:50:34 <@nirik:matrix.scrye.com> ugh. email is... not a good way to tranfer this.
2024-01-16 17:50:39 <@humaton:fedora.im> nope
2024-01-16 17:50:57 <@humaton:fedora.im> but should I just send them link to random internet place?
2024-01-16 17:51:08 <@humaton:fedora.im> or upload it to github?
2024-01-16 17:51:31 <@nirik:matrix.scrye.com> we have to be very very very careful there... we cannot distribute this.
2024-01-16 17:51:36 <@humaton:fedora.im> ofc that random internet place can be my server but it feels weird
2024-01-16 17:51:42 <@nirik:matrix.scrye.com> so it needs to be some way to only go to them and no one else.
2024-01-16 17:51:53 <@humaton:fedora.im> right
2024-01-16 17:51:59 <@humaton:fedora.im> so email it is 
2024-01-16 17:52:16 <@nirik:matrix.scrye.com> I guess. anyhow, sounds like it's being worked on.
2024-01-16 17:52:25 <@nirik:matrix.scrye.com> might update the ticket with info so others know
2024-01-16 17:52:50 <@humaton:fedora.im> patrikp:  can you split the composes into 15M chunks so it can get through filters?
2024-01-16 17:54:04 <@nirik:matrix.scrye.com> !releng 11787
2024-01-16 17:54:05 <@zodbot:fedora.im> **releng #11787** (https://pagure.io/releng/issue/11787):**create EPEL 10 package signing key**

● **Opened:** a month ago by carlwgeorge
● **Last Updated:** a month ago
● **Assignee:** humaton
2024-01-16 17:54:14 <@nirik:matrix.scrye.com> this is done right? or was there something left here?
2024-01-16 17:54:34 <@humaton:fedora.im> I made the keys added people but didnt put it anywhere
2024-01-16 17:54:40 <@patrikp:matrix.org> The f40 ones? (Or I guess all)
2024-01-16 17:54:49 <@humaton:fedora.im> patrikp: all
2024-01-16 17:55:06 <@humaton:fedora.im> and Ideally if you can use tars param --tape-size
2024-01-16 17:55:23 <@humaton:fedora.im> not split
2024-01-16 17:55:50 <@nirik:matrix.scrye.com> jednorozec: ok. We can add it to epel-release, but... epel-release-10... which doesn't exist yet, so I would say we should just close that ticket for now?
2024-01-16 17:56:20 <@humaton:fedora.im> yup closing
2024-01-16 17:57:22 <@nirik:matrix.scrye.com> oh, I wanted to talk about unretire's... some people seem to file those tickets before the re-review and we currently keep them open and tell them to let us know when it's done. I'd like to propose we instead close them and tell them to reopen them or file a new one when it's done.
2024-01-16 17:57:34 <@nirik:matrix.scrye.com> but also I guess we are closer to automating...
2024-01-16 17:57:58 <@humaton:fedora.im> That is what I did previously
2024-01-16 17:58:25 <@humaton:fedora.im> but then people started commenting on closed tickets or oppening new ones
2024-01-16 17:58:54 <@humaton:fedora.im> so yeah lets close it with "get back later"
2024-01-16 17:59:05 <@nirik:matrix.scrye.com> was reminded of this when someone opened like 10 unretires at the same time they just filed the re-reviews. ;(
2024-01-16 17:59:36 <@jnsamyak:matrix.org> but that makes sense, they can always reopen it; I'll keep that in mind as well, lessen the burden on tracker page as well
2024-01-16 18:00:01 <@nirik:matrix.scrye.com> ah... remembered another thing.
2024-01-16 18:00:10 <@nirik:matrix.scrye.com> !releng 11761
2024-01-16 18:00:11 <@zodbot:fedora.im> **releng #11761** (https://pagure.io/releng/issue/11761):**missing artifacts from f39-rc-1.5 (final release)**

● **Opened:** 2 months ago by kevin
● **Last Updated:** a month ago
● **Assignee:** kevin
2024-01-16 18:00:27 <@jnsamyak:matrix.org> !info mail has been sent to devel-announce for the postponement of mass rebuild 
2024-01-16 18:01:14 <@nirik:matrix.scrye.com> This is done, but... the checksums in the torrents are not signed. ;( Can we sign those and remake those respin torrents?
2024-01-16 18:02:25 <@nirik:matrix.scrye.com> once thats done we cna close this.
2024-01-16 18:03:31 <@jnsamyak:matrix.org> ok I need to check this once, it was confusing because the torrent was created after we created the checksums for those respins
2024-01-16 18:03:32 <@nirik:matrix.scrye.com> oh and we are over time. ;(
2024-01-16 18:03:44 <@jnsamyak:matrix.org> do we need to so something more?
2024-01-16 18:03:47 <@nirik:matrix.scrye.com> yeah, somehow the signed one isn't the one in there.
2024-01-16 18:05:03 <@jnsamyak:matrix.org> weirddd, I can check but hopefully on thrusday/friday, or is it needed urgently nirik ?
2024-01-16 18:05:42 <@nirik:matrix.scrye.com> well, it's waited this long. ;) Please ping me if you need more info... we can look for sure
2024-01-16 18:06:18 <@jnsamyak:matrix.org> sure thing, need to check what happned there once, will reach ou thanksss :D
2024-01-16 18:06:28 <@jnsamyak:matrix.org> that's all from my end
2024-01-16 18:08:19 <@jnsamyak:matrix.org> hehe, I will be heading off to bed almost midnight, see ya all folks and thanks jednorozec for hosting us :D
2024-01-16 18:08:24 <@jnsamyak:matrix.org> jednorozec++
2024-01-16 18:08:35 <@nirik:matrix.scrye.com> night!
2024-01-16 18:08:38 <@humaton:fedora.im> !endmeeting