16:00:11 <jednorozec> #startmeeting RELENG (2023-06-06) 16:00:11 <zodbot> Meeting started Tue Jun 6 16:00:11 2023 UTC. 16:00:11 <zodbot> This meeting is logged and archived in a public location. 16:00:11 <zodbot> The chair is jednorozec. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 16:00:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00:11 <zodbot> The meeting name has been set to 'releng_(2023-06-06)' 16:00:11 <jednorozec> #meetingname releng 16:00:11 <jednorozec> #chair nirik sharkcz pbrobinson phsmoura dustymabe jednorozec 16:00:11 <zodbot> The meeting name has been set to 'releng' 16:00:11 <zodbot> Current chairs: dustymabe jednorozec nirik pbrobinson phsmoura sharkcz 16:01:21 <jnsamyak> hello folks! 16:01:21 <jnsamyak> .hi jnsamyak 16:01:22 <zodbot> jnsamyak: jnsamyak 'Samyak Jain' <samyak.jn11@gmail.com> 16:01:46 <patrikp[m]> Good evening. 👋 16:02:18 <nirik> morning 16:03:27 <jednorozec> sooo 16:03:52 * nirik had a few small topics. 16:04:22 <jednorozec> go for it nirik, I dont have much just few small things as-well 16:04:43 <nirik> ok 16:04:50 <nirik> #topic koji db 16:05:05 <nirik> so, I reinstalled/upgraded koji's db server last week... 16:05:33 <nirik> its however not entirely stable. ;( every once in a while it gets stuck cpus... 16:05:42 <nirik> but then recovers. 16:06:10 <jednorozec> magic of postgres 16:06:52 <jednorozec> is it reporting somewhere when it is happening? 16:08:19 <samyak> checks 16:08:24 <nirik99> sigh, stupid bridge 16:08:26 <samyak> nope 16:08:56 <nirik99> anyhow, on koji: 16:08:59 <nirik99> [Tue Jun 6 14:46:35 2023] watchdog: BUG: soft lockup - CPU#60 stuck for 23s! [postmaster:1431 16:09:05 <nirik99> that kind of thing. ;( 16:09:31 <jednorozec> oh 16:10:01 <nirik99> and it's not very idle, but then again it wasn't before... and in fact it might have been doing this before I just didn't notice. ;) 16:10:32 <nirik99> I might try and update the bios in the host, but that would need a short outage. ;( 16:10:50 <nirik99> Next thing I had was 16:10:52 <nirik99> #toolbx 16:11:01 <jednorozec> as for toolbox 16:11:02 <nirik99> they are ok with moving to a kickstart... 16:11:22 <jednorozec> yeah, so I think wthis might be a good work for guys jnsamyak and patrikp[m] 16:11:23 <nirik99> which would be much nicer for us. (and take the container pipeline out of it) 16:11:41 <jednorozec> we can go together through the kickstart and create a new one for toolbox 16:11:58 <nirik99> sure! I'd check with toolbx maintainers first... they might already be working on it. Or at least can provide feedback 16:12:20 <nirik99> should be a fun little project. :) 16:12:22 <jednorozec> yup 16:12:38 <samyak> nice! 16:12:59 <nirik99> I think I had one other thing, but can't recall what it was yet. ;) So... 16:13:06 <jednorozec> it been a while when I wrote some kickstarts 16:13:16 <jednorozec> but it didnt change much I presume 16:13:28 <jednorozec> so 16:13:31 <jednorozec> I have some 16:13:40 <jednorozec> and you can jum in when it comes back 16:13:53 <jednorozec> .releng 11422 16:13:54 <zodbot> jednorozec: Issue #11422: Please send openh264-2.3.1-1.el9 to Cisco - releng - Pagure.io - https://pagure.io/releng/issue/11422 16:13:59 <jednorozec> so epel and opnh264 16:14:04 <nirik99> ah yes. 16:14:11 <jednorozec> I started working on PR for epel-release 16:14:15 <nirik99> we need to add the repo to epel-release 16:14:17 <nirik99> cool. 16:14:22 <jednorozec> but I am not sure how the metalink URL should look 16:14:49 <jednorozec> and during that I realised that mm might need some changes to actually recognize epel9-openh264 16:15:17 <nirik99> yeah, check with adrian. Or I can. 16:15:37 <jednorozec> righ 16:15:51 <nirik99> I wanted to talk to him about moving mm to rhel9 anyhow. 16:15:56 <jednorozec> yeah 16:16:03 <jednorozec> so that was the other thing i noticed today 16:16:05 <jednorozec> its rhel7 16:16:25 <nirik99> yep. 16:17:40 <jednorozec> ok, so I will open the PR with placeholder for metalink 16:17:55 <nirik99> sounds good. 16:18:02 <jednorozec> and ping adrian about mm and new repos 16:18:14 <nirik99> I just pinged him in admin. ;) 16:18:21 <jednorozec> cool 16:18:26 <jednorozec> this one 16:18:28 <nirik99> but I think he's in de, so might not be around. 16:18:28 <jednorozec> .releng 11454 16:18:29 <zodbot> jednorozec: Issue #11454: Lenovo: F38 updated official respin iso for P1G6 - releng - Pagure.io - https://pagure.io/releng/issue/11454 16:18:45 <jednorozec> nirik99, yeah I think he is I will follow up there 16:18:57 <jednorozec> so 16:19:02 <jednorozec> the resping sig did a respin 16:19:12 <jednorozec> does this involve us? 16:19:14 <nirik99> yeah, so on this... I think if it passes testing we need to put it somewhere so it's saved ? 16:19:18 <jednorozec> can we close it? 16:19:27 <nirik99> I think we have to put it somewhere. 16:19:33 <jednorozec> I see 16:19:58 <jednorozec> but its in pub/alt 16:20:23 <nirik99> yeah, but respins sig doesn't save them... they make new ones and delete the old ones I think. 16:20:32 <jednorozec> oh 16:20:40 <jednorozec> so they have the latest one only 16:20:53 <nirik99> live-respins-archive 16:21:05 <nirik99> but I guess it's already there. ;) 16:21:13 <jednorozec> yeah they put it there 16:21:18 <nirik99> so yeah, if all is well with lenovo... then... 16:21:51 <jednorozec> I will ping on the ticket and eventually close 16:22:04 <nirik99> sounds good. +1 16:22:32 <jednorozec> and 16:22:34 <jednorozec> its dns 16:22:37 <jednorozec> .releng 11439 16:22:38 <zodbot> jednorozec: Issue #11439: rpm-ostree installer image builds fail since Fedora-Rawhide-20230524.n.0 - releng - Pagure.io - https://pagure.io/releng/issue/11439 16:22:42 <jednorozec> but where? 16:23:34 <jednorozec> I was looking into changes we applied on the infra around the date it started failing 16:23:54 <nirik99> ok, thats a good idea... find anything? 16:23:56 <samyak> patrikp[m] just for the reference this was the issue I was talking to you about in the 1:1^ (sorry to hijack the thread) :P 16:24:00 <jednorozec> I found one redirect 16:24:03 <jednorozec> that was changed 16:24:04 <jednorozec> https://pagure.io/fedora-infra/ansible/c/b0b0acd4e53d6e4bfd2f0d91c531ea1b5cb6a26c?branch=main 16:24:31 <jednorozec> but it does not seem to be related just one thing cought mi eye 16:24:33 <nirik99> yeah, that shouldn't affect this. Thats the website. 16:24:37 <nirik99> yeah 16:25:49 <jednorozec> the only thing that was interesting was this 16:25:49 <jednorozec> https://pagure.io/fedora-infra/ansible/blob/b0b0acd4e53d6e4bfd2f0d91c531ea1b5cb6a26c/f/playbooks/include/proxies-redirects.yml#_844 16:26:03 <jednorozec> it changes from redirect to redirectmatch and regexp 16:26:41 <jednorozec> while reading this I realized that my knowledge about our proxies is very limited 16:26:54 * jednorozec checks on matrix 16:27:36 <nirik99> yeah, thats the docs/end user website... this one is the compose/actual content site. 16:27:52 <nirik99> I am not sure I understand the flow on this site tho. 16:28:03 <nirik99> ( ostree.fedoraproject.org that is) 16:28:24 <jednorozec> huh 443 16:28:41 <nirik99> it redorects some things, but I don't see how it gets the actual content. It must proxy it somewhere. 16:29:27 <nirik99> in the past when we have had weird problems its often been related to http/2... we could try disabling that and see if it changes anything. 16:29:49 <jednorozec> hmm 16:30:16 <jednorozec> disabling it where? 16:31:21 <nirik99> in the proxies... for just that one site. 16:31:30 <nirik99> but it's weird... because, it seems random. 16:31:43 <nirik99> which one and what arch fail varies day to day 16:31:51 <jednorozec> yes 16:32:00 <jednorozec> that makes it more interesting 16:32:36 <nirik99> so, I guess the site here is actually kojipkgs... 16:32:38 <nirik99> not clear 16:33:25 <nirik99> here's a failure with todays kinote: https://kojipkgs.fedoraproject.org//work/tasks/820/101860820/runroot.log 16:33:40 <nirik99> the failure happens after all the downloads/installs of rpms... in the post when it runs a ostree command. 16:34:16 <jednorozec> that pulls refs from mirror 16:34:40 <nirik99> I wish it had more info on what exact url gave that error. ;) 16:34:48 <jednorozec> yeah 16:36:03 <nirik99> well, we could try disabling h2 on kojipkgs, but I kinda hate to as h2 is more efficent and they get a ton of hits. 16:36:27 <nirik99> but I am also not even sure it's kojipkgs there. 16:37:17 <nirik99> I asked in the ticket if anyone knows how to find more out. 16:37:34 <jednorozec> thanks 16:38:55 <samyak> I'm just trying to understand the error, from what I can see the specific command that failed is: 16:38:55 <samyak> something like this 16:38:55 <samyak> ostree --repo=/var/tmp/lorax/lorax.0_fntr7r/installtree/ostree/repo pull --mirror fedora fedora/rawhide/aarch64/kinoite 16:38:55 <samyak> t also shows that the script attempted to clean up a temporary directory before exiting with an exit code of 1, indicating a failure. And 502 error typically signifies a bad gateway or a temporary issue with the server that prevented it from fulfilling the request. So what should be the first point of debugging checking the server status etc? 16:40:48 <nirik99> yeah, so it's trying to pull down and mirror a copy of the kinoite repo and thats failing. 16:41:06 <nirik99> but Its unclear to me at least what host/remote it's using there 16:41:24 <nirik99> it could be kojipkgs.fedoraproject.org or ostree.fedoraproject.org or something else. 16:41:34 <samyak> ah got it, another dumb queston that I'll fire is 16:41:56 <nirik99> no no, not dumb at all. ;) 16:42:01 <samyak> could we try out these commands mannually on the machine? 16:42:26 <nirik99> sadly it's running in a vm that was created for that compose/build... so it's long since nuked. ;( 16:42:38 <nirik99> we could possibly re-run it manually as a scratch build and get into it... 16:42:57 <samyak> yep the latter answers my question will explore more on that bit 16:43:06 <samyak> thanks! :D 16:44:13 <jednorozec> so 16:44:15 <jednorozec> one more 16:44:19 <jednorozec> .releng 11451 16:44:20 <zodbot> jednorozec: Issue #11451: Side tag for Python 3.12 - releng - Pagure.io - https://pagure.io/releng/issue/11451 16:44:40 <nirik99> this is done? 16:44:43 <jednorozec> I am jsut not sure about the signing pr https://pagure.io/fedora-infra/ansible/pull-request/1477 16:44:54 <jednorozec> nirik99, can you have a look? 16:45:01 <jednorozec> if the PR is merged the ticket is done 16:45:42 <nirik99> ah, yes. I can do that... 16:47:26 <nirik99> ok, one minor issue... 16:48:13 <jednorozec> oh ima is missing 16:50:56 <jednorozec> fixed 16:51:05 <jednorozec> #topic Open Floor 16:53:35 <jednorozec> If you have nothing to discuss I will close this meeting in few minutes 16:53:36 <nirik> merged. I can push it in a min 16:53:48 <samyak> thanks for the meeting folks, i have nothing to add, will go to take my beauty sleep :P see you all in the next 0/ 16:54:49 <nirik> have a good night samyak! 16:55:43 <jednorozec> #endmeeting