16:00:11 <jednorozec> #startmeeting RELENG (2023-06-06)
16:00:11 <zodbot> Meeting started Tue Jun  6 16:00:11 2023 UTC.
16:00:11 <zodbot> This meeting is logged and archived in a public location.
16:00:11 <zodbot> The chair is jednorozec. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:11 <zodbot> The meeting name has been set to 'releng_(2023-06-06)'
16:00:11 <jednorozec> #meetingname releng
16:00:11 <jednorozec> #chair nirik sharkcz pbrobinson phsmoura dustymabe jednorozec
16:00:11 <zodbot> The meeting name has been set to 'releng'
16:00:11 <zodbot> Current chairs: dustymabe jednorozec nirik pbrobinson phsmoura sharkcz
16:01:21 <jnsamyak> hello folks!
16:01:21 <jnsamyak> .hi jnsamyak
16:01:22 <zodbot> jnsamyak: jnsamyak 'Samyak Jain' <samyak.jn11@gmail.com>
16:01:46 <patrikp[m]> Good evening. 👋
16:02:18 <nirik> morning
16:03:27 <jednorozec> sooo
16:03:52 * nirik had a few small topics.
16:04:22 <jednorozec> go for it nirik, I dont have much just few small things as-well
16:04:43 <nirik> ok
16:04:50 <nirik> #topic koji db
16:05:05 <nirik> so, I reinstalled/upgraded koji's db server last week...
16:05:33 <nirik> its however not entirely stable. ;( every once in a while it gets stuck cpus...
16:05:42 <nirik> but then recovers.
16:06:10 <jednorozec> magic of postgres
16:06:52 <jednorozec> is it reporting somewhere when it is happening?
16:08:19 <samyak> checks
16:08:24 <nirik99> sigh, stupid bridge
16:08:26 <samyak> nope
16:08:56 <nirik99> anyhow, on koji:
16:08:59 <nirik99> [Tue Jun  6 14:46:35 2023] watchdog: BUG: soft lockup - CPU#60 stuck for 23s! [postmaster:1431
16:09:05 <nirik99> that kind of thing. ;(
16:09:31 <jednorozec> oh
16:10:01 <nirik99> and it's not very idle, but then again it wasn't before... and in fact it might have been doing this before I just didn't notice. ;)
16:10:32 <nirik99> I might try and update the bios in the host, but that would need a short outage. ;(
16:10:50 <nirik99> Next thing I had was
16:10:52 <nirik99> #toolbx
16:11:01 <jednorozec> as for toolbox
16:11:02 <nirik99> they are ok with moving to a kickstart...
16:11:22 <jednorozec> yeah, so I think wthis might be a good work for guys jnsamyak and patrikp[m]
16:11:23 <nirik99> which would be much nicer for us. (and take the container pipeline out of it)
16:11:41 <jednorozec> we can go together through the kickstart and create a new one for toolbox
16:11:58 <nirik99> sure! I'd check with toolbx maintainers first... they might already be working on it. Or at least can provide feedback
16:12:20 <nirik99> should be a fun little project. :)
16:12:22 <jednorozec> yup
16:12:38 <samyak> nice!
16:12:59 <nirik99> I think I had one other thing, but can't recall what it was yet. ;) So...
16:13:06 <jednorozec> it been a while when I wrote some kickstarts
16:13:16 <jednorozec> but it didnt change much I presume
16:13:28 <jednorozec> so
16:13:31 <jednorozec> I have some
16:13:40 <jednorozec> and you can jum in when it comes back
16:13:53 <jednorozec> .releng 11422
16:13:54 <zodbot> jednorozec: Issue #11422: Please send openh264-2.3.1-1.el9 to Cisco - releng - Pagure.io - https://pagure.io/releng/issue/11422
16:13:59 <jednorozec> so epel and opnh264
16:14:04 <nirik99> ah yes.
16:14:11 <jednorozec> I started working on PR for epel-release
16:14:15 <nirik99> we need to add the repo to epel-release
16:14:17 <nirik99> cool.
16:14:22 <jednorozec> but I am not sure how the metalink URL should look
16:14:49 <jednorozec> and during that I realised that mm might need some changes to actually recognize epel9-openh264
16:15:17 <nirik99> yeah, check with adrian. Or I can.
16:15:37 <jednorozec> righ
16:15:51 <nirik99> I wanted to talk to him about moving mm to rhel9 anyhow.
16:15:56 <jednorozec> yeah
16:16:03 <jednorozec> so that was the other thing i noticed today
16:16:05 <jednorozec> its rhel7
16:16:25 <nirik99> yep.
16:17:40 <jednorozec> ok, so I will open the PR with placeholder for metalink
16:17:55 <nirik99> sounds good.
16:18:02 <jednorozec> and ping adrian about mm and new repos
16:18:14 <nirik99> I just pinged him in admin. ;)
16:18:21 <jednorozec> cool
16:18:26 <jednorozec> this one
16:18:28 <nirik99> but I think he's in de, so might not be around.
16:18:28 <jednorozec> .releng 11454
16:18:29 <zodbot> jednorozec: Issue #11454: Lenovo: F38 updated official respin iso for P1G6 - releng - Pagure.io - https://pagure.io/releng/issue/11454
16:18:45 <jednorozec> nirik99, yeah I think he is I will follow up there
16:18:57 <jednorozec> so
16:19:02 <jednorozec> the resping sig did a respin
16:19:12 <jednorozec> does this involve us?
16:19:14 <nirik99> yeah, so on this... I think if it passes testing we need to put it somewhere so it's saved ?
16:19:18 <jednorozec> can we close it?
16:19:27 <nirik99> I think we have to put it somewhere.
16:19:33 <jednorozec> I see
16:19:58 <jednorozec> but its in pub/alt
16:20:23 <nirik99> yeah, but respins sig doesn't save them... they make new ones and delete the old ones I think.
16:20:32 <jednorozec> oh
16:20:40 <jednorozec> so they have the latest one only
16:20:53 <nirik99> live-respins-archive
16:21:05 <nirik99> but I guess it's already there. ;)
16:21:13 <jednorozec> yeah they put it there
16:21:18 <nirik99> so yeah, if all is well with lenovo... then...
16:21:51 <jednorozec> I will ping on the ticket and eventually close
16:22:04 <nirik99> sounds good. +1
16:22:32 <jednorozec> and
16:22:34 <jednorozec> its dns
16:22:37 <jednorozec> .releng 11439
16:22:38 <zodbot> jednorozec: Issue #11439: rpm-ostree installer image builds fail since Fedora-Rawhide-20230524.n.0 - releng - Pagure.io - https://pagure.io/releng/issue/11439
16:22:42 <jednorozec> but where?
16:23:34 <jednorozec> I was looking into changes we applied on the infra around the date it started failing
16:23:54 <nirik99> ok, thats a good idea... find anything?
16:23:56 <samyak> patrikp[m] just for the reference this was the issue I was talking to you about in the 1:1^ (sorry to hijack the thread) :P
16:24:00 <jednorozec> I found one redirect
16:24:03 <jednorozec> that was changed
16:24:04 <jednorozec> https://pagure.io/fedora-infra/ansible/c/b0b0acd4e53d6e4bfd2f0d91c531ea1b5cb6a26c?branch=main
16:24:31 <jednorozec> but it does not seem to be related just one thing cought mi eye
16:24:33 <nirik99> yeah, that shouldn't affect this. Thats the website.
16:24:37 <nirik99> yeah
16:25:49 <jednorozec> the only thing that was interesting was this
16:25:49 <jednorozec> https://pagure.io/fedora-infra/ansible/blob/b0b0acd4e53d6e4bfd2f0d91c531ea1b5cb6a26c/f/playbooks/include/proxies-redirects.yml#_844
16:26:03 <jednorozec> it changes from redirect to redirectmatch and regexp
16:26:41 <jednorozec> while reading this I realized that my knowledge about our proxies is very limited
16:26:54 * jednorozec checks on matrix
16:27:36 <nirik99> yeah, thats the docs/end user website... this one is the compose/actual content site.
16:27:52 <nirik99> I am not sure I understand the flow on this site tho.
16:28:03 <nirik99> ( ostree.fedoraproject.org that is)
16:28:24 <jednorozec> huh 443
16:28:41 <nirik99> it redorects some things, but I don't see how it gets the actual content. It must proxy it somewhere.
16:29:27 <nirik99> in the past when we have had weird problems its often been related to http/2... we could try disabling that and see if it changes anything.
16:29:49 <jednorozec> hmm
16:30:16 <jednorozec> disabling it where?
16:31:21 <nirik99> in the proxies... for just that one site.
16:31:30 <nirik99> but it's weird... because, it seems random.
16:31:43 <nirik99> which one and what arch fail varies day to day
16:31:51 <jednorozec> yes
16:32:00 <jednorozec> that makes it more interesting
16:32:36 <nirik99> so, I guess the site here is actually kojipkgs...
16:32:38 <nirik99> not clear
16:33:25 <nirik99> here's a failure with todays kinote: https://kojipkgs.fedoraproject.org//work/tasks/820/101860820/runroot.log
16:33:40 <nirik99> the failure happens after all the downloads/installs of rpms... in the post when it runs a ostree command.
16:34:16 <jednorozec> that pulls refs from mirror
16:34:40 <nirik99> I wish it had more info on what exact url gave that error. ;)
16:34:48 <jednorozec> yeah
16:36:03 <nirik99> well, we could try disabling h2 on kojipkgs, but I kinda hate to as h2 is more efficent and they get a ton of hits.
16:36:27 <nirik99> but I am also not even sure it's kojipkgs there.
16:37:17 <nirik99> I asked in the ticket if anyone knows how to find more out.
16:37:34 <jednorozec> thanks
16:38:55 <samyak> I'm just trying to understand the error, from what I can see the specific command that failed is:
16:38:55 <samyak> something like this
16:38:55 <samyak> ostree --repo=/var/tmp/lorax/lorax.0_fntr7r/installtree/ostree/repo pull --mirror fedora fedora/rawhide/aarch64/kinoite
16:38:55 <samyak> t also shows that the script attempted to clean up a temporary directory before exiting with an exit code of 1, indicating a failure.  And 502  error typically signifies a bad gateway or a temporary issue with the server that prevented it from fulfilling the request. So what should be the first point of debugging checking the server status etc?
16:40:48 <nirik99> yeah, so it's trying to pull down and mirror a copy of the kinoite repo and thats failing.
16:41:06 <nirik99> but Its unclear to me at least what host/remote it's using there
16:41:24 <nirik99> it could be kojipkgs.fedoraproject.org or ostree.fedoraproject.org or something else.
16:41:34 <samyak> ah got it, another dumb queston that I'll fire is
16:41:56 <nirik99> no no, not dumb at all. ;)
16:42:01 <samyak> could we try out these commands mannually on the machine?
16:42:26 <nirik99> sadly it's running in a vm that was created for that compose/build... so it's long since nuked. ;(
16:42:38 <nirik99> we could possibly re-run it manually as a scratch build and get into it...
16:42:57 <samyak> yep the latter answers my question will explore more on that bit
16:43:06 <samyak> thanks! :D
16:44:13 <jednorozec> so
16:44:15 <jednorozec> one more
16:44:19 <jednorozec> .releng 11451
16:44:20 <zodbot> jednorozec: Issue #11451: Side tag for Python 3.12 - releng - Pagure.io - https://pagure.io/releng/issue/11451
16:44:40 <nirik99> this is done?
16:44:43 <jednorozec> I am jsut not sure about the signing pr https://pagure.io/fedora-infra/ansible/pull-request/1477
16:44:54 <jednorozec> nirik99, can you have a look?
16:45:01 <jednorozec> if the PR is merged the ticket is done
16:45:42 <nirik99> ah, yes. I can do that...
16:47:26 <nirik99> ok, one minor issue...
16:48:13 <jednorozec> oh ima is missing
16:50:56 <jednorozec> fixed
16:51:05 <jednorozec> #topic Open Floor
16:53:35 <jednorozec> If you have nothing to discuss I will close this meeting in few minutes
16:53:36 <nirik> merged. I can push it in a min
16:53:48 <samyak> thanks for the meeting folks, i have nothing to add, will go to take my beauty sleep :P see you all in the next 0/
16:54:49 <nirik> have a good night samyak!
16:55:43 <jednorozec> #endmeeting