14:02:12 <kalev> #startmeeting Fedora Flatpak Packaging SIG
14:02:12 <zodbot> Meeting started Mon Apr 17 14:02:12 2023 UTC.
14:02:12 <zodbot> This meeting is logged and archived in a public location.
14:02:12 <zodbot> The chair is kalev. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
14:02:12 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:02:12 <zodbot> The meeting name has been set to 'fedora_flatpak_packaging_sig'
14:02:13 <kalev> #meetingname flatpak-sig
14:02:13 <zodbot> The meeting name has been set to 'flatpak-sig'
14:02:13 <kalev> #topic Init process
14:02:35 <kalev> I think it's meeting time -- sorry, I am completely unprepared here
14:02:55 <kalev> anyone around for the flatpak meeting?
14:03:11 <travier> .hello siosm
14:03:12 <OwenTaylor[m]> I'm around.
14:03:12 <zodbot> travier: siosm 'Timothée Ravier' <travier@redhat.com>
14:05:28 <travier> If https://calendar.fedoraproject.org/SIGs/ is correct, then it's meeting time
14:05:39 * kalev nods.
14:05:53 <travier> Item I think for today: https://pagure.io/releng/issue/11385
14:06:09 <kalev> excellent, let's get started then
14:06:13 <kalev> #topic Silverblue aarch64 installer image compose always fails on F38 and Rawhide
14:06:36 <kalev> I'm still in a previous meeting, should be wrapping up any minute now
14:08:00 <kalev> I think nirik thought that this is caused by for the flatpak installation code going through cdn, while it should go through the no-cdn url, https://registry-no-cdn.fedoraproject.org
14:08:23 <kalev> it's a bit weird because it's only failing on aarch64 and not x86_64
14:08:56 <kalev> I looked at pungi-fedora and the template files and didn't see anything obvious wrong
14:14:38 <kalev> ok, other meeting is over so I'm fully here now
14:17:05 <kalev> OwenTaylor[m]: any ideas what could be going wrong with the aarch64 flatpak install? is it possible that the system fedora remote is somehow leaking into the anaconda install and it's not using the custom no-cdn remote?
14:18:22 <OwenTaylor[m]> So, it broke about 2 weeks ago? That's pretty odd.
14:18:33 <nirik> I looked more at this, and couldn't figure it out. ;( it's quite weird. We do know when it started... but nothing there that changed seems related. ;(
14:18:36 <OwenTaylor[m]> If it never worked, that would be more unerstandable.
14:20:19 <OwenTaylor[m]> Do we have comparisons of that part of the aarch64 vs. x86_64 compose logs?
14:20:51 <kalev> I think it might coincide with the rolling out of F38 flatpak runtime, https://pagure.io/pungi-fedora/pull-request/1155 that should have been first in March 30 compose
14:21:14 <nirik> should have logs in koji for everything...
14:21:26 <nirik> but the 30th worked I think... day after was the first failure?
14:21:36 <kalev> hah, bizarre
14:22:04 * nirik likely missed something, so definitely good to have someone else dig into it.
14:22:19 <kalev> were there any proxy changes at the same time?
14:23:20 <nirik> we did a bunch of updates/reboots on the 29th...
14:24:04 <OwenTaylor[m]> I don't actually see how that registry-no-cdn setting is doing anything. If you go to https://registry-no-cdn.fedoraproject.org/index/static?label:org.flatpak.ref:exists=1&os=linux&architecture=amd64&tag=latest - that has "Registry": "https://registry.fedoraproject.org/" which is what Flatpak will actually go to to download blobs
14:25:14 <kalev> ohh! so the flatpak remote may be set up to point to no-cdn, but the individul blobs are still downloaded from the cdn?
14:25:44 <OwenTaylor[m]> So, I'd expect if there's a difference there, it's not that x86_64 and aarch64 are using different URLs, but rather that the cdn url is somehow working for the x86_64 builders
14:26:08 <OwenTaylor[m]> # builders shouldn't use the cdn for flatpak building.
14:26:09 <OwenTaylor[m]> RewriteCond expr "! -R '10.3.169.0/24'"
14:26:36 <OwenTaylor[m]> That's the actual mechanism where the builders escape the CDN. So, maybe the aarch64 builders stopped matching that netmask?
14:27:10 <nirik> ha. They don't match that...
14:27:21 <nirik> but thats still odd that they broke...
14:28:23 * nirik heads to another meeting. we can fix the cdn thing and see if it helps any for sure.
14:29:08 <kalev> nirik: sounds good, thanks! I'll be around later to help if I can in any way, just let me know
14:29:20 <OwenTaylor[m]> `Yes, indeed, I can't think why they were working. Wild flailing: unless we weren't actually embedding flatpaks in the aarch64 images until the f38 cycle + f38 runtime?
14:29:58 <yselkowitz[m]> .hello yselkowitz
14:29:58 <kalev> hm, that was indeed the case, but tpopela[m] fixed it several months ago I think to embed aarch64 flatpaks
14:29:58 <zodbot> yselkowitz[m]: yselkowitz 'Yaakov Selkowitz' <yselkowi@redhat.com>
14:30:09 <kalev> hi Yaakov!
14:31:30 <kalev> https://pagure.io/pungi-fedora/c/0351ce9f83c80040dea4cb89950e82056317697b was 4 months ago
14:33:31 <kalev> maybe aarch64 builders somehow were stuck on an older pungi-fedora version and the updates that nirik did on the 29th somehow caused them pull the latest config, which made them start embedding aarch64 flatpaks again? seems a bit far fetched, but who knows :)
14:34:40 <OwenTaylor[m]> I guess w could look at the results of the last successful compose and see if it actually embeds flatpaks
14:37:04 <kalev> I can do that and dig a bit and compare the logs and results, but let's see first if the rewritecond fixes help
14:37:35 <OwenTaylor[m]> https://kojipkgs.fedoraproject.org/compose/branched/Fedora-38-20230331.n.0/logs/aarch64/Silverblue/ostree_installer-5/lorax.log
14:37:50 <OwenTaylor[m]> Note that it has lorax-embed.tmpl but not lorax-embed-flatpaks.tmpl
14:39:09 <kalev> ha, that really points to pungi-fedora not being up to date
14:39:11 <OwenTaylor[m]> Errr, scratch that, it does have lorax-embed-flatpaks
14:39:36 <kalev> ah. OK, and it can't have been not up to date because it was installing f38 flatpaks
14:42:40 <OwenTaylor[m]> Yeah, there's no difference that I can see between the command the succeeded on 3/31 and the command that timed out on 4/01
14:44:05 * kalev concurs.
14:45:58 <OwenTaylor[m]> I think updating the RewriteCond, see if that helps, and if so, mark it down to "unspecified network changes"
14:46:54 <kalev> yep, that's my thinking as well
14:47:50 <kalev> anything else for today? I haven't looked at flatpak stuff at all after the previous meeting and don't have anything
14:48:11 <kalev> yselkowitz[m]: oh, I see you've done a tons of F38 rebuilds -- awesome, thank you!
14:48:55 <OwenTaylor[m]> I was just trying to locate the ELN rebuild infrastructure to take a look at it and see how reusable it could be for a flatpaks buildroot, but haven't yet succeeded in finding it
14:49:02 <yselkowitz[m]> kde 5&6 are done, a big batch of gnome and other apps pending: https://bodhi.fedoraproject.org/updates/FEDORA-FLATPAK-2023-6ffdc113bf
14:49:20 <kalev> awesome!
14:51:01 <yselkowitz[m]> a handful of apps are blocked by missing libappindicator, included as part of https://src.fedoraproject.org/modules/flatpak-common/pull-request/4
14:51:38 <yselkowitz[m]> also a notable change to geany to include (most) geany-plugins: https://src.fedoraproject.org/flatpaks/geany/pull-request/1
14:52:19 <yselkowitz[m]> I am having some issues building pdfarranger though
14:53:16 <yselkowitz[m]> https://release-engineering.github.io/mbs-ui/module/16607
14:54:19 <kalev> nice, adding geany plugins should make it much more usable
14:54:30 <yselkowitz[m]> the problem seems to be that python-pikepdf uses %pyproject_buildrequires and depends on pillow, which the flatpak-common build of python-pillow does not satisfy
14:56:43 <kalev> it could be that %pyproject_buildrequires is only looking at /usr and not generating provides for things that get installed in /app, or something like that
14:56:55 <yselkowitz[m]> pretty sure it's not
14:57:42 <kalev> OwenTaylor[m]: is https://gitlab.com/redhat/centos-stream/ci-cd/distrosync/distrobuildsync the sync infrastructure?
14:57:56 <yselkowitz[m]> if pillow weren't in common then my usual workaround has been to build all such python packages simultaneously (so they all build against the platform packages), but with pillow in common it's already there
14:58:16 <yselkowitz[m]> and dropping common here isn't feasible either
14:58:30 <yselkowitz[m]> I'll try to file an issue on that when I can
14:59:03 <yselkowitz[m]> but we're out of time, so let's continue in channel
14:59:05 * kalev nods.
14:59:14 <kalev> thanks for coming, everybody!
14:59:15 <luna_> now for another 2 minutes :p
14:59:18 <luna_> *not
14:59:25 <kalev> hehe
14:59:27 <kalev> #endmeeting