14:02:12 #startmeeting Fedora Flatpak Packaging SIG 14:02:12 Meeting started Mon Apr 17 14:02:12 2023 UTC. 14:02:12 This meeting is logged and archived in a public location. 14:02:12 The chair is kalev. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions. 14:02:12 Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:02:12 The meeting name has been set to 'fedora_flatpak_packaging_sig' 14:02:13 #meetingname flatpak-sig 14:02:13 The meeting name has been set to 'flatpak-sig' 14:02:13 #topic Init process 14:02:35 I think it's meeting time -- sorry, I am completely unprepared here 14:02:55 anyone around for the flatpak meeting? 14:03:11 .hello siosm 14:03:12 I'm around. 14:03:12 travier: siosm 'Timothée Ravier' 14:05:28 If https://calendar.fedoraproject.org/SIGs/ is correct, then it's meeting time 14:05:39 * kalev nods. 14:05:53 Item I think for today: https://pagure.io/releng/issue/11385 14:06:09 excellent, let's get started then 14:06:13 #topic Silverblue aarch64 installer image compose always fails on F38 and Rawhide 14:06:36 I'm still in a previous meeting, should be wrapping up any minute now 14:08:00 I think nirik thought that this is caused by for the flatpak installation code going through cdn, while it should go through the no-cdn url, https://registry-no-cdn.fedoraproject.org 14:08:23 it's a bit weird because it's only failing on aarch64 and not x86_64 14:08:56 I looked at pungi-fedora and the template files and didn't see anything obvious wrong 14:14:38 ok, other meeting is over so I'm fully here now 14:17:05 OwenTaylor[m]: any ideas what could be going wrong with the aarch64 flatpak install? is it possible that the system fedora remote is somehow leaking into the anaconda install and it's not using the custom no-cdn remote? 14:18:22 So, it broke about 2 weeks ago? That's pretty odd. 14:18:33 I looked more at this, and couldn't figure it out. ;( it's quite weird. We do know when it started... but nothing there that changed seems related. ;( 14:18:36 If it never worked, that would be more unerstandable. 14:20:19 Do we have comparisons of that part of the aarch64 vs. x86_64 compose logs? 14:20:51 I think it might coincide with the rolling out of F38 flatpak runtime, https://pagure.io/pungi-fedora/pull-request/1155 that should have been first in March 30 compose 14:21:14 should have logs in koji for everything... 14:21:26 but the 30th worked I think... day after was the first failure? 14:21:36 hah, bizarre 14:22:04 * nirik likely missed something, so definitely good to have someone else dig into it. 14:22:19 were there any proxy changes at the same time? 14:23:20 we did a bunch of updates/reboots on the 29th... 14:24:04 I don't actually see how that registry-no-cdn setting is doing anything. If you go to https://registry-no-cdn.fedoraproject.org/index/static?label:org.flatpak.ref:exists=1&os=linux&architecture=amd64&tag=latest - that has "Registry": "https://registry.fedoraproject.org/" which is what Flatpak will actually go to to download blobs 14:25:14 ohh! so the flatpak remote may be set up to point to no-cdn, but the individul blobs are still downloaded from the cdn? 14:25:44 So, I'd expect if there's a difference there, it's not that x86_64 and aarch64 are using different URLs, but rather that the cdn url is somehow working for the x86_64 builders 14:26:08 # builders shouldn't use the cdn for flatpak building. 14:26:09 RewriteCond expr "! -R '10.3.169.0/24'" 14:26:36 That's the actual mechanism where the builders escape the CDN. So, maybe the aarch64 builders stopped matching that netmask? 14:27:10 ha. They don't match that... 14:27:21 but thats still odd that they broke... 14:28:23 * nirik heads to another meeting. we can fix the cdn thing and see if it helps any for sure. 14:29:08 nirik: sounds good, thanks! I'll be around later to help if I can in any way, just let me know 14:29:20 `Yes, indeed, I can't think why they were working. Wild flailing: unless we weren't actually embedding flatpaks in the aarch64 images until the f38 cycle + f38 runtime? 14:29:58 .hello yselkowitz 14:29:58 hm, that was indeed the case, but tpopela[m] fixed it several months ago I think to embed aarch64 flatpaks 14:29:58 yselkowitz[m]: yselkowitz 'Yaakov Selkowitz' 14:30:09 hi Yaakov! 14:31:30 https://pagure.io/pungi-fedora/c/0351ce9f83c80040dea4cb89950e82056317697b was 4 months ago 14:33:31 maybe aarch64 builders somehow were stuck on an older pungi-fedora version and the updates that nirik did on the 29th somehow caused them pull the latest config, which made them start embedding aarch64 flatpaks again? seems a bit far fetched, but who knows :) 14:34:40 I guess w could look at the results of the last successful compose and see if it actually embeds flatpaks 14:37:04 I can do that and dig a bit and compare the logs and results, but let's see first if the rewritecond fixes help 14:37:35 https://kojipkgs.fedoraproject.org/compose/branched/Fedora-38-20230331.n.0/logs/aarch64/Silverblue/ostree_installer-5/lorax.log 14:37:50 Note that it has lorax-embed.tmpl but not lorax-embed-flatpaks.tmpl 14:39:09 ha, that really points to pungi-fedora not being up to date 14:39:11 Errr, scratch that, it does have lorax-embed-flatpaks 14:39:36 ah. OK, and it can't have been not up to date because it was installing f38 flatpaks 14:42:40 Yeah, there's no difference that I can see between the command the succeeded on 3/31 and the command that timed out on 4/01 14:44:05 * kalev concurs. 14:45:58 I think updating the RewriteCond, see if that helps, and if so, mark it down to "unspecified network changes" 14:46:54 yep, that's my thinking as well 14:47:50 anything else for today? I haven't looked at flatpak stuff at all after the previous meeting and don't have anything 14:48:11 yselkowitz[m]: oh, I see you've done a tons of F38 rebuilds -- awesome, thank you! 14:48:55 I was just trying to locate the ELN rebuild infrastructure to take a look at it and see how reusable it could be for a flatpaks buildroot, but haven't yet succeeded in finding it 14:49:02 kde 5&6 are done, a big batch of gnome and other apps pending: https://bodhi.fedoraproject.org/updates/FEDORA-FLATPAK-2023-6ffdc113bf 14:49:20 awesome! 14:51:01 a handful of apps are blocked by missing libappindicator, included as part of https://src.fedoraproject.org/modules/flatpak-common/pull-request/4 14:51:38 also a notable change to geany to include (most) geany-plugins: https://src.fedoraproject.org/flatpaks/geany/pull-request/1 14:52:19 I am having some issues building pdfarranger though 14:53:16 https://release-engineering.github.io/mbs-ui/module/16607 14:54:19 nice, adding geany plugins should make it much more usable 14:54:30 the problem seems to be that python-pikepdf uses %pyproject_buildrequires and depends on pillow, which the flatpak-common build of python-pillow does not satisfy 14:56:43 it could be that %pyproject_buildrequires is only looking at /usr and not generating provides for things that get installed in /app, or something like that 14:56:55 pretty sure it's not 14:57:42 OwenTaylor[m]: is https://gitlab.com/redhat/centos-stream/ci-cd/distrosync/distrobuildsync the sync infrastructure? 14:57:56 if pillow weren't in common then my usual workaround has been to build all such python packages simultaneously (so they all build against the platform packages), but with pillow in common it's already there 14:58:16 and dropping common here isn't feasible either 14:58:30 I'll try to file an issue on that when I can 14:59:03 but we're out of time, so let's continue in channel 14:59:05 * kalev nods. 14:59:14 thanks for coming, everybody! 14:59:15 now for another 2 minutes :p 14:59:18 *not 14:59:25 hehe 14:59:27 #endmeeting