#meeting-2:fedoraproject.org log

<@coremodule:fedora.im>

16:00:28

!startmeeting Fedora ARM Working Group Meeting

<@meetbot:fedora.im>

16:00:29

Meeting started at 2024-02-20 16:00:28 UTC

<@meetbot:fedora.im>

16:00:29

The Meeting name is 'Fedora ARM Working Group Meeting'

<@coremodule:fedora.im>

16:00:45

!topic roll call

<@coremodule:fedora.im>

16:00:52

Hello all, who is around this morning?

<@jforbes:fedora.im>

16:00:56

morning

<@pboy:fedora.im>

16:01:01

!hi

<@zodbot:fedora.im>

16:01:02

Peter Boy (pboy)

<@pwhalen:fedora.im>

16:01:03

Good morning!

<@coremodule:fedora.im>

16:02:59

!topic 1) ==== Stable Release ====

<@coremodule:fedora.im>

16:03:07

!info Fedora-39-20240220.0

<@coremodule:fedora.im>

16:03:14

!link https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=39&build=Fedora-Cloud-39-20240220.0&groupid=5

<@coremodule:fedora.im>

16:03:21

Anything for F39?

<@pboy:fedora.im>

16:04:08

We should fix arm-image-installer here, too. I think

<@pboy:fedora.im>

16:04:53

(and we have to fix it for f40 and rawhide)(

<@coremodule:fedora.im>

16:05:56

Peter Boy: can you elaborate here?

<@pboy:fedora.im>

16:07:07

Yeah. LVM group introduced with F38/39 a new feature that causes arm-image-installer to fail on a Server host. We had discussed this e.g. in bug https://bugzilla.redhat.com/show_bug.cgi?id=2246871

<@pboy:fedora.im>

16:07:28

We did know at that time about the "new feature".

<@pboy:fedora.im>

16:07:33

did N OT

<@pboy:fedora.im>

16:08:36

I summerized the issue here https://bugzilla.redhat.com/show_bug.cgi?id=2247872#c10

<@coremodule:fedora.im>

16:10:16

pwhalen: I wonder if arm-image-installer could be modified to add this?

<@coremodule:fedora.im>

16:10:19

!link https://bugzilla.redhat.com/show_bug.cgi?id=2247872#c10

<@pwhalen:fedora.im>

16:10:37

I can try to work around it, its not really a bug in the script though

<@pboy:fedora.im>

16:11:32

@pwhalen well it's probably not a bug, but a requirement to adjust to the new feature of LVM, I think

<@pwhalen:fedora.im>

16:11:44

the file shouldnt be there on the disk images, and is wrong (uses vda rather than whatever device they're on)

<@pwhalen:fedora.im>

16:12:06

but its still wrong for us. Even without uses the script, you can use lvm commands without editing

<@pwhalen:fedora.im>

16:12:14

*can't*

<@pwhalen:fedora.im>

16:12:37

but its still wrong for us. Even without uses the script, you can't use lvm commands without editing

<@pboy:fedora.im>

16:12:41

@pwhalen that file is a new feature which is important for LVM. So we can't delete it.

<@pwhalen:fedora.im>

16:13:12

I understand, but the details included are not correct. It says we use VDA. Anything not virt will not be using vda

<@pboy:fedora.im>

16:14:20

@pwhalen yes, that is the other part. we have to remove that file from the image and generate a new file during first boot which is adapted to the target system.

<@pwhalen:fedora.im>

16:15:18

Right, but this is outside the scope of the script. Its a bug, outside of the script.

<@pwhalen:fedora.im>

16:15:49

I can try to fix it in the script, but its a completely seperate from the script, imo

<@pboy:fedora.im>

16:16:23

Yes, we must do 2 steps. 1. adjust the script, 2. adjust the image.

<@pwhalen:fedora.im>

16:17:37

Is this still an issue in F40?

<@pboy:fedora.im>

16:18:17

pwhalen I tried to fix the script. But I didn't understand the logik and got always a syntax error, I think, you can do that in 4 min, I would need some hours :-)

<@pwhalen:fedora.im>

16:18:24

If this is still an issue in F40, we should fix it in the image creation.

<@pboy:fedora.im>

16:18:47

yes, it is still an issue in F40 and we have to fix the image.

<@pwhalen:fedora.im>

16:18:59

Not sure we can edit the file to fix it though, as we dont know where the install will be

<@pwhalen:fedora.im>

16:19:11

is sda, mmc etc

<@pboy:fedora.im>

16:19:19

An I have to fix the generic VM image as well, but I don't know how to configure first boot.

<@pwhalen:fedora.im>

16:19:30

so, ahve we looked at fixing it in the kickstart?

<@pboy:fedora.im>

16:20:20

Yes, we have to look into kickstart to fix the image. But that alone is not enough. We also have to adjust the script.

<@pboy:fedora.im>

16:21:03

The device file is now a feature to ensure a correct working of LVM

<@pwhalen:fedora.im>

16:21:25

I understand for f39, the ship has saild. But adding hacks to the script is'nt the best solution imo

<@pwhalen:fedora.im>

16:22:48

I understand for f39, the ship has sailed. But adding hacks to the script is'nt the best solution imo

<@pboy:fedora.im>

16:23:00

I dont know if 'hack' is a good description. According to David Teigland we have to add a --devices /dev/$medium to each vg* command.

<@pboy:fedora.im>

16:23:28

That way we restore the previous behavior of the command.

<@pwhalen:fedora.im>

16:23:39

..and if someone uses dd to write the image?

<@pboy:fedora.im>

16:24:58

If we remove the device file from the image and create an adjusted one at first boot, anything OK. But arm-image-installer wouldn't work without the fixes proposed by David.

<@pwhalen:fedora.im>

16:25:42

I can look at it, but this is a bug in the image. The device listed in the file isn't correct. When I looked before f39 GA, I think I just deleted the file and the lvm commands worked ok.

<@pwhalen:fedora.im>

16:26:45

the script should never be a blocker for Fedora.

<@pboy:fedora.im>

16:27:06

The new feature is that LVM by default only uses devices as specified in the device file. Therefore we MUST create an adjusted file. And we have to adjust arm-image-installer to be able to access the new device for mounting the image file.

<@coremodule:fedora.im>

16:28:07

I agree with this. The script is just a "quality of life" enhancement for ARM/IoT. dd works too.

<@pwhalen:fedora.im>

16:28:13

I think other solutions should be explored. This presents many issues

<@pboy:fedora.im>

16:29:04

@coremodule No the adjustment of the sciript is not a nice to have. It is a requirement to install Server on ARM at all!

<@pwhalen:fedora.im>

16:29:19

btu Peter Boythis is a regression then

<@pwhalen:fedora.im>

16:29:27

the script was never needed.

<@pboy:fedora.im>

16:30:04

Hm, without the script I can't produce a bootable file system. Therefore it is required.

<@coremodule:fedora.im>

16:30:10

I don't think that's correct... You can't dd the image to the media you want to boot from?

<@pwhalen:fedora.im>

16:30:19

And that is a regression as of F39.

<@coremodule:fedora.im>

16:30:26

Peter Boy: what hardware are you using?

<@pboy:fedora.im>

16:30:42

I can dd for Rasberry, for all other hardware I have to use the script.

<@pwhalen:fedora.im>

16:30:56

no, you can use dd commands

<@jlinton:fedora.im>

16:31:09

I thought the fedora/server requirement was strictly the install isos?

<@pboy:fedora.im>

16:31:29

I test on several hardware, currently Raspberry Pi 4, Radxa Pi4 and Pine64Pro

<@pwhalen:fedora.im>

16:31:40

The script just makes it easier to write out uboot, copy the ssh key. You can do it all manually

<@pboy:fedora.im>

16:32:17

That's right. But do we want the admins to do that?

<@pwhalen:fedora.im>

16:32:39

I want images to not regress and rely on other tools to fix.

<@pboy:fedora.im>

16:33:47

It is not a image regressen. It is a new behavios to enhance LVM security and stability - at least that's what thje LVM group says.

<@pwhalen:fedora.im>

16:34:48

I think the best solution is to delete the file in the kickstart. It's wrong anyways.

<@coremodule:fedora.im>

16:35:24

As it stands now, there is no criterion that mandates the arm-image-installer script to be functional. We have install criteria, but dd accomplishes this. It's a "nice to have" but isn't required. If dd works, then the criteria are satisfied. Peter Boy , if you want, you can suggest a criterion addition, but as it stands now, we can't block on it not working.

<@pboy:fedora.im>

16:35:41

pwhalen no it is not! It is a new feature to enhance security and reliability - according to LVM group!

<@pwhalen:fedora.im>

16:36:08

not very reliable in this case though?

<@pwhalen:fedora.im>

16:36:52

its wrong for our usecase. A regression in the functionality. systems won't use vda. could it not use something else rather than device?

<@pwhalen:fedora.im>

16:37:14

but, its not the scripts problem.

<@pwhalen:fedora.im>

16:37:24

We can fix it, but thats hacking it up

<@pwhalen:fedora.im>

16:38:00

And I never want to be in the position I was in for F39 GA. This script should never cause a blocker.

<@pboy:fedora.im>

16:38:16

The goal of the new behavior is to prevent a system to mount devices that are connected to the hardware but don't belong to the running system but to other VMS, as an example. Again: according to LVM maintainers.

<@coremodule:fedora.im>

16:38:48

Is that a common usecase on an ARM system?

<@pwhalen:fedora.im>

16:38:50

Peter BoyI fully understand. Buts wrong.

<@pwhalen:fedora.im>

16:39:23

Peter BoyI fully understand. But its wrong.

<@pboy:fedora.im>

16:40:06

@coremodule That's the position of the LVM developers and maintainers. I would just like to make it working with all SBCs.

<@pwhalen:fedora.im>

16:41:26

so, if we delete the file, we go back to the previous 'insecurity' but lvm commands work

<@pboy:fedora.im>

16:41:31

And by the way, to remove the device file from the image does not resolve the issue. We would have to remove it from the running LVM system. And we really can't do that.

<@pwhalen:fedora.im>

16:41:46

we can do it in the kickstart

<@coremodule:fedora.im>

16:42:20

Peter Boy: Could you coordinate a member of the lvm team to join us next week at this meeting? We're running out of time here, but seems we're at a standstill regarding what to do. Having both teams together could ease communication, and then you don't have to act as a middleman either.

<@pboy:fedora.im>

16:42:37

pwhalen: No, we have to create a bootable filesystem which has a correct image file. That is the task at first boot, along with installing a user and password.

<@pboy:fedora.im>

16:43:07

@coremodule I could ask David Teigland, yes.

<@coremodule:fedora.im>

16:43:25

That would be great!

<@pboy:fedora.im>

16:43:31

And I can try to explain all details on mailing list.

<@coremodule:fedora.im>

16:43:55

!action pboy to invite a member of the LVM team to next week's meeting regarding the new LVM changes that are impacting server images.

<@pwhalen:fedora.im>

16:44:05

And if someone uses dd? They will need to solve it themselves.

<@pboy:fedora.im>

16:44:23

Just a question: who is the owner of the ARM SBC image file? Who cares about it?

<@pwhalen:fedora.im>

16:44:34

what does that mean?

<@pwhalen:fedora.im>

16:44:39

What image file?

<@pwhalen:fedora.im>

16:44:54

the disk image?

<@pboy:fedora.im>

16:45:19

the image file that arm-image-installer uses to create a bootable file system.

<@pwhalen:fedora.im>

16:46:11

Not really following the line of questioning here.

<@pboy:fedora.im>

16:47:13

I mean, who do we have to involve to modify the disk image?

<@pwhalen:fedora.im>

16:48:06

Modify how?

<@pboy:fedora.im>

16:48:39

To delete the wrong device file from the image and to create a proper one during first boot.

<@pwhalen:fedora.im>

16:48:52

on the kickstart, during image creation.

<@pwhalen:fedora.im>

16:49:07

creating it, you would need to write a service I guess.

<@pwhalen:fedora.im>

16:49:52

`server-lvm-hack.service`

<@coremodule:fedora.im>

16:50:31

Hey, we need to move on to cover the other topics. Let's reconvene next week with the LVM team and see if we can figure out a solution that works for both.

<@pwhalen:fedora.im>

16:50:37

ack

<@pboy:fedora.im>

16:50:42

ack

<@coremodule:fedora.im>

16:50:50

!topic 2) ==== Candidate Release ====

<@coremodule:fedora.im>

16:50:54

!info Fedora-40-20240220.n.0

<@coremodule:fedora.im>

16:50:58

!link https://openqa.fedoraproject.org/tests/overview?distri=fedora&version=40&build=Fedora-40-20240220.n.0&groupid=5

<@coremodule:fedora.im>

16:51:06

!info Raspberry Pi 4 and 400 won't boot into graphical environment

<@coremodule:fedora.im>

16:51:09

!link https://bugzilla.redhat.com/show_bug.cgi?id=2264415

<@coremodule:fedora.im>

16:51:13

!info Fedora 40 aarch64 Workstation disk image is oversize

<@coremodule:fedora.im>

16:51:17

!link https://bugzilla.redhat.com/show_bug.cgi?id=2247611

<@coremodule:fedora.im>

16:51:29

The two bugs listed above have been accepted as F40 Beta blockers. I haven

<@coremodule:fedora.im>

16:51:59

t had a chance to look at F40 ARM workstation on RPi, looks like some testing there would be great to confirm the graphical boot issue.

<@coremodule:fedora.im>

16:53:16

!topic 2) ==== Kernel Status ====

<@coremodule:fedora.im>

16:53:21

!info F39 kernel-6.7.5-200.fc39

<@coremodule:fedora.im>

16:53:25

!link https://koji.fedoraproject.org/koji/buildinfo?buildID=2405078

<@jlinton:fedora.im>

16:53:26

There is also the shim blacker

<@coremodule:fedora.im>

16:53:29

!info F40 kernel-6.8.0-0.rc5.41.fc40

<@jlinton:fedora.im>

16:53:31

There is also the shim blocker

<@coremodule:fedora.im>

16:53:33

!link https://koji.fedoraproject.org/koji/buildinfo?buildID=2406127

<@coremodule:fedora.im>

16:53:36

!info Rawhide kernel-6.8.0-0.rc5.41.fc41

<@coremodule:fedora.im>

16:53:40

!link https://koji.fedoraproject.org/koji/buildinfo?buildID=2406115

<@coremodule:fedora.im>

16:54:05

Good point, I missed that.

<@jlinton:fedora.im>

16:54:26

https://pagure.io/fedora-qa/blocker-review/issue/1450

<@coremodule:fedora.im>

16:54:28

jforbes, can you give a quick update on the kernel status?

<@pwhalen:fedora.im>

16:54:44

does 6.8 boot on hw for folks? Not booting in virt

<@jlinton:fedora.im>

16:55:40

It was booting on the x13s

<@coremodule:fedora.im>

16:55:41

I haven't tried personally

<@jlinton:fedora.im>

16:56:12

and probably my pi4/etc

<@pwhalen:fedora.im>

16:56:49

I was seeing issues with kernels with the git suffix, the initial rc build seems ok

<@pwhalen:fedora.im>

16:57:16

But in virt mostly, trying hw today

<@coremodule:fedora.im>

16:57:32

moving on for time's sake

<@coremodule:fedora.im>

16:57:33

!topic 3) ==== Bootloader Status ====

<@coremodule:fedora.im>

16:57:37

!info uboot-tools-2024.04-0.2.rc2.fc40

<@coremodule:fedora.im>

16:57:41

!link https://koji.fedoraproject.org/koji/buildinfo?buildID=2403298

<@coremodule:fedora.im>

16:57:53

New uboot as of 5 days ago, could use testing here too

<@pwhalen:fedora.im>

16:58:14

Just booted on Rpi4

<@coremodule:fedora.im>

16:58:26

good to know, which image did you boot pwhalen?

<@linuxfriend01:fedora.im>

16:58:33

Can I test it also on Pinebook Pro ?

<@pwhalen:fedora.im>

16:58:41

(need to doublecheck the version)

<@coremodule:fedora.im>

16:58:42

Andreas Reschke: sure!

<@linuxfriend01:fedora.im>

16:59:03

Great, will do it soon

<@pwhalen:fedora.im>

16:59:05

Andreas Reschkeyes please! I dont have a serial console for that..always leary

<@coremodule:fedora.im>

16:59:21

!topic 5) ==== Open Floor ====

<@coremodule:fedora.im>

16:59:33

We're gonna skip rawhide since I'm not worried about it at this stage

<@coremodule:fedora.im>

16:59:37

Anything for open floor?

<@pwhalen:fedora.im>

17:00:02

Nothing from me. I will look at the server issue more though, try to comment on that bug.

<@coremodule:fedora.im>

17:00:21

Thanks pwhalen , that will help clear up the communication channels.

<@pboy:fedora.im>

17:00:40

I'll try to better explain the issue on mailing list.

<@pwhalen:fedora.im>

17:01:06

Peter, I do understand it. I disagree the solution is with the script.

<@coremodule:fedora.im>

17:01:12

!endmeeting