16:02:41 <dcavalca> #startmeeting Fedora ELN SIG (2022-04-22)
16:02:41 <zodbot> Meeting started Fri Apr 22 16:02:41 2022 UTC.
16:02:41 <zodbot> This meeting is logged and archived in a public location.
16:02:41 <zodbot> The chair is dcavalca. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:02:41 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:02:41 <zodbot> The meeting name has been set to 'fedora_eln_sig_(2022-04-22)'
16:02:53 <dcavalca> #meetingname eln
16:02:53 <zodbot> The meeting name has been set to 'eln'
16:02:59 <SSmoogen> here
16:03:17 <dcavalca> #chair sgallagh SSmoogen dcavalca
16:03:17 <zodbot> Current chairs: SSmoogen dcavalca sgallagh
16:03:23 <dcavalca> #topic init process
16:03:38 <dcavalca> sgallagh is out today fyi
16:03:47 <dcavalca> .hi
16:03:49 <zodbot> dcavalca: Something blew up, please try again
16:03:52 <zodbot> dcavalca: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:03:55 <dcavalca> lol thanks zodbot
16:04:07 <asosedkin> t184256 is me
16:04:09 * t184256 is asosedkin
16:04:30 <dcavalca> .hello2
16:04:31 <zodbot> dcavalca: Something blew up, please try again
16:04:34 <zodbot> dcavalca: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:04:50 <dcavalca> ok nevermind
16:04:54 <jkonecny[m]> Hi everyone
16:05:20 <jforbes> Do we have an agenda?
16:05:31 <dcavalca> #topic Old Business
16:05:44 <dcavalca> jforbes: looking now
16:06:39 <dcavalca> doesn't look like it, at least not on devel@
16:06:56 <dcavalca> checking logs for the last one
16:07:23 <dcavalca> ok, so for old business, we had two actions from the last one
16:08:31 <dcavalca> sgallagh and asosedkin to finalize the branch-vs-rawhide approach
16:08:31 <dcavalca> clang Conditionalize openssl in ELN to behave as C9S
16:08:31 <dcavalca> anybody has updates on these?
16:08:31 <t184256> we've decided to go for a separate branch following c9s
16:09:13 <michel> .hello salimma
16:09:14 <zodbot> michel: salimma 'Michel Alexandre Salim' <michel@michel-slm.name>
16:09:15 <t184256> it hasn't been landed yet, as we've started with a mass-rebuild first and half the stuff's broken there by itself
16:09:44 <jkonecny[m]> .hello jkonecny
16:09:45 <zodbot> jkonecny[m]: jkonecny 'Jiří Konečný' <jkonecny@redhat.com>
16:09:49 <t184256> if Fedora ELN war installable and rebuildable, that'd be just great =/
16:10:03 <dcavalca> iirc we did get a good compose earlier in the week
16:10:20 <dcavalca> or at least, one that didn't make my builds explode :)
16:10:34 <t184256> for clang, he's landed the OpenSSL change, I believe
16:10:52 <jkonecny[m]> Any issues with the installation or just packages?
16:10:52 <Eighth_Doctor> .hello ngompa
16:10:53 <t184256> can we also have one installable with anaconda? =)
16:10:53 <zodbot> Eighth_Doctor: ngompa 'Neal Gompa' <ngompa13@gmail.com>
16:11:25 <asosedkin> jkonecny[m]: https://github.com/fedora-eln/eln/issues/88
16:11:33 <jkonecny[m]> Do you have BZ or something about the installation issue?
16:11:49 <dcavalca> yeah please file a ticket if you haven't already
16:11:49 <jkonecny[m]> Looking
16:12:05 <dcavalca> ah thanks
16:12:06 <t184256> there are many, masking each other. reproducer's attached
16:13:38 <jkonecny[m]> I wonder about this
16:13:41 <t184256> https://github.com/fedora-eln/eln/issues/87 also qualifies, but is likely workaroundable
16:13:53 <dcavalca> dumb question: do we have something to do this kind of testing already in fedora?
16:14:07 <jkonecny[m]> OpenQA
16:14:41 <jkonecny[m]> And Anaconda has kickstart tests but it's more a project upstream testing than Fedora
16:14:46 <Eighth_Doctor> why isn't ELN running through OpenQA testing?
16:14:56 <dcavalca> cool, yeah that looks worthwhile to do
16:16:10 <jkonecny[m]> Anaconda project had ELN included in our upstream tests bit we removed that because of stability issues...
16:16:39 <jkonecny[m]> But these were really just unit tests not integration tests
16:16:46 <dcavalca> jkonecny[m]: can we add it back and get issues filed against ELN when it breaks?
16:17:01 <dcavalca> but yeah, I think we need both actually
16:18:05 <dcavalca> Eighth_Doctor: I suspect because nobody's hooked it up yet
16:18:20 <jkonecny[m]> We could probably even add the integration tests we have, issue is that these are not easy to maintain and team don't have a capacity for that really
16:19:06 <dcavalca> jkonecny[m]: is the concerns that the tests break because of ELN specifically, or something else?
16:19:31 <jkonecny[m]> I can try to convince team to put the ELN tests back again, the issue is again time spend on that
16:20:21 <dcavalca> I think it'd be worth a try, we can't really improve this if we don't have visibility on when things break
16:21:17 <jkonecny[m]> ok, I'm not precise here. We are still running the ELN unit tests daily: https://github.com/rhinstaller/anaconda/actions/runs/2205114928
16:21:32 <jkonecny[m]> we just disabled them on pull requests
16:21:38 <jkonecny[m]> that was painful
16:22:00 <jkonecny[m]> so yes, it's still running and it's even green for some time
16:22:04 <dcavalca> oh good
16:22:09 <dcavalca> yeah I don't think we need this on PRs
16:22:34 <jkonecny[m]> if you want, we might even add the kickstart tests part but not a simple thing to create that nor maintain
16:23:03 <jkonecny[m]> https://github.com/rhinstaller/kickstart-tests
16:23:45 <dcavalca> oh yeah that does look useful
16:23:57 <jkonecny[m]> right now we are running that on Rawhide, RHEL8 and RHEL9 https://github.com/rhinstaller/kickstart-tests/actions/runs/2204792769
16:24:06 <dcavalca> and would probably help us catch breakage like the one t184256 hit down the road
16:24:34 <SSmoogen> Eighth_Doctor, I expect 1) someone needs to help Adam make it work. 2) Someone needs to actually work on giving openqa more resources as it is usually overburdened doing the tests for things it has now
16:25:50 <jkonecny[m]> It should be "easy" to enable if we have a boot.iso which will be used for the testing
16:25:59 <jkonecny[m]> Something like release candidate
16:26:48 <jkonecny[m]> I mean the kickstart tests
16:26:51 <dcavalca> looks like we do
16:26:53 <dcavalca> e.g. https://odcs.fedoraproject.org/composes/production/Fedora-ELN-10-20220422.0/compose/BaseOS/x86_64/iso/Fedora-ELN-10-20220422.0-x86_64-boot.iso
16:27:11 <jkonecny[m]> Completely different issue is to solve failures and flakes
16:27:42 <dcavalca> I think just getting signal to begin with would be great, we can make these non-blocking until they get stable enough
16:27:49 <dcavalca> and in the meantime fix the outstanding issues
16:28:40 <dcavalca> SSmoogen: for 2), would funding some cloud instances help with that?
16:31:01 <SSmoogen> no openqa runs on bare metal hardware
16:32:29 <SSmoogen> at least it did when I was part of CPE and was asked to move it to cloud. It is its own 'cloud' in some ways and needs to have bare metal to run all its vms and virtual devices that tests are captured on
16:32:52 <Eighth_Doctor> that shouldn't be an obstacle, iirc, aws metal instances would be able to simulate that
16:33:12 <Eighth_Doctor> I'm pretty sure between davdunc and Davide Cavalca, we could get some of those in place
16:33:30 <Eighth_Doctor> dunno if there are ARM metal instances, but if there are, that'd also be useful
16:33:33 <davdunc[m> yea.
16:33:38 <SSmoogen> there are arm metal instances
16:34:04 <davdunc[m> there are Intel, AMD,and Graviton (neoverse) metal instances.
16:34:23 <Eighth_Doctor> then those would be useful for OpenQA
16:34:47 <SSmoogen> sure.. you guys knock yourself out :)
16:35:13 <dcavalca> this is a bit of a tangent, but yeah, my point was that if the blocker here is capacity, we can try to help with that
16:36:59 <SSmoogen> I think if it can be done, then that would be great. It would require some heavy lifting of people who know AWS networking, serial/etc. These boxes regularly get wedged and have to be dealt with by hand
16:37:49 <t184256> I can help with periodic testing on x86 VMs and reporting failures, but somebody please make it go green at least once
16:38:09 <davdunc[m> SSMoogen happy to help with that. I can definitely provide guidance and tech contribution on the AWS side.
16:38:28 <SSmoogen> adamw, ^^ you have a victim who wants to help with openqa into the cloud
16:38:57 <SSmoogen> my work here is done
16:39:16 <Eighth_Doctor> we should probably find some folks on IBM Cloud side to give us POWER and Z too
16:39:22 <Eighth_Doctor> those are really big gaps
16:39:52 <SSmoogen> seriously though, davdunc[m and Eighth_Doctor please set up something with adamw and help him on it to see if it is possible after F36 goes out
16:40:03 <Eighth_Doctor> sure
16:40:15 <Eighth_Doctor> we already have plans to talk about Cloud Edition testing anyway
16:40:24 <Eighth_Doctor> we can just add this on top
16:41:00 <davdunc> ack SSmoogen we'll work with him in the regular QA meetings too.
16:41:22 <dcavalca> #action Eighth_Doctor davdunc dcavalca meet with adamw to discuss openqa in the cloud to help with ELN testing capacity
16:42:01 <dcavalca> #action jkonecny[m] look into enabling kickstart tests for ELN
16:42:28 <dcavalca> did I miss anything else from the previous discussion?
16:43:54 <jkonecny[m]> Yeah, do we have something like nightly boot.iso for ELN?
16:44:22 <dcavalca> jkonecny[m]: yeah, it ends up here: https://odcs.fedoraproject.org/composes/production/latest-Fedora-ELN/compose/BaseOS/x86_64/iso/
16:44:35 <dcavalca> but the name has the compose date in it
16:44:55 <dcavalca> if you need a stable URL I suppose we could try and get it symlinked to boot.iso or something like that
16:45:17 <jkonecny[m]> That would be definitely helpful
16:46:02 <dcavalca> #action dcavalca file an issue to get a stable boot.iso symlink for the latest compose
16:46:28 <dcavalca> I say file an issue because I have no clue how to actually fix this, but hopefully someone else does :p
16:46:59 <jkonecny[m]> Thank you
16:47:21 <dcavalca> alright, anything else on this before we move on?
16:49:32 <dcavalca> alright, moving on
16:49:46 <dcavalca> #topic New Business
16:50:05 <dcavalca> I wanted to signal boost https://bugzilla.redhat.com/show_bug.cgi?id=2077299
16:50:23 <dcavalca> I'm seeing reliable build failures on fedora-eln-ppc64le in copr
16:50:33 <dcavalca> (and, interestingly, only in copr)
16:51:40 <michel> That's weird
16:51:59 <michel> It didn't even get started
16:51:59 <SSmoogen> hmmm I don't know if all copr ppc builds are done on power9 systems
16:52:16 <dcavalca> it seems to be hitting a bunch of SIGILL in the postinst for ca-certificates
16:52:26 <praiskup> all builds prefer Power8 unless we talk about EL9 builds
16:52:33 <dcavalca> which I assume is why it then dies in the transaction
16:52:58 <praiskup> ah, ELN .. not sure, does ELN ppc64le require Power9 builders?
16:53:04 <dcavalca> yeah this is ELN
16:53:05 <michel> Is there any significant difference between the rawhide PPC and eln PPC builders?
16:53:55 <praiskup> michel, no difference in Copr
16:54:05 <SSmoogen> I am not sure if it does or not. I would expect that if it did hard require it, then all the eln builds would fail
16:54:17 <SSmoogen> s/it/power9/
16:54:37 <praiskup> why?
16:55:20 <SSmoogen> I would figure that the instructions which are p9/p10 only would be in glibc
16:55:45 <adamw> SSmoogen: davdunc hah, they say that *now* ;)
16:55:52 <dcavalca> https://download.copr.fedorainfracloud.org/results/@meta/drgn/fedora-eln-ppc64le/04294758-python-drgn/builder-live.log.gz is one example fyi
16:55:59 <SSmoogen> and other items which would cause trying to run code on an 8 to go SIGILL
16:56:18 <SSmoogen> dcavalca, so those 3 were not the only builds which were failing?
16:56:32 <praiskup> yes, from "Permanently added '2620:52:3:1:dead:beef:cafe:c210'" that was the Power8 machine
16:56:40 <dcavalca> SSmoogen: I had a few more but they were older so I didn't bother adding them to the BZ
16:56:47 <praiskup> the power9 machines have IPv4 addresses
16:57:11 <dcavalca> praiskup: can we try pinning eln to the power9 builders to see if that fixes this?
16:57:47 <dcavalca> SSmoogen: but yeah, it was reliably failing for me on every build
16:58:18 <dcavalca> we're almost out of time, anything else folks wanted to discuss?
16:58:53 <praiskup> dcavalca, done
16:59:17 <dcavalca> thanks praiskup, I'll try again and report back on the BZ
16:59:49 <praiskup> dcavalca, but ATM we small amount of Power9 builders .. it may take longer to get the builder
17:00:16 <dcavalca> that's fine, at least for my usecase we're using these for CI with packit
17:00:26 <dcavalca> thanks for the help
17:00:32 <dcavalca> alright, we're out of time
17:00:40 <dcavalca> thanks everyone, really good discussion today
17:00:45 <dcavalca> #endmeeting