<@meetbot:fedora.im>
08:57:24
Meeting ended at 2023-11-30 08:57:22 UTC
<@lenkaseg:fedora.im>
16:00:03
!startmeeting Infrastructure (2023-11-30)
<@meetbot:fedora.im>
16:00:03
Meeting started at 2023-11-30 16:00:03 UTC
<@meetbot:fedora.im>
16:00:03
The Meeting name is 'Infrastructure (2023-11-30)'
<@lenkaseg:fedora.im>
16:00:12
!meetingname infrastructure
<@leo:fedora.im>
16:00:16
!hi
<@zodbot:fedora.im>
16:00:18
Leo Puvilland (leo) - he / him / his
<@lenkaseg:fedora.im>
16:00:20
!chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg patrikp
<@lenkaseg:fedora.im>
16:00:27
!info Agenda is at: https://board.net/p/fedora-infra
<@lenkaseg:fedora.im>
16:00:34
About our team: https://docs.fedoraproject.org/en-US/cpe/
<@lenkaseg:fedora.im>
16:00:40
Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
<@lenkaseg:fedora.im>
16:00:45
!topic namaste
<@nirik:matrix.scrye.com>
16:00:48
!hi
<@humaton:fedora.im>
16:00:50
!hello
<@zodbot:fedora.im>
16:00:50
Kevin Fenzi (kevin) - he / him / his
<@zodbot:fedora.im>
16:00:51
Tomáš Hrčka (humaton) - he / him / his
<@pcreech:matrix.org>
16:00:54
!hello pcreech17
<@zodbot:fedora.im>
16:00:56
Patrick Creech (pcreech17)
<@lenkaseg:fedora.im>
16:00:57
Hi everybody!
<@Zlopez:matrix.org>
16:00:59
!hello
<@zodbot:fedora.im>
16:01:02
Michal Konecny (zlopez)
<@lenkaseg:fedora.im>
16:01:16
!hello
<@zodbot:fedora.im>
16:01:17
Lenka Segura (lenkaseg)
<@devic3:matrix.org>
16:01:26
Hello
<@lenkaseg:fedora.im>
16:01:30
!topic New folks introductions
<@lenkaseg:fedora.im>
16:01:39
Do we have new folks here?
<@devic3:matrix.org>
16:02:05
yep, I'm new
<@leo:fedora.im>
16:02:25
welcome!
<@lenkaseg:fedora.im>
16:02:29
devic3, welcome! I saw your email in introductions
<@lenkaseg:fedora.im>
16:02:44
Nice to see new folks ehre
<@pcreech:matrix.org>
16:02:48
Welcome!
<@nirik:matrix.scrye.com>
16:02:50
Hey devic3. welcome!
<@devic3:matrix.org>
16:03:01
that;s awesome, hello everyone :)
<@devic3:matrix.org>
16:03:11
that's awesome, hello everyone :)
<@lenkaseg:fedora.im>
16:03:44
Let's move to next chair!
<@lenkaseg:fedora.im>
16:03:54
!info chair 2023-12-07 nirik
<@patrikp:matrix.org>
16:04:12
Good evening. 👋
<@lenkaseg:fedora.im>
16:04:13
Do we have someone for the week following that>
<@lenkaseg:fedora.im>
16:04:14
?
<@humaton:fedora.im>
16:04:44
that 14th?
<@lenkaseg:fedora.im>
16:04:47
december 14
<@humaton:fedora.im>
16:04:47
I can do that one
<@lenkaseg:fedora.im>
16:04:49
yup
<@nirik:matrix.scrye.com>
16:04:59
I think we should cancel the 21st, 28th, and possibly the 6th?
<@lenkaseg:fedora.im>
16:05:02
!info chair 2023-12-14 jednorozec
<@lenkaseg:fedora.im>
16:05:21
good idea
<@humaton:fedora.im>
16:05:21
yeah we should cancel those.
<@leo:fedora.im>
16:05:27
+1
<@seddik:fedora.im>
16:05:32
morning
<@seddik:fedora.im>
16:05:36
!hi
<@humaton:fedora.im>
16:05:37
but someby should cover oncall
<@zodbot:fedora.im>
16:05:39
seddik alaouiismaili (seddik)
<@humaton:fedora.im>
16:05:45
and we should split it
<@humaton:fedora.im>
16:05:52
I dont mind doing the holidays week
<@nirik:matrix.scrye.com>
16:06:01
So, what we have done in the past over the holidays is 'no one is oncall, please file a ticket' type of thing...
<@Zlopez:matrix.org>
16:06:02
+1 for cancelling
<@lenkaseg:fedora.im>
16:06:46
!info chair 2023-12-21 cancelled
<@lenkaseg:fedora.im>
16:06:51
!info chair 2023-12-28 cancelled
<@lenkaseg:fedora.im>
16:06:56
!info chair 2024-01-04 cancelled
<@darknao:fedora.im>
16:07:03
!hi
<@zodbot:fedora.im>
16:07:04
Francois Andrieu (darknao) - he / him / his
<@lenkaseg:fedora.im>
16:07:08
!topic announcements and information
<@lenkaseg:fedora.im>
16:07:12
Hi darknao!
<@lenkaseg:fedora.im>
16:07:24
!info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0830 UTC in https://matrix.to/#/#meeting-3:fedoraproject.org
<@humaton:fedora.im>
16:07:28
But I still dont mind taking the oncall, as it goes lot of people contribute while on holidays...
<@lenkaseg:fedora.im>
16:07:33
!info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
<@nirik:matrix.scrye.com>
16:07:53
jednorozec: sure, if you want to, thats fine with me... hopefully no one uses it. ;)
<@lenkaseg:fedora.im>
16:08:34
Do we have some announcements?
<@aheath1992:matrix.org>
16:08:36
!hi
<@zodbot:fedora.im>
16:08:38
Andrew Heath (aheath1992)
<@nirik:matrix.scrye.com>
16:09:09
!info torrent is now moved from torrent02 (rhel7) to torrent01 (rhel9). Please report any problems with it.
<@Zlopez:matrix.org>
16:09:29
!info Staging authentication is currently broken as we are trying to figure out upgrading IPA to RHEL9
<@nirik:matrix.scrye.com>
16:10:14
I started to do people to rhel9, but... we need to finish off the move of planet first. ;) I may destroy the people01 I made for now and do it again later...
<@lenkaseg:fedora.im>
16:11:22
Thanks for the info!
<@lenkaseg:fedora.im>
16:11:29
let's move to oncall?
<@lenkaseg:fedora.im>
16:12:00
!topic Oncall
<@lenkaseg:fedora.im>
16:12:07
!info https://fedoraproject.org/wiki/Infrastructure/Oncall
<@lenkaseg:fedora.im>
16:12:14
https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
<@lenkaseg:fedora.im>
16:12:31
!info zlopez is on call from 2023-11-24 to 2023-11-30
<@lenkaseg:fedora.im>
16:12:39
!info Leo is on call from 2023-12-01 to 2023-12-07
<@lenkaseg:fedora.im>
16:12:52
#!info <name> is on call from 2023-12-08 to 2023-12-14
<@lenkaseg:fedora.im>
16:13:12
do he have a volunteer for week 8-14 december?
<@humaton:fedora.im>
16:13:44
if there is noone I can take it
<@lenkaseg:fedora.im>
16:13:59
sold!
<@lenkaseg:fedora.im>
16:14:10
!info jednorozec is on call from 2023-12-08 to 2023-12-14
<@lenkaseg:fedora.im>
16:14:17
!info Summary of last week: (from current oncall )
<@humaton:fedora.im>
16:14:18
BTW is there mention anywhere that oncall person should claim irc and matrix?
<@leo:fedora.im>
16:14:33
i think they’re désynchroniser
<@leo:fedora.im>
16:14:41
i think they’re desynchronized
<@nirik:matrix.scrye.com>
16:14:44
The matrix side isn't setup to allow anyone to do it... so I just did it now from the admin side.
<@nirik:matrix.scrye.com>
16:14:52
the irc side does need adjusting.
<@humaton:fedora.im>
16:14:53
ahhh
<@leo:fedora.im>
16:15:00
!oncall
<@zodbot:fedora.im>
16:15:00
The following people are oncall:
● @leo:fedora.im (leo) Current Time for them: 08:15 (America/Los_Angeles)
If they do not respond, please file a ticket (https://pagure.io/fedora-infrastructure/issues)
<@nirik:matrix.scrye.com>
16:15:01
or perhaps...
<@nirik:matrix.scrye.com>
16:15:19
we should just make the irc side say "please go to matrix or file a ticket" ?
<@leo:fedora.im>
16:15:29
+1
<@humaton:fedora.im>
16:15:41
this
<@Zlopez:matrix.org>
16:15:47
I didn't get any ping when I was oncall
<@pcreech:matrix.org>
16:15:54
As the current irc "oncall" +1
<@pcreech:matrix.org>
16:16:01
been that way for a few weeks now :D hehe
<@pcreech:matrix.org>
16:16:08
and that's basically what i've done
<@nirik:matrix.scrye.com>
16:16:27
I can adjust it.
<@Zlopez:matrix.org>
16:16:41
We finally fixed the IPA on staging \o/
<@Zlopez:matrix.org>
16:17:11
Thanks darknao for help
<@lenkaseg:fedora.im>
16:18:04
moving on..
<@lenkaseg:fedora.im>
16:18:08
!topic Monitoring discussion [nirik]
<@lenkaseg:fedora.im>
16:18:16
!info https://nagios.fedoraproject.org/nagios
<@lenkaseg:fedora.im>
16:18:28
Go over existing out items and fix
<@nirik:matrix.scrye.com>
16:18:37
So, not too much going on there. There's some certs I need to renew and a few machines low on disk or with broken disks... but thats all normal...
<@nirik:matrix.scrye.com>
16:18:46
overall it's looking not too bad.
<@nirik:matrix.scrye.com>
16:18:56
I haven't looked at how zabbix is going...
<@seddik:fedora.im>
16:19:36
we cannot connect to zabbix with guest account ?
<@nirik:matrix.scrye.com>
16:19:47
It should have guest yeah...
<@nirik:matrix.scrye.com>
16:19:52
is that not working?
<@seddik:fedora.im>
16:20:27
I tried , without success
<@nirik:matrix.scrye.com>
16:20:30
The prod one is up but only has the server so far... need to ask David Kirwan any status
<@nirik:matrix.scrye.com>
16:20:46
saibug: the prod one, or the staging one/
<@nirik:matrix.scrye.com>
16:20:46
?
<@dkirwan:fedora.im>
16:20:57
o/
<@dkirwan:fedora.im>
16:21:09
guest access should work, least it did let me double check
<@leo:fedora.im>
16:21:12
works for me
<@dkirwan:fedora.im>
16:21:34
https://zabbix.fedoraproject.org/index.php?enter=guest
<@dkirwan:fedora.im>
16:21:36
works for me
<@leo:fedora.im>
16:21:38
just can’t log in w/fas obviously
<@leo:fedora.im>
16:21:43
just click the guest button
<@nirik:matrix.scrye.com>
16:21:45
yeah, working here with both
<@seddik:fedora.im>
16:21:54
https://zabbix.fedoraproject.org
<@dkirwan:fedora.im>
16:21:55
FAS access restricted to members of `sysadmin-noc`
<@seddik:fedora.im>
16:22:09
yes yes
<@leo:fedora.im>
16:22:11
“sign in as guest”
<@seddik:fedora.im>
16:22:19
it works fine
<@seddik:fedora.im>
16:22:24
;)
<@dkirwan:fedora.im>
16:23:00
update on zabbix if you're interested nirik !
<@nirik:matrix.scrye.com>
16:23:11
definitely. :)
<@dkirwan:fedora.im>
16:23:24
so server deployed, but the same config which allows hosts to auto register in staging.. does not appear to work lol
<@dkirwan:fedora.im>
16:23:30
so debugging :(
<@dkirwan:fedora.im>
16:23:50
think its a firewall issue, made changes to the iptables similarly to what we did for staging..
<@nirik:matrix.scrye.com>
16:23:51
the old curse of operations work: It worked in staging. ;)
<@leo:fedora.im>
16:24:12
the infrastructure version of it works on my laptop
<@nirik:matrix.scrye.com>
16:24:47
ok, good luck with it, shout out if you need more hands with any of it.
<@dkirwan:fedora.im>
16:24:57
`Unable to connect to [zabbix01.iad2.fedoraproject.org]:10051 [cannot connect to [[zabbix01.iad2.fedoraproject.org]:10051]: [4] Interrupted system call]`
<@dkirwan:fedora.im>
16:25:03
latest error on an agent ;D
<@nirik:matrix.scrye.com>
16:26:18
huh, weird, those [s seem odd, but that might just be it's logging? like it's litterally trying to pass '[zabbix01.iad2.fedoraproject.org]' as the hostname?
<@nirik:matrix.scrye.com>
16:26:49
anyhow, we don't have to debug it here. Thanks for the update! :)
<@lenkaseg:fedora.im>
16:27:03
Thanks for the updates!
<@lenkaseg:fedora.im>
16:27:10
let's move to the next topic
<@lenkaseg:fedora.im>
16:27:22
would that be learning topic with jednorozec?
<@humaton:fedora.im>
16:27:31
So
<@lenkaseg:fedora.im>
16:27:34
!topic Learning topic
<@lenkaseg:fedora.im>
16:27:48
floor's yours!
<@humaton:fedora.im>
16:28:00
there was query by SouthernG on one of the previous meetings about how to build server respin images
<@seddik:fedora.im>
16:28:15
Fedora openshift ??
<@seddik:fedora.im>
16:28:23
;)
<@seddik:fedora.im>
16:28:32
deploy apps there
<@leo:fedora.im>
16:29:02
oh that’s relatively simple i can explain that for you in dm or wherever
<@leo:fedora.im>
16:29:22
oh that’s relatively simple i can explain that for you in dm or wherever appropriate
<@leo:fedora.im>
16:30:23
oh that’s relatively simple i can explain that for you here or in dm or wherever appropriate
<@humaton:fedora.im>
16:30:24
so should I continue or do you want to talk openshift?
<@nirik:matrix.scrye.com>
16:31:13
jednorozec: go ahead and we can do another openshift one soon?
<@humaton:fedora.im>
16:31:27
okay
<@humaton:fedora.im>
16:31:44
So just shor recap from the first pungi learning topic.
<@humaton:fedora.im>
16:31:59
Pungi is the tool we use to do stuff that koji does not know how to do
<@humaton:fedora.im>
16:32:05
like bootable iso images
<@humaton:fedora.im>
16:32:15
here is documentaion for it https://docs.pagure.org/pungi/index.html
<@humaton:fedora.im>
16:32:48
I will try to shed some light on how the Server image is created
<@humaton:fedora.im>
16:33:31
We will be using latest rawhide compose for this purpose. All information about the compose can be found on this URL https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/
<@humaton:fedora.im>
16:35:02
The process is little bit complex, because koji itself does not know how to build bootable images
<@humaton:fedora.im>
16:36:05
So lets start, I have one rhetorical question for you. What is the .iso image made of?
<@smooge:fedora.im>
16:36:14
1's and 0's
<@humaton:fedora.im>
16:36:20
hah!
<@humaton:fedora.im>
16:36:28
you always get me sir
<@humaton:fedora.im>
16:36:40
yeah and those 1&0 are in rpm packages
<@humaton:fedora.im>
16:37:11
There for the image is created from dnf/yum repositories of packages that we want to include
<@humaton:fedora.im>
16:39:11
So first thing we need to do is to gather the packages we want to use, ususally we want packages but also -debuginfo packages so users/developers can actually debug things.
<@humaton:fedora.im>
16:39:29
```createrepo_c /mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/compose/Server/x86_64/debug/tree --outputdir=/mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/compose/Server/x86_64/debug/tree --pkglist=/mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/work/x86_64/repo_package_list/Server.x86_64.debuginfo.conf --cachedir=/var/cache/pungi/createrepo_c/Fedora-0 --update --update-md-path=/mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/work/x86_64/repo/f40 --skip-stat --database --checksum=sha256 --unique-md-filenames --workers=10 --xz --general-compress-type gz --zck --zck-dict-dir=/usr/share/fedora-repo-zdicts/rawhide
```
<@humaton:fedora.im>
16:39:31
hmm
<@humaton:fedora.im>
16:39:35
how do I do code
<@humaton:fedora.im>
16:39:48
anyway
<@humaton:fedora.im>
16:39:49
createrepo_c /mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/compose/Server/x86_64/debug/tree --outputdir=/mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/compose/Server/x86_64/debug/tree --pkglist=/mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/work/x86_64/repo_package_list/Server.x86_64.debuginfo.conf --cachedir=/var/cache/pungi/createrepo_c/Fedora-0 --update --update-md-path=/mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/work/x86_64/repo/f40 --skip-stat --database --checksum=sha256 --unique-md-filenames --workers=10 --xz --general-compress-type gz --zck --zck-dict-dir=/usr/share/fedora-repo-zdicts/rawhide
<@humaton:fedora.im>
16:40:00
creates repository with -debuginfo package
<@humaton:fedora.im>
16:40:24
createrepo_c /mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/compose/Server/x86_64/os --outputdir=/mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/compose/Server/x86_64/os --pkglist=/mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/work/x86_64/repo_package_list/Server.x86_64.rpm.conf --groupfile=/mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/work/x86_64/comps/comps-Server.x86_64.xml --cachedir=/var/cache/pungi/createrepo_c/Fedora-0 --update --update-md-path=/mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/work/x86_64/repo/f40 --skip-stat --database --checksum=sha256 --unique-md-filenames --workers=10 --xz --general-compress-type gz --zck --zck-dict-dir=/usr/share/fedora-repo-zdicts/rawhide
<@humaton:fedora.im>
16:40:54
will create repository with regular packages
<@humaton:fedora.im>
16:42:12
for anyone interested here is log from the job https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/logs/x86_64/createrepo-Server.rpm.x86_64.log
<@humaton:fedora.im>
16:42:51
As you can see there is few things that are passed to the createrepo_c
<@humaton:fedora.im>
16:43:14
list of packages Server.x86_64.rpm.conf
<@humaton:fedora.im>
16:43:37
and comps file comps-Server.x86_64.xml
<@humaton:fedora.im>
16:44:15
Both of them can be generated by pungi or provided by SCM url
<@humaton:fedora.im>
16:46:16
With the repositories at hand we will need to create a bootable iso that is done by the buildinstall task https://docs.pagure.org/pungi/phases.html#buildinstall
<@humaton:fedora.im>
16:47:54
That boot iamge is than used by pungi to create the actuall Server.iso
<@humaton:fedora.im>
16:48:51
since the pungi steps are running in chroot we haveto use the runroot task in koji to do things
<@humaton:fedora.im>
16:49:32
https://docs.pagure.org/koji/plugins/#runroot
<@leo:fedora.im>
16:49:50
need newline after first set of ticks iirc for code block
<@humaton:fedora.im>
16:50:20
let me try
<@humaton:fedora.im>
16:50:27
```
koji --profile=compose_koji runroot --nowait --task-id --use-shell --channel-override=compose --package=coreutils --package=xorriso --package=isomd5sum --package=lorax --package=which --mount=/mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0 f40-build x86_64 'rm -f /var/lib/rpm/__db*; rm -rf /var/cache/yum/*; set -x; bash /mnt/koji/compose/rawhide/Fedora-Rawhide-20231130.n.0/work/x86_64/tmp-Server/createiso-Fedora-Server-dvd-x86_64-Rawhide-20231130.n.0.iso.sh'
```
<@leo:fedora.im>
16:50:34
hm...
<@humaton:fedora.im>
16:50:40
anyway
<@leo:fedora.im>
16:50:40
```
test
```
<@humaton:fedora.im>
16:51:02
the command I pasted above si the one that was used to create latest rawhide server image
<@humaton:fedora.im>
16:51:39
```
test ```
<@leo:fedora.im>
16:51:45
weird
<@nirik:matrix.scrye.com>
16:51:55
I think it depends on client...
<@humaton:fedora.im>
16:51:57
the task itself was this one https://koji.fedoraproject.org/koji/taskinfo?taskID=109722386
<@lenkaseg:fedora.im>
16:52:16
wth is package xorriso :)
<@humaton:fedora.im>
16:52:36
https://www.gnu.org/software/xorriso/
<@humaton:fedora.im>
16:53:15
its a tool that allows to mange with POSIX fs into the iso format
<@humaton:fedora.im>
16:54:33
here is example of xoriso file with defined operations https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20231130.n.0/work/x86_64/tmp-Server/xorriso-140540202662336.txt
<@humaton:fedora.im>
16:54:44
but I am derailing and we are running out of time
<@humaton:fedora.im>
16:55:10
SO the lask koji tas is of interest to anyone who want to build their own versions of Server variant
<@humaton:fedora.im>
16:56:16
Both log files https://kojipkgs.fedoraproject.org//work/tasks/2386/109722386/root.log and https://kojipkgs.fedoraproject.org//work/tasks/2386/109722386/runroot.log
<@humaton:fedora.im>
16:57:25
have all the steps mentioned today are saved to https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/logs/ and config https://kojipkgs.fedoraproject.org/compose/rawhide/latest-Fedora-Rawhide/work/
<@humaton:fedora.im>
16:57:43
SO
<@humaton:fedora.im>
16:58:03
that is crash course to creating bootable installer image of Fedora Linux Server
<@humaton:fedora.im>
16:58:25
Questions are welcome
<@lenkaseg:fedora.im>
16:58:30
Thank you jednorozec
<@leo:fedora.im>
16:58:37
thank you!
<@lenkaseg:fedora.im>
16:58:45
jednorozec++
<@zodbot:fedora.im>
16:58:47
lenkaseg gave a cookie to humaton. They now have 50 cookie(s), 1 of which were obtained in the Fedora 39 release cycle
<@Zlopez:matrix.org>
16:59:21
jednorozec++
<@zodbot:fedora.im>
16:59:24
zlopez gave a cookie to humaton. They now have 51 cookie(s), 2 of which were obtained in the Fedora 39 release cycle
<@lenkaseg:fedora.im>
16:59:34
i will end the meeting now, but feel free to continue with questions and open floor
<@leo:fedora.im>
16:59:38
jednorozec++
<@zodbot:fedora.im>
16:59:40
leo gave a cookie to humaton. They now have 52 cookie(s), 3 of which were obtained in the Fedora 39 release cycle
<@lenkaseg:fedora.im>
17:00:08
ah, maybe the questions (if any) should be recorded
<@nirik:matrix.scrye.com>
17:00:13
Thanks jednorozec
<@humaton:fedora.im>
17:01:05
Feel free to close the meeting, I am putting this into blogpost and any eventual questions will be addressed there
<@lenkaseg:fedora.im>
17:01:29
Alright then!
<@lenkaseg:fedora.im>
17:01:30
!endmeeting