16:01:21 <dtometzki> #startmeeting Infrastructure (2023-01-19)
16:01:21 <zodbot> Meeting started Thu Jan 19 16:01:21 2023 UTC.
16:01:21 <zodbot> This meeting is logged and archived in a public location.
16:01:21 <zodbot> The chair is dtometzki. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:01:21 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:01:21 <zodbot> The meeting name has been set to 'infrastructure_(2023-01-19)'
16:01:21 <dtometzki> #meetingname infrastructure
16:01:21 <zodbot> The meeting name has been set to 'infrastructure'
16:01:21 <dtometzki> #chair nirik zlopez nb bodanel dtometzki jnsamyak
16:01:21 <zodbot> Current chairs: bodanel dtometzki jnsamyak nb nirik zlopez
16:01:21 <dtometzki> #info Agenda is at: https://board.net/p/fedora-infra
16:01:21 <dtometzki> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:01:22 <dtometzki> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:01:22 <dtometzki> #topic greetings!
16:01:29 <smooge> hello
16:01:36 <erolkskn> Hey there!
16:02:00 <nirik> morning
16:02:13 <jnsamyak> .hi
16:02:15 <zodbot> jnsamyak: jnsamyak 'Samyak Jain' <samyak.jn11@gmail.com>
16:02:28 <erolkskn> .hello erolkeskin
16:02:29 <zodbot> erolkskn: erolkeskin 'Erol Keskin' <erolkeskin.dev@gmail.com>
16:02:40 <dtometzki> .hi
16:02:42 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <linux@tometzki.de>
16:02:58 <mkonecny> .hello zlopez
16:02:59 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@pacse.eu>
16:03:16 * jnsamyak thinks zodbot is taking a leave today
16:03:22 <dtometzki> hello together nice to meet you
16:03:36 <dtometzki> #topic New folks introductions
16:03:36 <dtometzki> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:03:36 <dtometzki> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:07 <dtometzki> welcome to all new peopleś
16:04:15 <erolkskn> Im certainly not a new one but i haven't been here for a long time (i guess 3 months), its great to comeback
16:04:27 <dtometzki> hi erolkskn
16:05:00 <erolkskn> hi dtometzki
16:05:11 <dtometzki> erolkskn, Perhaps you can say anything about you
16:05:44 <dtometzki> ?
16:05:50 <erolkskn> ofc
16:06:00 <dtometzki> what are you hobby intrest ?
16:06:15 <dtometzki> ofc ?
16:06:42 <erolkskn> i was working on fedbadges project 3 months ago but some job changes happened back then so i suddenly find myself in a position to take a break from the development of fedbadges and other fedora projects
16:07:01 <nirik> welcome back erolkskn. :)
16:07:10 <dtometzki> great
16:07:15 <erolkskn> now i'm working on an experimental architecture for fedbadges again
16:07:25 <erolkskn> nirik thanks! great to see you guys again :)
16:08:15 <dtometzki> so then let us go to the next topic
16:08:19 <nirik> erolkskn: there's been some group forming on badges... might check with them and see whats been done since you were last around.
16:08:19 <dtometzki> #topic Next chair
16:08:20 <dtometzki> #info magic eight ball says:
16:08:20 <dtometzki> #info chair 2023-01-12 - eddiejenningsjr
16:08:20 <dtometzki> #info chair 2023-01-19 - dtometzki
16:08:20 <dtometzki> #info chair 2023-01-26 - phsmoura
16:08:20 <dtometzki> #info chair 2023-02-02 - mkonecy
16:08:35 <mkonecny> I see a typo in my name :-D
16:08:45 <dtometzki> ohh
16:08:59 <nirik> mkonecny: perhaps it's someone with a similar name doing it? ;)
16:09:01 <erolkskn> nirik: yeap we are in contact with sandro and we had the first meeting this week to start things :)
16:09:46 <mkonecny> nirik: Could be :-)
16:09:49 <dtometzki> mkonecny, i corrected it directly
16:10:01 <mkonecny> Thanks
16:10:31 <dtometzki> anyone available for 02-09 ?
16:11:06 <jnsamyak> Book me up jnsamyak
16:11:56 <dtometzki> #info chair 2023-02-09 - jnsamyak
16:12:06 <dtometzki> thanks jnsamyak
16:12:33 <dtometzki> so
16:12:38 <dtometzki> #topic announcements and information
16:12:38 <dtometzki> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:12:38 <dtometzki> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:12:38 <dtometzki> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:12:40 <dtometzki> #info oncall should also handling #fedora-releng pings if possible
16:12:42 <dtometzki> #info New design of Anitya available to try on https://stg.release-monitoring.org
16:12:44 <dtometzki> #info F38 mass rebuild starts on 2023-01-18
16:12:46 <dtometzki> #info F38 branching from rawhide on  2023-02-07
16:12:48 <dtometzki> #info SCM Requests are now processed automatically
16:13:01 <dtometzki> any additional announcements ?
16:13:11 <dtometzki> or information
16:14:15 <dtometzki> no then letz go forward
16:14:22 <dtometzki> #topic Oncall
16:14:22 <dtometzki> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:14:22 <dtometzki> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:14:22 <dtometzki> ## .oncalltakeeu .oncalltakeus
16:14:22 <dtometzki> #info eddiejenningsjr is on call from 2023-01-05 to 2023-01-12
16:14:23 <dtometzki> #info nirik is on call from 2023-01-13 to 2023-01-20
16:14:25 <dtometzki> #info dtometzki is on call from 2023-01-20 to 2023-01-27
16:14:51 <dtometzki> hmm anything is wrong here with the dates ?
16:15:09 <dtometzki> Isnt it from thrusday to thursday ?
16:15:28 <nirik> normally yeah
16:15:59 <dtometzki> i will take oncall from today and coreect the dates
16:16:05 <dtometzki> i will take oncall from today and correct the dates
16:16:08 <darknao> I can take the next available round if needed
16:16:09 <nirik> sounds good
16:17:32 <dtometzki> #info darknao is on call from 2023-01-26 to 2023-02-02
16:18:33 <dtometzki> anyone more ?
16:18:40 <dtometzki> thanks darknao
16:19:08 <dtometzki> .oncalltakeeu
16:19:09 <zodbot> dtometzki: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
16:20:29 <dtometzki> #info Summary of last week: (from current oncall )
16:20:39 <dtometzki> nirik, your part ?
16:21:15 <nirik> not much activity, pretty quiet...
16:21:19 <dtometzki> msg from zodbot it worked now after identify
16:21:43 <dtometzki> ok nice to hear
16:22:15 <dtometzki> then point 2 on the topic list
16:22:19 <dtometzki> #topic Monitoring discussion [nirik]
16:22:19 <dtometzki> #info https://nagios.fedoraproject.org/nagios
16:22:19 <dtometzki> #info Go over existing out items and fix
16:22:31 <dtometzki> nirik, again :-) ?
16:22:31 <nirik> so, lets see...
16:22:43 <nirik> I filed a new ticket about some alerts we should clean up:
16:22:59 <nirik> https://pagure.io/fedora-infrastructure/issue/11090
16:23:35 <nirik> other than that, pretty much normal alerts... there's a few we are getting a lot tho:
16:24:02 <dtometzki> ok thanks for information
16:24:03 <nirik> There's a sync of the container registry index... and sometimes it's getting 0 length and alerting. Need to track that down
16:24:07 <nirik> and
16:24:33 <nirik> sometimes we are having proxies hit OOM and kill httpd... need to figure out whats going on there.
16:24:41 <nirik> otherwise, pretty much nomal. ;)
16:25:12 <dtometzki> ok
16:26:02 <dtometzki> I think i will start with the learning topic today we have a discussion round
16:26:05 <nirik> if no questions, we can move on... if anyone wants to work on any of that let me know
16:26:33 <dtometzki> #topic Learning topic
16:26:33 <dtometzki> ```
16:26:33 <dtometzki> #topic Upcoming learning topics
16:26:33 <dtometzki> #info 2023-01-19 Home Lab Setup Roundtable Discussion
16:26:33 <dtometzki> #info 2023-02-02 Using Btrbk for BTRFS snapshot backups [eddiejennings]
16:26:33 <dtometzki> #info 2023-02-16 Automating SCM Requests [mkonecny]
16:27:01 <dtometzki> Discussion is open Home Lab Setup Roundtable Discussion
16:27:08 <nirik> oh yeah. fun. :)
16:27:38 <nirik> so, I recently replaced my home server... I had a blog post or two on it...
16:28:06 <nirik> https://www.scrye.com/wordpress/nirik/2023/01/06/ryzen-powerhouse-pc-detailed-linux-review/ and https://www.scrye.com/wordpress/nirik/2023/01/06/ryzen-powerhouse-pc-detailed-linux-review/
16:28:27 <dtometzki> my own home lap is small 1. thinkpad with fedora some rasberry pi and since last week two visionfive 2
16:28:39 <nirik> oops... https://www.scrye.com/wordpress/nirik/2022/12/13/some-thoughts-on-a-new-home-server/ is the first one
16:29:02 <nirik> dtometzki: riscv! :)
16:29:11 <dtometzki> ohh and a small virtual maschine on AWS
16:29:31 <dtometzki> nirik, yes it is a great board
16:29:50 <mkonecny> What are you running on visionfive 2?
16:30:02 <nirik> currently I have the main server and some vm's on it doing my domain... I've been pondering doing some kind of k8s setup, but I am not sure which to choose.
16:30:24 <smooge> mine is a 8 year old dell server and 8 VM's running on a Windows 11 Dell precision plus 2 raspberry pis waiitng for me to build a koji for them
16:30:27 <erolkskn> ryzen9 5900x with 32gb ram running Fedora/KDE, i did want to get a 5950x in the first place but when i saw it's price i backed down from it :D
16:30:33 <dtometzki> at the moment runs a debian image and i hope fedora will follow :-)
16:30:46 <smooge> oh wait sorry nirik was still talking
16:30:50 <nirik> openshift is fine, but ocp means it would be tied to my work account which I don't want. ;) okd might be an option, but not sure how much it lags currently? k3s sounds interesting.
16:30:55 <dtometzki> i had no mor time since last week :-(
16:31:19 <nirik> smooge: naw, it's all good. :)
16:31:44 <darknao> i've one k3s cluster running on 3 hp elitedesk 800 g2. That run quite well with small power footprint
16:32:04 <darknao> and a couple of NUC devices for VM needs
16:32:07 <darknao> also low power
16:32:20 <darknao> and powerful enough for my lab needs
16:32:24 <nirik> darknao: k3s appeals to me from a complexity standpoint.. ;) How much it translates to other setups I am not sure.
16:32:43 <nirik> do you run k3s in a vm? or just on the bare metal?
16:32:44 <dtometzki> darknao, ohh and heating for room :-)
16:33:22 <mkonecny> I'm running a CentOS 9 Stream on HPE Microserver Gen 10, it's used as a samba server and backup server, I'm also running a telegram bot on it
16:33:23 <darknao> both baremetal and vms. My workers are mostly baremetal, except one that run on VM on my NUC, mainly because I've 10G networking on it
16:33:32 <darknao> and one master also running on VM
16:33:37 <mkonecny> I was migrating from TrueNAS to CentOS 9 Stream and the migration was rough
16:33:45 <darknao> oh, and they are all running FCOS :)
16:33:54 <nirik> if folks don't care about noise and heat and power usage and space... https://deepdiscountservers.com has some nice HW... it's retired cloud nodes. Lots of cpus/memory for not much
16:34:27 <nirik> darknao: nice!
16:34:38 <smooge> i am actually having to care about noise/heat/power more than I would like.
16:34:49 <nirik> yeah, I find myself caring more than I used to.
16:35:22 <mkonecny> I'm running it in attic, so I don't have anyone to complain, but it's silent anyway
16:35:26 <nirik> mkonecny: what filesystem? :)
16:35:29 <darknao> I use some Akasa Turing case on my NUCs, for dead zero noise
16:35:51 <darknao> (they are running in my living room, so that's mandatory)
16:35:53 <nirik> The ryzen I got from silentpc is... really really quiet.
16:35:55 <mkonecny> nirik: I think I went with BTRFS
16:36:00 <mkonecny> Let me check
16:36:08 <erolkskn> nirik: guessing the servers in there is EOL ?
16:36:28 <nirik> erolkskn: yeah... old, but run current fedora just fine.
16:36:58 <mkonecny> nirik: So it's XFS
16:37:06 <nirik> my last box was from them... dell c1100... 10 years old now, still works fine (but it's very loud and kinda slow). 72GB memory and 24 cpus tho
16:38:04 <mkonecny> I was primarily aiming to have a big LVM pool from my drives and didn't really care about filesystem :-)
16:38:21 <mkonecny> nirik: That's a nice machine
16:38:25 <nirik> so, do folks tend to seperate their "home infrastructure" from "lab" machines? or just have everything in the same bucket?
16:38:41 <erolkskn> i've always refrained from buying EOL servers even though they are really cheap for their specs. now i do hear your experiences on them, i want to buy :)
16:39:23 <mkonecny> I don't have any lab machine, which answers your question :-D
16:39:46 <darknao> yeah my lab is pretty much my home infrastructure :p
16:40:10 <mkonecny> I'm usually just playing in VM or on the server directly, I'm the only one using it, so it doesn't matter that much
16:40:14 <nirik> I used to have a dedicated 'test' lan/machines... but not for a long while now
16:40:23 <darknao> but most machines are in clusters, so I can afford playing with one without losing something critical
16:41:03 <eddiejenningsjr> Bah. Was distracted by work, but am here now.
16:41:43 <smooge> as more infrastructure has moved to the cloud... my lab has become more lab versus other things
16:42:02 <nirik> how about networking? how many folks are using 10G? 2.5G? 1G? 100MB? :)
16:42:14 <darknao> 10G fiber here
16:42:24 <darknao> for both internet and home networking
16:42:27 <smooge> the house is 100MB but the room is 1G
16:42:43 <nirik> I'm still on 1G internally... uplink is DSL, so it doesn't matter really
16:43:09 <mkonecny> I have 1G network at home
16:43:13 <nirik> new machine has 2.5G ports... but not sure it's worth getting a 2.5G switch...
16:43:22 <eddiejenningsjr> In my house my father and ran cat6e cable. But the switch I have is 1G switch.  WAN is t-mobile home Internet, so never getting 1G WANA.
16:44:05 <smooge> i didn't know there was a 2.5G network
16:44:29 <erolkskn> 1gbit internal 75mbit WAN (getting more speed from ISP in here can easily race with buying golds from a store)
16:44:30 <darknao> I use some Mikrotik switches for 10G, it's cheaper than I was first expected
16:44:32 <nirik> yeah, we had them run cat6 here when we built this house. But yeah, 1G currently.
16:44:35 <eddiejenningsjr> Working on setting up a Linode VM with WireGuard to defeat CGNAT
16:45:59 <nirik> How about wireless? :) Here I have a Linksys E8450 (wifi6) connected to a port on the server (so it can be isolated from all my wired traffic). Running openwrt. ;)
16:46:48 <eddiejenningsjr> One Ubiquiti AP. Considering changing to TPlink and installing a couple of more APs
16:46:49 <nirik> I also have 3 printers... and I am not sure any of them work. ;( printers are just the worst.
16:47:21 <darknao> ah, that's my weak point I guess, I have an old Unify AP 2.5GHz still running. I need to upgrade that to the latest model at some point
16:47:26 <mkonecny> nirik: I have some cheap Mikrotik, it works :-)
16:48:16 <nirik> the linksys is pretty nice... get pretty good speeds.
16:48:35 <nirik> rx bitrate: 1921.5 MBit/s 160MHz HE-MCS 9 HE-NSS 2 HE-GI 0 HE-DCM 0
16:48:35 <nirik> tx bitrate: 1152.8 MBit/s 160MHz HE-MCS 5 HE-NSS 2 HE-GI 0 HE-DCM 0
16:49:07 <smooge> how are people administering their systems?
16:49:14 <nirik> And for the big question... what OS on your bare hardware? :) All fedora here.
16:49:57 <smooge> Alma, CentOS, Windows, Fedora, and a Ubuntu VM
16:50:08 <eddiejenningsjr> Fedora 37 on my single hypervisor.
16:50:43 <darknao> kube cluster is on CoreOS, VM hypervisors are on Ubuntu (Proxmox), and all VMs are running either Fedora or RHEL
16:50:57 <dtometzki> so the last 5min and we will close the nice discussion
16:50:59 <nirik> smooge: ansible somewhat, but sadly my main vm predates that and is just manually edited/managed. I have poked at cockpit some recently, it's gotten pretty nice for things like rebooting or updating or seeing performance stuff.
16:51:46 <nirik> I keep meaning to redo that main vm, but... it's like 20 years of stuff all tangled up. ;(
16:52:24 <eddiejenningsjr> I try to do any “production” stuff on my lab server with Ansible, from this past summer’s wipe and reinstall
16:53:07 <mkonecny> I'm using cockpit and ssh to manage my server, so nothing complicated
16:54:31 <eddiejenningsjr> Have any of you used oVirt before?  I’ve thought of poking with it to get a sense of what RHEV is like.
16:55:46 <dtometzki> #topic Open Floor
16:55:49 <nirik> I'd suggest not... oVirt is going away. ;(
16:56:36 <eddiejenningsjr> Ah. Good to know (ok oVirt going away).
16:56:43 <darknao> is it?
16:56:51 <eddiejenningsjr> /s/ok/on
16:57:11 <nirik> well, thats the impression I get... dunno how true it is.
16:57:17 <darknao> I mean, I know RHV is going away, but I thought oVirt was going to stay
16:57:30 <nirik> oh yes, sorry...
16:57:42 <erolkskn> One thing i'd like to add before meeting ends, I've stumbled upon with weird issue in datagrepper while i was working on this new experimental architecture for fedbadges. For some reason it doesn't return results for some topics when a high date range specified in search criteria. It constantly gives 502 response. And then I thought to myself may be I could troubleshoot this performance
16:57:44 <erolkskn> issue but i guess i do need to have read only access to datanommer db server. So that i can see what's going on underneath when this search criteria specified. Is there any chance that i could get readonly access to datanommer/datagrepper server? Btw i also created relevant issue about this in datagrepper repo: https://github.com/fedora-infra/datagrepper/issues/541
16:57:51 <eddiejenningsjr> Today I learned: RHEV is going away. :(
16:58:39 <darknao> you can try kubevirt if you want :)
16:59:06 <nirik> erolkskn: well, you can try locally? the db is at: https://infrastructure.fedoraproject.org/infra/db-dumps/datanommer.dump.xz but... it's really big I warn you.
16:59:23 <dtometzki> R
16:59:35 <erolkskn> oh there is a dump ? great i will try that
16:59:54 <dtometzki> I will close the meeting many thanks to all and for the great discussion
17:00:09 <erolkskn> 64Gigs :O
17:00:13 <dtometzki> #endmeeting