16:00:57 <eddiejenningsjr> #startmeeting Infrastructure (2022-10-27)
16:00:57 <zodbot> Meeting started Thu Oct 27 16:00:57 2022 UTC.
16:00:57 <zodbot> This meeting is logged and archived in a public location.
16:00:57 <zodbot> The chair is eddiejenningsjr. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:57 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:57 <zodbot> The meeting name has been set to 'infrastructure_(2022-10-27)'
16:01:13 <eddiejenningsjr> #meetingname infrastructure
16:01:13 <zodbot> The meeting name has been set to 'infrastructure'
16:01:24 <nirik> morning
16:01:27 <eddiejenningsjr> #chair nirik mobrien zlopez bodanel dtometzki jnsamyak computerkid
16:01:27 <eddiejenningsjr> #info Agenda is at: https://board.net/p/fedora-infra
16:01:27 <eddiejenningsjr> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:01:27 <eddiejenningsjr> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:01:27 <zodbot> Current chairs: bodanel computerkid dtometzki eddiejenningsjr jnsamyak mobrien nirik zlopez
16:01:29 <aheath1992> morning
16:01:33 <ledvd> Hello all
16:01:37 <eddiejenningsjr> #topic greetings!
16:01:46 <eddiejenningsjr> Good morning / afternoon / evening all! :D
16:03:07 <Penguinpee> hello gui1ty
16:03:13 <Penguinpee> .hello gui1ty
16:03:14 <eddiejenningsjr> #topic New folks introductions
16:03:14 <zodbot> Penguinpee: gui1ty 'Sandro .' <gui1ty@penguinpee.nl>
16:03:14 <eddiejenningsjr> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:03:14 <eddiejenningsjr> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:03:37 <eddiejenningsjr> Do we have anyone new with us today?  If so, let yourself be known!
16:03:37 <phsmoura> .hi
16:03:38 <zodbot> phsmoura: phsmoura 'Pedro Moura' <pmoura@redhat.com>
16:03:50 <eddiejenningsjr> .hello eddiejennings
16:03:52 <zodbot> eddiejenningsjr: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:03:52 <ledvd> Yes I'm new :-)
16:04:05 <eddiejenningsjr> Welcome ledvd!
16:04:05 <ledvd> I sent an introduction to the list a few days ago
16:04:52 <ledvd> Thanks! I'm David, here's a link to my intro https://lists.fedoraproject.org/archives/list/fedora-join@lists.fedoraproject.org/thread/2TKMFS2XJT3WBTERSUAGPKP2DLBD7U4P/
16:04:55 <mkonecny> .hello zlopez
16:04:56 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@psmail.xyz>
16:05:26 <eddiejenningsjr> Excellent.  Any other new people with us today?
16:05:51 <Penguinpee> welcome ledvd
16:06:08 <ledvd> Thanks 10,99Penguinpee
16:06:21 <eddiejenningsjr> Hearing none, let's move along.
16:06:35 <eddiejenningsjr> #topic Next chair
16:06:35 <eddiejenningsjr> #info magic eight ball says:
16:06:35 <eddiejenningsjr> # info chair 2022-10-27 - eddiejenningsjr
16:06:57 <eddiejenningsjr> We need a chair for next week 2022-11-03.  It's a fun, easy way to be involved :D
16:07:08 <eddiejenningsjr> Remember, if I can do it, anyone can!
16:07:31 <Penguinpee> I think I already volunteered for next week
16:08:08 <eddiejenningsjr> Ah.  I didn't see it on board.net.  Sold to Penguinpee!
16:08:15 <nirik> cool
16:08:35 <Penguinpee> yeah, maybe zodbot had a day off again. not sure.
16:08:50 <eddiejenningsjr> # info chair 2022-11-03 - Penguinpee
16:08:53 <eddiejenningsjr> ha!  markdown and Element :P
16:09:08 <eddiejenningsjr> any takers for 2022-11-10?
16:09:52 <phsmoura> I can take it
16:10:31 <eddiejenningsjr> Sold to phsmoura!
16:10:55 <eddiejenningsjr> #info chair 2022-11-10 - phsmoura
16:11:20 <eddiejenningsjr> That gives us two weeks, which is fine for now.
16:11:21 <eddiejenningsjr> On to the next stop
16:11:34 <eddiejenningsjr> #topic announcements and information
16:11:45 <eddiejenningsjr> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:11:45 <eddiejenningsjr> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:11:45 <eddiejenningsjr> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:11:45 <eddiejenningsjr> #info oncall should also handling #fedora-releng pings if possible
16:11:45 <eddiejenningsjr> #info f37 final freeze is in effect
16:11:57 <eddiejenningsjr> Is there another go/no-go meeting today?
16:12:09 <Penguinpee> yes. most likely no-go
16:12:22 <mkonecny> #info Upgraded FMN is now running on production
16:12:25 <eddiejenningsjr> #info next go/no-go is today 2022-10-27
16:13:51 <eddiejenningsjr> Any other announcements?
16:14:59 <eddiejenningsjr> Hearing none, we'll continue forward
16:15:09 <eddiejenningsjr> #topic Oncall
16:15:29 <eddiejenningsjr> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:15:29 <eddiejenningsjr> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:15:29 <eddiejenningsjr> ##.oncalltakeeu .oncalltakeus
16:15:55 <eddiejenningsjr> #info eddiejenningsjr is on call from 2022-10-14 to 2022-10-20
16:15:55 <eddiejenningsjr> #info eddiejenningsjr is on call from 2022-10-20 to 2022-10-27
16:15:55 <eddiejenningsjr> #info dtometzki is on call from 2022-10-27 to 2022-11-03
16:16:33 <eddiejenningsjr> We are covered for this week.  How about 2022-11-03 to 2022-11-10?
16:16:58 <nirik> I can take it if noone else wants it... or we can fill it later.
16:17:51 <eddiejenningsjr> We can fill it later.  Perhaps someone next week will be eager. :)
16:17:59 <nirik> sure
16:18:23 <eddiejenningsjr> #info Summary of last week: (from current oncall )
16:18:30 <eddiejenningsjr> So . .   :P
16:19:04 <eddiejenningsjr> There may have been some pings I missed due to the bridge between IRC and Matrix.  I did see a few for some fedoraproject.org URLs failing
16:19:23 <eddiejenningsjr> But alas, they were outside of systems I could reach or things I can touch as an apprentice to do any real troubleshooting.
16:19:51 <nirik> the bridge had a lot of issues... hopefully it's back to stable now
16:19:58 <eddiejenningsjr> There may or may not be a correlation between Nirik's return from vacation and the resolution of said problems :)
16:20:31 * Penguinpee thinks nirik took the bridge on vacation...
16:20:37 <nirik> ha
16:20:39 <eddiejenningsjr> But other than the one day, pings were fairly quiet for the last two weeks
16:21:02 <eddiejenningsjr> And speaking of Nirik. . .
16:21:03 <eddiejenningsjr> #topic Monitoring discussion [nirik]
16:21:03 <eddiejenningsjr> #info https://nagios.fedoraproject.org/nagios
16:21:03 <eddiejenningsjr> #info Go over existing out items and fix
16:21:20 <nirik> lets see
16:21:59 <nirik> we have been hitting some weird firewall issues... RH networking is trying to isolate it and fix it.
16:22:32 <nirik> otherwise nagios looks pretty much the same as it has. ;)
16:22:40 <nirik> badges continues to alert. :(
16:22:56 * Penguinpee sighs
16:23:26 <Penguinpee> it's broken in so many ways. I disabled Copr bades, yet they still show in the logs. :-\
16:23:28 <nirik> not your fault. ;)
16:23:43 <Penguinpee> I know. It's just sad.
16:24:17 <nirik> anyhow, we can move on I think, unless anyone has questions...
16:24:44 <Penguinpee> There's some more insight on https://discussion.fedoraproject.org/t/getting-fedora-badges-back-in-shape/42942
16:24:48 <eddiejenningsjr> None from me.  I just wish I had time to dig into stuff to help :(
16:25:23 <eddiejenningsjr> #topic Learning topic
16:25:41 <eddiejenningsjr> We do have a learning topic for this week.  Nirik's scheduled to give an overview of Koji.
16:25:58 <eddiejenningsjr> Are you ready Nirik, or since you just came off of vacation, do we need to wait a week?
16:26:12 <nirik> I can bloviate on koji anytime. ;)
16:26:38 <Penguinpee> take it away
16:26:41 <eddiejenningsjr> :D  Take it away!
16:26:44 <nirik> so, sure, let me go ahead. :) Feel free to ask questions as you think of them...
16:27:27 <nirik> koji is our main build system. It's been that for quite a long while and took over from another simplier system called plague. ;)
16:28:07 <nirik> koji is written in python and depends on a number of other things to do it's work
16:28:16 <dtometzki> .hi
16:28:17 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <linux@tometzki.de>
16:28:26 <dtometzki> sorry a little bit late
16:28:52 <nirik> it consists of a hub (or hubs, you can run more than 1. We run 2 of them) a database server for it's database and builders for doing the actual building.
16:29:43 <nirik> The hub is passive. builders check in with the hub and ask for jobs to run, which the hub gives them based on a bunch of factors.
16:30:17 <nirik> when a builder takes a task, it runs that locally... if it's a rpm build it's done in a mock chroot.
16:30:34 <nirik> if it's a image build it might call oz to do that or it might call livemedia-creator
16:30:58 <nirik> once the task is completed, the builder uploads the results back to the hub
16:32:14 <nirik> There's a special kind of task... called a runroot task where this isn't the case. In the case of runroot jobs the builder takes the job, but writes the output to a nfs mount locally instead of uploading back to the hub.
16:32:36 <Penguinpee> does koji check if an rpm is installable?
16:32:37 <nirik> (thats done in the cases where the output is some large thing usually)
16:32:57 <nirik> well, depends on what you mean.
16:33:10 <nirik> when a user requests a build, they tell the hub
16:33:35 <nirik> the builder pulls that task and first makes a src.rpm, then using that src.rpm it tries to build it.
16:33:56 <nirik> either step might fail. koji doesn't care tho, it just returns the error(s) back
16:34:56 <Penguinpee> the result of the build in case of packages is one ore  more rpms. i recently had a successful build in copr which fedora-review reported as not installable.
16:35:27 <nirik> no, there's no testing or anything in koji for that. you can definitely build a rpm that cannot be installed (by missing runtime requires or the like)
16:35:42 * Penguinpee nods
16:36:01 <nirik> koji has a concept of tags for groups of packages. tags can inherit from other tags.
16:36:25 <nirik> so for example, the 'f38' tag is all those packages that are currently shipped as 'rawhide'
16:37:19 <nirik> We use tags to control signing and CI/openqa as well... a build might land in a particular tag and be tested and moved to another tag when passing or signed and moved to another tag, etc.
16:37:59 <nirik> koji is mostly managed via its cli... it does have a web interface, but you're limited on what can be done there.
16:38:18 <nirik> koji list-history is very handy to see what tags a package/build went though
16:38:36 <Penguinpee> tags are also very helpful when submitting a bunch of new packages which depend on each other (dependency chain). I used that very recently.
16:39:57 <nirik> lots of things interface with koji... robosignatory (signing), CI/Openqa, pungi (pulls builds and makes composes to ship out), osbs (build containers), mbs (build modules), odcs (build composes on demand), etc
16:40:30 <nirik> it's done well because it's pretty generic and easy to add things on top of. :) There's been talk over the years of making a 2.0 version or something, but it's never happened
16:41:50 <nirik> koji of course has a fedora-messaging plugin, so when actions happen it emits messages to our messagbus... ie, build was tagged into X, build started, build finished, etc... allowing other apps to act on specific things that happen in koji.
16:42:43 <nirik> The koji database and nfs store are quite massive. Easily the biggest things we have. They have all builds ever shipped in fedora since fc6/f7 days.
16:43:11 <Penguinpee> how massive? ball park figure
16:43:17 <nirik> the db is about a TB on disk
16:43:36 <eddiejenningsjr> Non-trivial amount of space
16:43:44 * Penguinpee nods
16:43:47 <nirik> the main koji nfs volume is about 90tb. We have split out some of it
16:44:00 <nirik> we need to split it more (thats on my todo list)
16:44:38 <nirik> there's about another 65TB in archives and 5TB in ostrees
16:45:23 <nirik> We have just about 200 builders currently.
16:45:47 <nirik> koji list-hosts --enabled | wc -l
16:45:47 <nirik> 202
16:45:56 <nirik> most of them are vm's.
16:46:36 <nirik> I think thats most of the overview. Anyone have more specific areas or questions?
16:50:09 <eddiejenningsjr> Thank you for the topic and overview :)
16:50:20 <eddiejenningsjr> In our last few minutes, let's move to . . .
16:50:22 <eddiejenningsjr> #topic Open Floor
16:50:44 <eddiejenningsjr> Any other questions, comments, discussions, snarky remarks? :)
16:51:07 <Penguinpee> go / no-go over in #fedora-meeting in 10 min. should be interesting ;)
16:51:41 <Penguinpee> seems there's another heartbleed coming, yet no-one dares to speak up.
16:52:06 <eddiejenningsjr> Personal plug:  What I thought was going to be a long, but manageable commute has turned into 9-12 hours / week in my car.  If any of you know of remote Linux Admin positions or something that's close to Athens, GA, let me know :)
16:52:13 <nirik> Penguinpee: it's not about daring. ;)
16:52:34 <eddiejenningsjr> GA = Gorgia, United States
16:52:36 <Penguinpee> nirik: I'm teasing
16:52:49 <eddiejenningsjr> Yeah.  I like the gig, but commute isn't sustainable. :(
16:53:28 <eddiejenningsjr> and that's with only going into office 3 days / week,  other two are remote.
16:54:49 <eddiejenningsjr> If there are no other discussion topics, I move to end 5 minutes early :)
16:55:03 <nirik> +1 to end
16:55:14 <petebuffon> hang in there eddiejennings I'm sure it will work out soon :)
16:55:27 <eddiejenningsjr> Thanks :)
16:55:29 <eddiejenningsjr> And on that note
16:55:31 <eddiejenningsjr> #endmeeting