16:04:38 <jrichardson> #startmeeting Infrastructure (2022-03-3)
16:04:38 <zodbot> Meeting started Thu Mar  3 16:04:38 2022 UTC.
16:04:38 <zodbot> This meeting is logged and archived in a public location.
16:04:38 <zodbot> The chair is jrichardson. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:04:38 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:04:38 <zodbot> The meeting name has been set to 'infrastructure_(2022-03-3)'
16:04:38 <petebuffon> morning
16:04:38 <jrichardson> #meetingname infrastructure
16:04:38 <jrichardson> #chair nirik siddharthvipul mobrien zlopez pingou bodanel dtometzki jnsamyak computerkid
16:04:38 <jrichardson> #info Agenda is at: https://board.net/p/fedora-infra
16:04:38 <jrichardson> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:04:38 <zodbot> The meeting name has been set to 'infrastructure'
16:04:38 <zodbot> Current chairs: bodanel computerkid dtometzki jnsamyak jrichardson mobrien nirik pingou siddharthvipul zlopez
16:04:39 <jrichardson> #topic greetings!
16:04:44 <petebuffon> .hello petebuffon
16:04:46 <zodbot> petebuffon: petebuffon 'Peter Buffon' <pabuffon@gmail.com>
16:04:47 <eddiejennings> .hi
16:04:49 <zodbot> eddiejennings: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:04:59 <nirik> morning all
16:04:59 <jrichardson> .hi
16:05:00 <zodbot> jrichardson: jrichardson 'James Richardson' <jamricha@redhat.com>
16:05:32 <eddiejennings> jrichardson, Thanks for starting the meeting.  I'm now available :D
16:05:43 <jrichardson> no prob ;)
16:06:17 <eddiejennings> #topic New folks introductions
16:06:40 <eddiejennings> bah.  Do I need to do '#chair' to be able to interact with zod?
16:07:50 <jrichardson> #addchair eddiejennings
16:08:07 <eddiejennings> #addchair eddiejennings
16:08:09 <darknao> .hi
16:08:10 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:08:34 <jrichardson> addchair eddiejennings
16:08:42 <jrichardson> hmm
16:09:37 <AustinPowered[m]> .hi
16:09:38 <zodbot> AustinPowered[m]: Sorry, but user 'AustinPowered [m]' does not exist
16:09:59 <austinpowered> .hi
16:10:00 <zodbot> austinpowered: austinpowered 'T.C. Williams' <fedoraproject@wootenwilliams.com>
16:10:23 <eddiejennings> Do we have anyone new with us?  If so, introduce yourself and say a little bit about yourself :)
16:10:38 <eddiejennings> Don't be shy.  nirik and the rest won't bite :D
16:10:46 <nirik> every day seems like a new day, does that count? no? ;)
16:11:15 <austinpowered> I still feel like a newbie drinking from a fire hose
16:11:44 <eddiejennings> nirik do you have some sage advice on how to allow me to interact with zod since jrichardson started the meeting in my absence?
16:12:32 <jrichardson> ill log off
16:12:35 <nirik> naw...
16:12:39 <jrichardson> see if that helps
16:12:40 <nirik> #chair eddiejennings
16:12:40 <zodbot> Current chairs: bodanel computerkid dtometzki eddiejennings jnsamyak jrichardson mobrien nirik pingou siddharthvipul zlopez
16:12:45 <nirik> thats what you need ^
16:12:48 <nirik> should be all fine now.
16:12:51 <eddiejennings> nirik has the power!
16:13:05 <eddiejennings> Hearing no new folks chime in, let's move along.
16:13:09 <eddiejennings> #topic Next chair
16:13:09 <eddiejennings> #info magic eight ball says:
16:13:09 <eddiejennings> #info chair 2022-03-10 - dtometzki
16:13:09 <eddiejennings> #info chair 2022-03-17 - mkonecny
16:13:13 <mkonecny> .hello zlopez
16:13:14 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@psmail.xyz>
16:13:33 <eddiejennings> Any volunteers for 3/24?
16:14:59 <eddiejennings> It's not critical we have one right now, since the next two weeks are covered.  But if anyone is interested let us know.
16:15:46 <eddiejennings> Hearing none, let's move along.
16:15:51 <eddiejennings> #topic announcements and information
16:15:51 <eddiejennings> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:15:51 <eddiejennings> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:15:51 <eddiejennings> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:15:53 <eddiejennings> #info matrix sig is forming, see https://discussion.fedoraproject.org/t/bi-weekly-meeting-for-matrix-sig-or-matrix-team/35947/3 if interested
16:15:56 <eddiejennings> #info Bugzilla will introduce breaking Authentication API change on 28th February
16:15:58 <eddiejennings> #info Infrastructure freeze in place leading up to the Fedora 36 Beta release.
16:16:17 <eddiejennings> Any new announcements for this week?
16:16:18 <nirik> The bugzilla breakage is now Mar 15th
16:16:46 <nirik> #info thread on fedoraplanet on infrastructure list, chime in if you have thoughts on it.
16:16:46 <eddiejennings> #info Bugzilla will introduce breaking Authentication API change on 15th March
16:17:43 <eddiejennings> Any other announcements?
16:18:35 <nirik> nope
16:18:53 <eddiejennings> Excellent.  Moving along.
16:18:55 <eddiejennings> #topic Oncall
16:19:09 <eddiejennings> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:19:09 <eddiejennings> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:19:09 <eddiejennings> ## .oncalltakeeu .oncalltakeus
16:19:09 <eddiejennings> #info darknao on call from 2022-03-04 to 2022-03-10
16:19:09 <eddiejennings> #info Leo on call from 2022-03-11 to 2022-03-17
16:19:42 <eddiejennings> Any volunteers for 2022-03-18 - 2022-03-24?
16:20:26 <eddiejennings> I can do the week after from 2022-03-25 - 2022-03-31
16:20:39 <nirik> sure, I can...
16:20:58 <eddiejennings> #info eddiejennings on call from 2022-03-25 to 2022-03-31
16:21:26 <eddiejennings> sold to nirik!  3/18 - 3/24
16:21:41 <eddiejennings> #info nirik on call from 2022-03-18 to 2022-03-24
16:22:17 <eddiejennings> #info Summary of last week: (from current oncall )
16:22:32 <eddiejennings> What news from oncall from this past weeek?
16:22:40 <petebuffon> no pings
16:22:54 <eddiejennings> That's the best kind of news :D
16:22:57 <nirik> quiet is good
16:23:03 <petebuffon> indeed :)
16:23:14 <eddiejennings> And with that we hand the floor to nirik
16:23:16 <eddiejennings> #topic Monitoring discussion [nirik]
16:23:17 <mkonecny> We have a beta freeze, so this could be it
16:23:26 <eddiejennings> #info https://nagios.fedoraproject.org/nagios
16:23:26 <eddiejennings> #info Go over existing out items and fix
16:23:51 <nirik> so, we got a lot of clean up last week...
16:24:12 <nirik> bunch of our mgmt interfaces were in a weird state and that got cleared so they show up now.
16:24:41 <nirik> we still have one totally down machine, worker05.stg, but folks are at the datacenter today and will reset it.
16:25:07 <nirik> badges has been alerting a lot... not sure why. :( misc was looking into it.
16:25:49 <nirik> thats about it. mostly the normal stuff after that
16:25:55 <nirik> any questions? if not we can move on
16:26:40 <eddiejennings> Hearing none, let's move on.
16:27:01 <eddiejennings> I was out last week on honeymoon, but according to the agenda it looks like we have a learning topic this week on koji
16:27:03 <eddiejennings> thus. . .
16:27:22 <eddiejennings> #topic Learning topic
16:27:35 * nirik tries to remember if that was me, or mkonecny ?
16:27:38 <petebuffon> i started interacting with koji this week so i'm excited!
16:28:01 <petebuffon> also hope you had a nice time eddiejennings
16:28:07 <eddiejennings> On the board.net board it says you nirik, but I'm sure either would be fine :)
16:28:08 <mkonecny> nirik: It should be mentioned on the learning topic
16:28:31 <nirik> ah yeah, me.
16:28:39 <eddiejennings> petebuffon, I did. Thanks!
16:28:40 <nirik> ok, no worries, happy to pontificate on this for a bit. ;)
16:29:30 <nirik> So, koji is our main buildystem. We run a instance, but we aren't upstream development. There's a team of folks in Red Hat that maintain it upstream.
16:29:50 <nirik> Long ago it started out just building rpms, but now bunches of other things have been tacked on.
16:30:44 <nirik> koji has several parts. There are hub(s), builder(s) and usually a database server (I suppose you could run all of them on one host if you wanted, but we don't)
16:31:37 <nirik> The hubs are the heart of the system. On them, koji runs as a wsgi application. The hubs run the web interface to koji as well as a xmlrpc interface.
16:32:01 <nirik> the hubs talk to the database for keeping track of things. (only the hubs talk to the db, nothing else needs to)
16:32:26 <nirik> Builders talk to the hub and get build jobs from the hub.
16:32:56 <nirik> This sometimes confuses folks, but it's a pull model... the builders ask the hub 'hey, what can I build' and get jobs... the hubs never reach out to builders.
16:33:27 <nirik> In our setup we have a db ( db-koji01), two hubs (koji01 and koji02) and a LOT of builder.
16:33:59 <nirik> Builders take build jobs (if they can do it, based on arch, weight of the job, etc)
16:34:37 <nirik> Then typically for rpms at least, they run mock to setup a chroot, then do the build in that chroot, then they upload any results back to the hub
16:35:17 <nirik> For other things, say containers, a builder may run openshift code to hand off the build to OSBS to build.
16:35:43 <nirik> Or image builds, builder may call oz/ImageFactory, or live-media-creator
16:36:43 <nirik> There's also a build type called 'runroot'. This basically tells the builder to run the command and in these cases only, results are written out to nfs volume instead of uploading back to the hub.
16:37:08 <nirik> koji also keeps track of everything it can. It was designed to make sure you knew exactly what rpms were used to build your rpm.
16:37:26 <nirik> so it keeps track of every buildroot of every package that it build (and shipped).
16:38:27 <nirik> For builders, we have x86_64 builders (both vms and hw), aarch64 (vm's and hw), armv7 (vm's), ppc64le (vm's) and s390x (vm's)
16:38:52 <nirik> we have a staging setup also...
16:39:17 <nirik> and from time to time we take the production database, prune it some and load it into staging. This allows us to keep it up with prod mostly.
16:39:41 <nirik> we currently have around 200 builders.
16:40:19 <nirik> Sadly, koji doesn;'t have any ability to 'burst' and add/remove builders based on load. You have to setup the builder and add it and then it's there until you disable/remove it.
16:40:45 <nirik> There's also a process that keeps track of repos called 'kojira'
16:40:53 <nirik> For us it runs on koji02.
16:41:20 <nirik> So it looks and sees when something has built and was added to a build tag and regenerates the repo
16:41:45 <nirik> koji operates on a concept of tags. Builds are tagged with various tags for various reasons.
16:42:39 <nirik> Take a look at: 'koji list-tag-inheritance f37 --reverse' if you want to see how complex it can be. ;)
16:42:59 <nirik> Basically for rawhide now we have a f37 tag. Builds that are tagged in f37 are in rawhide (or will be in the next compose)
16:43:32 <nirik> when you build a rawhide package it goes to a f37-candidate tag, then it's signed and moved to a pending tag, etc...
16:44:30 <nirik> It's important to note that koji has no concept of whats 'newer', it only knows what order things were tagged into a tag. So, it can get the latest tagged build, but you can't ask it for the highest version, you have to do that yourself.
16:45:22 <nirik> While koji itself doesn't do any signing... it has some support around that. Our signing system imports signatures into koji and koji can then use that to write out a signed copy of a package.
16:46:13 <nirik> We also have a koji-gc set of scripts that does garbage collection on builds we aren't going to use. Things that aren't tagged in anything, or where there's more things tagged after it, or were never used in a buildroot, etc.
16:46:55 <nirik> Our backend storage for koji is a netapp NFS volume. It's mounted on koji01/02 and all builders in the runroot channel.
16:46:58 <nirik> It's gigantic.
16:47:21 <nirik> ntap-iad2-c02-fedora01-nfs01a:/fedora_koji   78T   76T  1.9T  98% /mnt/fedora_koji
16:47:36 <eddiejennings> wow
16:47:48 <nirik> It's on my todo list to split it up, but it's not easy as koji expects things to be in specific trees.
16:48:00 <petebuffon> is that size relatively stable or has it just been increasing over time?
16:48:08 <nirik> it jumps around.
16:48:41 <nirik> https://admin.fedoraproject.org/collectd/bin/graph.cgi?hostname=koji01.iad2.fedoraproject.org;plugin=df;plugin_instance=mnt-fedora_koji;type=df_complex;begin=-31622400
16:48:49 <nirik> but I'm sure it's slowly growing over time.
16:49:32 <petebuffon> got it
16:49:39 <nirik> (do note that it's somewhat confusing because the netapp volume has autogrow, so it will make it bigger if it gets close enough to full...)
16:50:09 <mkonecny> I remember we had a really nice diagram that was showing what the workflow of the tags is
16:50:10 <mkonecny> But I can't find it now
16:50:19 <nirik> The koji builders are all Fedora latest stable. After a new release we upgrade/reinstall them all to that version.
16:50:51 <nirik> This is because we often have things like glibc or rpm that make changes that would make rhel builders unable to build fedora packages.
16:51:39 <nirik> This isn't such a big deal these days because mock now uses bootstrap chroots (that is: it makes a chroot for the target os, then uses the rpm/dnf/yum from that to install the buildroot in another chroot and builds there)
16:52:04 <nirik> I think thats mostly what I can think of. Any questions on any of it?
16:53:15 <eddiejennings> Thank you for the overview nirik
16:53:17 <nirik> Happy to answer questions or explain more in detail whenever. :)
16:54:06 <eddiejennings> Let's move along
16:54:08 <eddiejennings> #topic Open Floor
16:54:18 <jrichardson> nirik++
16:54:24 <petebuffon> nirik++
16:54:37 <eddiejennings> Any other topics that need discussino
16:55:01 <phsmoura_> nirik++
16:55:23 <eddiejennings> nirik++
16:56:19 * nirik has nothing off hand
16:56:58 <eddiejennings> Any others?
16:58:12 <eddiejennings> I think that wraps it up.  Thank you again for the assist at the beginning jrichardson :)
16:58:21 <eddiejennings> #endmeeting