16:00:02 <nirik> #startmeeting Infrastructure (2023-03-30)
16:00:03 <zodbot> Meeting started Thu Mar 30 16:00:02 2023 UTC.
16:00:03 <zodbot> This meeting is logged and archived in a public location.
16:00:03 <zodbot> The chair is nirik. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:03 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:03 <zodbot> The meeting name has been set to 'infrastructure_(2023-03-30)'
16:00:13 <nirik> #meetingname infrastructure
16:00:13 <zodbot> The meeting name has been set to 'infrastructure'
16:00:20 <nirik> #chair nirik zlopez nb bodanel dtometzki jnsamyak
16:00:20 <zodbot> Current chairs: bodanel dtometzki jnsamyak nb nirik zlopez
16:00:24 <nirik> #info Agenda is at: https://board.net/p/fedora-infra
16:00:27 <nirik> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:32 <nirik> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:00:42 <nirik> #topic hola!
16:00:46 <mkonecny> .hello zlopez
16:00:47 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@pacse.eu>
16:01:25 <darknao> .hi
16:01:26 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:01:39 <jnsamyak> .hi2 jnsamyak
16:01:58 <nirik> morning everyone
16:02:07 <dherrera> .hi
16:02:08 <zodbot> dherrera: dherrera 'Diego Herrera' <dherrera@redhat.com>
16:02:25 <jnsamyak> .hello jnsamyak
16:02:26 <zodbot> jnsamyak: jnsamyak 'Samyak Jain' <samyak.jn11@gmail.com>
16:02:32 <jnsamyak> there you go
16:02:55 <nirik> #topic New folks introductions
16:02:59 <nirik> anyone new here today?
16:03:12 <nirik> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:03:12 <nirik> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:21 <nirik> seems not... lets move on
16:04:30 <nirik> #topic Next chair
16:04:37 <nirik> #info magic eight ball says:
16:04:43 <nirik> #info chair 2023-04-06 - mkonecny
16:04:55 <nirik> #info chair 2023-04-13 - eddiejenningsjr
16:04:57 <nirik> #info chair 2023-04-20 - ?
16:05:07 <nirik> anyone want the last one?
16:05:22 <jnsamyak> sign me up it's so long that I drive one ;)
16:06:16 <jednorozec> .hello humaton
16:06:17 <zodbot> jednorozec: humaton 'Tomáš Hrčka' <thrcka@redhat.com>
16:06:27 <nirik> cool. thanks
16:06:28 <nirik> #topic announcements and information
16:06:28 <nirik> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:06:33 <nirik> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:06:40 <nirik> #info Fedora 38 final freeze starts next tuesday
16:06:52 <nirik> any other info/announcements?
16:07:40 <nirik> oh... one more
16:07:56 <nirik> #info bodhi upgraded to 7.1.1
16:08:23 <nirik> ok, moving on then...
16:08:32 <nirik> #topic Oncall
16:08:41 <nirik> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:08:48 <nirik> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:08:58 <nirik> #info eddiejenningsjr is on call from 2023-03-17 to 2023-03-23
16:09:07 <nirik> #info  eddiejenningsjr is on call from 2023-03-24 to 2023-03-30
16:09:13 <nirik> #info ??? is on call from 2023-03-30 to 2023-04-06
16:09:21 <nirik> #info ??? is on call from 2023-04-06 to 2023-04-13
16:09:22 <jednorozec> I can take the nex one
16:09:40 <eddiejenningsjr> Not “it” for the next oncall :P
16:09:43 <nirik> thanks a bunch for taking two weeks Eddie Jennings, Jr.
16:09:51 <eddiejenningsjr> :)
16:10:33 <jednorozec> so 30 is sstarting from today?
16:10:42 <nirik> anyone for the last slot there?
16:10:45 <nirik> yeah
16:10:51 <jednorozec> .oncalltakeeu
16:10:51 <zodbot> jednorozec: Error: You don't have the alias.add capability. If you think that you should have this capability, be sure that you are identified before trying again. The 'whoami' command can tell you if you're identified.
16:11:25 <jednorozec> .oncalltakeeu
16:11:25 <zodbot> jednorozec: Kneel before zod!
16:11:33 <nirik> I guess i can take the next one...
16:11:46 <nirik> #info Summary of last week: (from current oncall )
16:11:56 <nirik> how did it go Eddie Jennings, Jr.
16:12:40 <eddiejenningsjr> Unless folks were summoning oncall without saying my name or using .oncall, it was quiet. Beautify of freezes perhaps?
16:13:08 <nirik> yeah, there were fewer tickets than normal too... perhaps everyone was on holidays?
16:13:16 <nirik> but I'll take it. ;)
16:14:27 <nirik> Oh, one thing I wanted to note: someone at our meetings last week noted that 'oncall' is kind of a bad name... it implies the person is always online and can do lots of things... they said there was another group where they had a role like this but they called it 'watcher'... watching for issues and helping people file tickets, etc...
16:14:48 <nirik> changing it might be too hard now, but thought I would mention it.
16:15:18 <jednorozec> gueardian of the infra
16:15:26 <jednorozec> *guardian
16:15:27 <mkonecny> People are already used to write .oncall when trying to summon the daemon
16:15:45 <nirik> yeah, thats why I think it's probibly not worth changing... but I thought it was an interesting take on things.
16:16:06 <eddiejenningsjr> Matrix makes it much easier, since I get a ping on my phone. I don’t have to always have IRC open.
16:16:42 * jednorozec gets pings from light bulbs
16:16:52 <jnsamyak> Yeah, may be add a info or something about the definition for it perhaps, so it will be a quick copy paste next time we ask someone to be oncall :P
16:17:06 <nirik> anyhow, I guess we should move on... didn't mean to derail things too much. ;)
16:17:51 <nirik> jnsamyak: yeah, we have: https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team which could be improved I am sure.
16:17:58 <nirik> #topic Monitoring discussion [nirik]
16:18:04 <nirik> #info https://nagios.fedoraproject.org/nagios
16:18:40 <nirik> not much change here... still have a bunch of small things that need cleaning up
16:18:42 <nirik> but overall we are doing ok.
16:19:28 <nirik> ok, so we had planned a learning topic today...
16:19:30 <nirik> #topic Learning topic
16:19:40 <nirik> #info 2023-03-30 Learn how do contribute to our infra docs + live fixes [darknao, nirik]
16:20:24 <nirik> darknao: you available? want to go over the flow of how docs are built/distributed? then I was thinking I could go over SOPs and we could look at the state of things and ask for help updating things.
16:21:13 <nirik> or I can go first...
16:21:34 <darknao> yes, sure I can start
16:21:48 <nirik> thanks!
16:22:13 <darknao> so the Fedora documentation is built with Antora, a documentation framework that use AsciiDoc as base
16:22:22 <darknao> https://antora.org
16:23:33 <darknao> this framework is configured to fetch all documentation repositories all around gitlab, github and pagure, and use that to build the final fedora documentation website
16:24:55 <darknao> the antora configuration with all the repositories listed can be found on gitlab: https://gitlab.com/fedora/docs/docs-website/docs-fp-o/-/blob/prod/site.yml
16:25:38 <darknao> any questions so far?
16:26:40 <eddiejenningsjr> Curious that GitLab holds the repo instead of Pagure
16:26:44 <nirik> I have one; is there a preferred site for these repos? should we consider moving our pagure one? or does it matter at this point?
16:27:27 <darknao> it doesn't matter really, each teams have their own preferences on that part
16:28:07 <darknao> most of the documentation team repositories were on pagure not so long ago, but we decided to move them to gitlab
16:28:38 <darknao> but it's up to the team, the documentation pipeline can handle any git forge
16:28:39 * nirik nods
16:28:41 <jnsamyak> +1 because, we have infra-docs-fpo in pagure
16:28:41 <jnsamyak> one of the questions I have is, how does the deployment works for all these documentations
16:30:06 <darknao> so the build pipeline, that run on our openshift cluster, get the antora playbook (the site.yml file I linked above) and fetch all the git repositories listed there
16:30:56 <darknao> then run the antora build system that generate the final documentation website with everything in it
16:31:24 <jnsamyak> I see, so everytime the PRs gets merge do we have an action that takes these changes into account or how does openshift reconciles with the changes (if this makes anysense)
16:31:47 <darknao> after that, the website is rsynced on all our proxies, and made available to everyone on docs.fedoraproject.org
16:31:54 <jnsamyak> PRs in pagure/MRs in gitlab :P
16:32:45 <darknao> the build run every hours on openshift
16:33:25 <nirik> ha, I just happened to go look at the docsbuilding project on openshift and it's currently failing.
16:33:30 <darknao> so once your PR is merged, it'll get automatically pulled in the next scheduled build
16:33:58 <jnsamyak> awesome, good to know, til!
16:34:00 <nirik> no wait. Thats super old. hum
16:34:12 <nirik> it's fine. ;)
16:34:43 <darknao> if your project is on Gitlab, there is a Gitlab CI pipeline that can build a preview website for your MRs
16:36:03 <darknao> now, on each documentation repository, you should find an Antora playbook that can be used to build a minified version of the doc website, you can use to preview your changes locally
16:36:17 <nirik> how about docs.stg.fedoraproject.org?
16:37:09 <darknao> staging use a specific Antora playbook, on the same main repository (but different branch): https://gitlab.com/fedora/docs/docs-website/docs-fp-o/-/blob/stg/site.yml
16:37:21 <darknao> the scheduling and build process is the same as prod
16:38:19 <mkonecny> So the staging instance is mainly for testing build changes?
16:40:03 <darknao> can be used to try new Antora version, a new website template, or just to deploy/test a new documentation project
16:41:06 <nirik> yeah, but PR/MR's and small content changes should just test locally or the like right?
16:41:41 <darknao> yeah small changes should be tested locally if possible
16:42:49 <darknao> I know some teams like to have a public link to review a new documentation page with a broader audience before making it publicly available on the prod website, and then use the staging instance for that
16:43:56 <nirik> Due to the large number of repos here I guess it's not possible to have openshift watch them and do builds when something merges right?
16:45:18 <darknao> yes, it's a bit tricky for that. We have that idea to use fedora messaging to get notified everytime one of the listed repo get updated and trigger a rebuild
16:45:48 <nirik> Might be workable yeah.
16:46:04 <mkonecny> That could be done
16:46:42 <nirik> do we have gitlab events yet? I know we have pagure and github...
16:47:42 <nirik> anyhow, shall I say a few things about our particular docs?
16:48:07 <darknao> please do :)
16:48:34 <nirik> We have https://docs.fedoraproject.org/en-US/infra/ which has infra info, SOPs and also now the releng release docs
16:48:45 <nirik> This is in https://pagure.io/infra-docs-fpo/
16:49:13 <nirik> When we moved to this repo from an internal one, zlopez filed tickets on all the SOPs so we knew to review and fix them up.
16:49:30 <nirik> But we haven't done too well there I'm afraid.
16:49:39 <nirik> There's a lot of outdated things in there.
16:50:00 <nirik> I updated our massupdate SOP this week when doing a massupdate (there's a pr for it, review welcome!)
16:50:27 <nirik> But, we also have our getting started docs on the wiki still. We need to remove that and move it into these docs and update it.
16:50:42 <nirik> I think jrichardson was going to try and work on some of that
16:51:28 <nirik> But really any help welcome. ;) Please submit PR's if you can, or review ones if you can't.
16:52:03 <nirik> One thing today: I reinstalled our bastion servers and ssh host keys changed for some folks. But there's configuration to fix that (posted to the infra list). We should update our ssh access SOP with that.
16:52:40 <nirik> I'm going to try and work more on docs this next quarter.
16:52:53 <nirik> Any questions/comments/ideas? :)
16:53:58 <nirik> ok then. ;)
16:54:02 <nirik> #topic Open Floor
16:54:07 <nirik> anything anyone would like to bring up?
16:54:41 <jnsamyak> Thanks darknao nirik for bringing this up, it was really helpful!
16:54:54 <eddiejenningsjr> Nothing from me. Happy opening day for those who are baseball fans!
16:55:40 <nirik> I'd also really like to work on the onboarding docs for apprentices... add more steps and things to do/explore... but thats a bigger topic I guess.
16:56:40 <mkonecny> We should probably rewrite the onboarding docs and add a list of tasks that the people should try
16:56:53 <nirik> yep. and nuke the wiki getting started pages.
16:57:09 <mkonecny> Yes, that too
16:58:18 <mkonecny> I was just wondering, how do you deal with Swap is low alert @nirik, just restart machine?
16:59:16 <nirik> recent rhel versions seem to more aggressively swap things out... so usually we just do a 'swapoff -a && swapon -a' but we probibly should look at just not installing with swap anymore, or making it bigger.
16:59:38 <nirik> if a machine does it a lot, we can change it in ansible so nagios doesn't monitor swap on that machine.
17:00:16 <mkonecny> I wonder if we still need a swap
17:00:33 <nirik> not sure. Fedora is happy without (since is has zram)
17:00:55 <nirik> anyhow, we are out of time, so lets continue in noc?
17:01:17 <mkonecny> Thanks for running it today
17:01:21 <nirik> #endmeeting