16:05:32 <phsmoura> #startmeeting Infrastructure (2023-03-02)
16:05:32 <zodbot> Meeting started Thu Mar  2 16:05:32 2023 UTC.
16:05:32 <zodbot> This meeting is logged and archived in a public location.
16:05:32 <zodbot> The chair is phsmoura. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:05:32 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:05:32 <zodbot> The meeting name has been set to 'infrastructure_(2023-03-02)'
16:05:32 <phsmoura> #meetingname infrastructure
16:05:32 <zodbot> The meeting name has been set to 'infrastructure'
16:05:32 <phsmoura> #chair nirik zlopez nb bodanel dtometzki jnsamyak
16:05:32 <phsmoura> #info Agenda is at: https://board.net/p/fedora-infra
16:05:32 <zodbot> Current chairs: bodanel dtometzki jnsamyak nb nirik phsmoura zlopez
16:05:32 <phsmoura> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:05:33 <phsmoura> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:05:35 <phsmoura> #topic greetings!
16:05:39 <phsmoura> .hi
16:05:40 <zodbot> phsmoura: phsmoura 'Pedro Moura' <pmoura@redhat.com>
16:06:26 <aheath1992> .hi
16:06:27 <zodbot> aheath1992: aheath1992 'Andrew Heath' <aheath1992@gmail.com>
16:06:43 <mkonecny> .hello zlopez
16:06:44 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@pacse.eu>
16:06:46 <nirik> morning
16:07:27 <phsmoura> #topic New folks introductions
16:07:27 <phsmoura> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:07:27 <phsmoura> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:07:57 <phsmoura> are there new people around? feel free to introduce yourselves
16:09:06 <phsmoura> #topic Next chair
16:09:06 <phsmoura> #info magic eight ball says:
16:09:07 <phsmoura> #info chair 2023-03-09 - penguinpee
16:09:07 <phsmoura> #info chair 2023-03-16 - ???
16:09:07 <phsmoura> #info chair 2023-03-23 - ???
16:09:07 <phsmoura> #info chair 2023-03-30 - ???
16:09:55 <phsmoura> anyone willing to be the host in next meetings?
16:10:00 <nirik> So, the week of the 23rd lots of us will be at a in person meeting of the cpe team. We may want to cancel that week?
16:10:26 <phsmoura> ah true :) forgot that for a moment
16:10:31 <phsmoura> +1 for cancelling
16:11:04 <nirik> But we still need people to run the meeting the other weeks. ;)
16:11:29 <aheath1992> I have never run a meeting but I can try
16:11:36 <mkonecny> +1 for cancelling
16:11:58 <phsmoura> yep, anyone willing to run this meeting in 2023-03-16?
16:12:14 <phsmoura> aheath1992: adding you then ok?
16:12:18 <aheath1992> Sure
16:12:31 <aheath1992> will lean on you guys if i need to
16:12:45 <phsmoura> np
16:12:55 <nirik> it's not hard... ;)
16:13:07 <phsmoura> now, for 2023-03-30?
16:13:42 <nirik> I guess I can if no one else.
16:14:33 <phsmoura> ok, this is our chair for this month
16:14:38 <phsmoura> #info chair 2023-03-09 - penguinpee
16:14:38 <phsmoura> #info chair 2023-03-16 - aheath1992
16:14:38 <phsmoura> #info chair 2023-03-23 - Canceled due to F2F CPE meeting
16:14:38 <phsmoura> #info chair 2023-03-30 - nirik
16:14:58 <nirik> +1
16:15:06 <phsmoura> #topic announcements and information
16:15:21 <phsmoura> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:15:21 <phsmoura> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:15:22 <phsmoura> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:15:22 <phsmoura> #info oncall should also handling #fedora-releng pings if possible
16:15:23 <phsmoura> #info Fedora 38 beta freeze in effect
16:15:36 <phsmoura> any other announcement?
16:17:20 <phsmoura> #topic Oncall
16:17:20 <phsmoura> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:17:20 <phsmoura> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:18:01 <phsmoura> info nirik is on call from 2023-02-23 to 2023-03-02
16:18:01 <phsmoura> #info eddiejennings is on call from 2023-03-02 to 2023-03-09
16:18:01 <phsmoura> #info darknao is on call from 2023-03-10 to 2023-03-16
16:18:21 <phsmoura> ops missed the # in the 1st info
16:19:09 <phsmoura> nirik: do you have a summary of this week?
16:19:12 <eddiejenningsjr> Yep. I’m one week removed from Covid and ready for on-call :P. I’ll take it one when I’m done with work meetings.
16:19:41 <nirik> Only one oncall ping (a bit ago) about some ppc64le builders not checking in... otherwise pretty quiet.
16:20:59 <phsmoura> eddiejenningsjr: sorry, I didnt understand. do you want us to remove your name for next oncall?
16:22:02 <eddiejenningsjr> No. Keep me on-call.
16:22:06 <eddiejenningsjr> I’m fine :)
16:22:22 <phsmoura> ok :)
16:22:55 <phsmoura> So anyone willing to be on call on 2023-03-17 to 2023-03-23 or  2023-03-24 to 2023-03-30?
16:24:06 <aheath1992> what does it take to be on call
16:25:02 <phsmoura> basically  handle incoming requests for work and triage them for the current week
16:25:07 <nirik> https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/#_the_oncall_role_in_our_team
16:26:34 <phsmoura> we are good for the next weeks
16:27:02 <phsmoura> eddiejenningsjr: dont forget to kneel before zod :)
16:27:10 <phsmoura> moving to monitoring
16:27:25 <phsmoura> #topic Monitoring discussion [nirik]
16:27:25 <phsmoura> #info https://nagios.fedoraproject.org/nagios
16:27:26 <phsmoura> #info Go over existing out items and fix
16:28:44 <nirik> No real changes here...
16:28:45 <nirik> I think alerts in general have been less of late, which is a good thing
16:29:56 <nirik> I think we can move on unless someone has something to bring up about it.
16:30:00 <mkonecny> Maybe because of the freeze
16:32:01 <phsmoura> usually we go through blocked/old tickets but today we have a learning topic
16:32:26 <phsmoura> nirik: can I change the topic so you can talk about Installing vm's with ansible?
16:33:07 <nirik> Sure!
16:33:15 <phsmoura> #topic Learning topic
16:33:20 <phsmoura> all yours :)
16:33:43 <nirik> So, we use ansible to deploy/install our vm's... it's pretty simple and works pretty well.
16:34:08 <nirik> The various playbooks that configure vm's all include a task file where the install magic happens:
16:34:19 <nirik> thats ansible/tasks/virt_instance_create.yml
16:34:57 <nirik> Basically what happens is that it first tries to ssh to the vm. If it can successfully do that, the entire instance create task is skipped and it goes on to the rest of the playbook to configure things.
16:35:54 <nirik> host variables contain a vmhost: variable for each vm. That tells it what host is supposed to run that vm
16:36:15 <nirik> So, the next thing it does is check the running vm's on that host. (ie, basically a 'virsh list --all' on it)
16:36:36 <nirik> if the vm is there, it exits, because something is wrong...
16:37:06 <nirik> if not, then it moves on to running a virt-install on the host. This install uses more host variables to pass virt-install a kickstart file and repos, etc.
16:37:32 <nirik> the virt-install calls are mostly in group_vars/all (because we need slightly different ones for various different arches, etc)
16:37:47 <nirik> ansible then waits for the virt-install to finish, the vm to start up and answer ssh.
16:38:03 <nirik> then it moves on to the rest of the playbook to configure the host.
16:39:30 <nirik> Thats basically the process in summary. Now there's lots of things we could do differently... for example, we could use a image file instead of doing a full install... but back in the day when we set this up it was thought that a new install was more clean / better because we could decide exactly what packages and how we wanted it setup.
16:40:03 <nirik> any questions? I'm trying to think if I missed anything. :)
16:40:34 <mkonecny> Link to role?
16:40:47 <nirik> Back when we had a openstack cloud we also had a similar task for cloud instances... it didn't do virt install, but just called nova to spin the instance up
16:41:21 <nirik> creating/installing the vm is a task... tasks/virt_instance_create.yml
16:41:33 <nirik> you can see it included in the various playbooks:
16:41:56 <nirik> - import_playbook: "/srv/web/infra/ansible/playbooks/include/virt-create.yml myhosts=koji_stg:koji"
16:42:10 <nirik> in the koji_hub.yml playbook for example
16:42:42 <nirik> Oh yeah, there's a layer of include there now because ansible didn't like including a bare task
16:43:02 <nirik> so playbooks/include/virt-create.yml just is a small playbook that includes the task.
16:44:02 <nirik> The kickstart files are in a kickstart repo. It's not public currently because it has a bunch of historical stuff with passwords and all kinds of things. I have someday a plan to make a new public repo for those...
16:44:34 <nirik> or perhaps we could just pull them into our ansible repo
16:45:33 <nirik> I think thats everything...
16:46:07 <mkonecny> That's good to know about kickstart repo
16:46:25 <nirik> Happy to answer questions on it.
16:47:46 <nirik> if nothing else, we could move to open floor. ;)
16:48:46 <phsmoura> ok, lets move to the last topic
16:48:48 <phsmoura> #topic Open Floor
16:49:16 <mkonecny> nirik: I will probably try to shadow you for some of the things you do
16:49:54 <mkonecny> Because I still sometimes feel lost in our infrastructure :-D
16:50:06 <nirik> sure happy to share what I know. :)
16:51:16 <mkonecny> Today I tried to just remove some spam and even when following guide I couldn't do some of the things
16:51:34 <mkonecny> .ticket 11158
16:51:35 <zodbot> mkonecny: Issue #11158: Spam report: paulijennifer - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/11158
16:52:16 <nirik> our guide probibly needs updating.
16:52:28 <aheath1992> I have a PR that needs review and I need to sync on the status of another project
16:52:35 <nirik> I was considering trying to focus some more time on updating docs soon
16:52:49 <mkonecny> I can help with that
16:54:03 <nirik> I can review pr's and such... but note that it's only nearing 9am here, so I have not had time to read my email yet. ;)
16:54:48 <aheath1992> No rush I also want to sync where we are with the syslog to Splunk work as well
16:55:27 <nirik> I think thats all ready, we just need to make changes on log01... but... might be something for after freeze.
16:55:37 <aheath1992> Ack sounds good
16:56:23 <nirik> and the PR might need updating for the certs/etc...
16:56:48 <nirik> oh, and we are now sending all our openshift app logs there, so... the volume is bigger. Hopefully splunk will be ok with that.
16:57:21 <aheath1992> They should if not I can Open a ticket to have them resize
16:58:03 <nirik> we can always adjust things after.
16:58:18 <nirik> and hopefully the data is of use there. ;) (for them and us)
16:59:21 <aheath1992> yep Im sure we can make use of the data some how
17:01:29 <phsmoura> I think thats all for this meeting
17:02:34 <nirik> yep.
17:02:40 <phsmoura> Thank you all for joining and have a great weekend :)
17:02:50 <phsmoura> #endmeeting