16:00:06 <samyak-jn> #startmeeting Infrastructure (2021-05-13)
16:00:06 <zodbot> Meeting started Thu May 13 16:00:06 2021 UTC.
16:00:06 <zodbot> This meeting is logged and archived in a public location.
16:00:06 <zodbot> The chair is samyak-jn. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:06 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:06 <zodbot> The meeting name has been set to 'infrastructure_(2021-05-13)'
16:00:13 <nirik> morning everyone
16:00:19 <samyak-jn> #meetingname infrastructure
16:00:19 <zodbot> The meeting name has been set to 'infrastructure'
16:00:30 <samyak-jn> #chair nirik smooge siddharthvipul mobrien zlopez pingou bodanel dtometzki jnsamyak
16:00:30 <zodbot> Current chairs: bodanel dtometzki jnsamyak mobrien nirik pingou samyak-jn siddharthvipul smooge zlopez
16:00:37 <samyak-jn> #info Agenda is at: https://board.net/p/fedora-infra
16:00:48 <samyak-jn> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:58 <samyak-jn> #topic aloha
16:01:18 <samyak-jn> Welcome all to today's meeting!
16:01:25 <samyak-jn> .hello jnsamyak
16:01:26 <zodbot> samyak-jn: jnsamyak 'Samyak Jain' <samyak.jn11@gmail.com>
16:02:06 <bodanel> .hello bodanel
16:02:06 <zodbot> bodanel: bodanel 'Bogdan Benea' <benea_bogdan@yahoo.com>
16:02:58 <jednorozec> .hello humaton
16:02:59 <zodbot> jednorozec: humaton 'Tomáš Hrčka' <thrcka@redhat.com>
16:03:03 <t0xic0der> .hello t0xic0der
16:03:04 <zodbot> t0xic0der: t0xic0der 'Akashdeep Dhar' <akashdeep.dhar@gmail.com>
16:03:08 <siddharthvipul> .hello siddharthvipul1
16:03:08 <zodbot> siddharthvipul: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com>
16:03:09 <mizdebsk> .hello2
16:03:11 <zodbot> mizdebsk: mizdebsk 'Mikolaj Izdebski' <mizdebsk@redhat.com>
16:03:20 <Southern_Gentlem> .hello jbwillia
16:03:21 <siddharthvipul> geez, I have been missing this meeting a lot
16:03:24 <zodbot> Southern_Gentlem: jbwillia 'Ben Williams' <vaioof@gmail.com>
16:03:27 <computerkid> .hello2
16:03:28 <zodbot> computerkid: computerkid 'Grayson Penland' <gpenland06@gmail.com>
16:03:44 <computerkid> Hi everyone
16:03:54 <t0xic0der> o/
16:03:59 <dtometzki> .hi
16:04:00 <samyak-jn> \o/
16:04:00 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <linux@tometzki.de>
16:04:08 <siddharthvipul> \o
16:04:11 <nirik> good crowd today
16:04:17 <siddharthvipul> nirik: yess :D
16:04:32 <dtometzki> hello together
16:05:13 <samyak-jn> Hello all once again! Okay, so moving on...
16:05:26 <samyak-jn> #topic New folks introductions
16:05:34 <samyak-jn> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:05:42 <samyak-jn> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:05:57 <samyak-jn> Hi welcome those who are joining us for the first time!
16:06:14 <samyak-jn> Please introduce yourself to the tea, :)
16:06:26 <samyak-jn> s/tea,/team
16:06:46 <siddharthvipul> ah that's alright.. I often introduce myself to the coffee.. it doesn't like me anymore
16:07:06 <dtometzki> :-)
16:07:13 <t0xic0der> More like you do not like the coffee ;-)
16:07:16 <samyak-jn> hahaha :)
16:07:16 <samyak-jn> Talking about I had coffee hear ;)
16:07:23 <siddharthvipul> t0xic0der: noooo, I LOVE IT
16:07:31 <samyak-jn> s/hear/here lol
16:07:32 <siddharthvipul> It gives me anxiety though :"(
16:08:01 * nirik should go get more coffee here.
16:08:01 <Southern_Gentlem> you like it, but your body doesnt
16:08:12 <siddharthvipul> Southern_Gentlem: yeah, that's how I would put it
16:09:06 <samyak-jn> No new faces around? We will go forward to our next topic in one minute.
16:10:06 <samyak-jn> Okay, so moving forward!
16:10:12 <samyak-jn> #topic Next chair
16:10:19 <samyak-jn> #info magic eight ball says:
16:10:23 <samyak-jn> #info chair 2021-05-20 -  nirik
16:10:28 <siddharthvipul> whatever date it is, I volunteer :D
16:10:31 <samyak-jn> #info chair 2021-05-27 - dtometzki
16:10:35 <samyak-jn> #info chair 2021-06-03 -???
16:10:42 <siddharthvipul> samyak-jn: me me
16:10:45 <nirik> siddharthvipul: running off the cliff? :)
16:11:03 <samyak-jn> siddharthvipul: hahaha, we are packed for this month, you start june for us then! ;)
16:11:08 <siddharthvipul> nirik: oh I am compensating for missing this meeting for 2 times now :P
16:11:13 <siddharthvipul> samyak-jn: works
16:11:19 <samyak-jn> #info chair 2021-06-03 - siddharthvipul
16:11:28 <nirik> well, welcome back. :)
16:11:33 <siddharthvipul> wow, it's May already
16:11:40 <dtometzki> yeah
16:11:47 <samyak-jn> yea
16:11:49 <siddharthvipul> nirik:  ^-^ I had been trying to sleep on time.. so was missing a few meetings
16:12:12 <samyak-jn> So we can move forward to our next topic since we are pretty covered ;)
16:12:17 <nirik> +1
16:12:18 <samyak-jn> Is it ok for all?
16:12:33 <bodanel> sure
16:12:34 <t0xic0der> Sure
16:12:46 <siddharthvipul> +1
16:12:54 <samyak-jn> #topic announcements and information
16:13:02 <samyak-jn> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:13:08 <samyak-jn> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:13:14 <samyak-jn> #info work is being done on getting COPR proper powerPC systems
16:13:20 <samyak-jn> #info CPE is hiring! https://communityblog.fedoraproject.org/community-platform-engineering-is-hiring/
16:13:26 <samyak-jn> #info nest with Fedora CFP is open! https://communityblog.fedoraproject.org/announcing-dates-cfp-for-nest-with-fedora/ (Aug 5th-8th)
16:13:32 <samyak-jn> #info mobrien is away 2021-05-10 to 2021-06-20
16:13:53 <samyak-jn> Please add if you have any new announcements to make! :D
16:13:57 <siddharthvipul> #info CentOS Dojo happening, and we have a badge for that, register at: https://hopin.com/events/centos-dojo-online-may-2021
16:14:03 <siddharthvipul> I have another one
16:14:56 <siddharthvipul> fetching link, just a sec
16:15:13 <mobrien> .hi
16:15:14 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
16:15:20 <mobrien> Sorry I'm late
16:15:23 <samyak-jn> mobrien[m]: hiya!
16:15:29 <nirik> hey! You're supposed to be parenting. ;)
16:15:41 <siddharthvipul> #info Fedora Council video meeting today on fedora server revitalization https://fedoraproject.org/wiki/Council/Video_Meetings#Fedora_Council_Video_Meetings
16:15:50 <siddharthvipul> oh oh, another one.. let me fetch another link
16:16:20 <mobrien> nirik they are never too young to learn about Fedora
16:16:49 <siddharthvipul> #info Fedora will be present in Linux App Summit and join us for office hour @1300 UTC saturday
16:17:02 <siddharthvipul> p.s. we have a badge for this ;)
16:17:05 <nirik> good point!
16:17:28 <t0xic0der> siddharthvipul: Drop a Hoppin link, will you please?
16:17:33 <t0xic0der> ;-)
16:17:49 <samyak-jn> <siddharthvipul "#info Fedora will be present in "> Nice, i guess we had to register for it right, needs to be done from my side!
16:18:01 <samyak-jn> t0xic0der: haha +1
16:18:30 <siddharthvipul> t0xic0der: LAS is not on hopin
16:18:41 <siddharthvipul> for CentOS dojo, you can see the hopin link in announcement :D
16:18:53 <siddharthvipul> let me fetch register link though
16:19:02 <t0xic0der> Yep, that would be helpful. Thanks.
16:19:19 <siddharthvipul> #link https://linuxappsummit.org/register/
16:19:19 <jednorozec> t0xic0der, https://app.hopin.com/events/centos-dojo-online-may-2021
16:20:10 <t0xic0der> jednorozec: Thanks. I am attending CentOS Dojo but it was LAS´s link that I was looking for. :)
16:20:18 <jednorozec> oh sorry :D
16:20:42 <siddharthvipul> I can't think of anything else to announce :)
16:20:50 <samyak-jn> Thanks for the announcements.
16:20:50 <samyak-jn> Any additional announcements or information, if someone like to?
16:21:07 <samyak-jn> if not we move forward in 1 min..
16:21:47 * siddharthvipul will be back in a minute
16:22:08 <samyak-jn> #topic Oncall
16:22:16 <samyak-jn> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:22:25 <samyak-jn> #info dtometzki is on call for 2021-05-06 to 2021-05-13
16:22:32 <samyak-jn> #info darknao is on call for 2021-05-13 to 2021-05-20
16:22:39 <samyak-jn> #info bodanel is on call for 2021-05-20 to 2021-05-27
16:22:48 <samyak-jn> #info ??? is on call for 2021-05-27 to 2021-06-03
16:22:50 <dtometzki> there was only one request about a full mail queue
16:23:07 <samyak-jn> dtometzki, always on time! Thanks.
16:23:49 <samyak-jn> Any other or additional information from other peoples?
16:24:10 <siddharthvipul> samyak-jn: you can add me for oncall duty
16:24:25 <samyak-jn> Here you go!
16:24:34 <samyak-jn> #info siddharthvipul  is on call for 2021-05-27 to 2021-06-03
16:25:07 <samyak-jn> I think we can move forward since we are packed for on calls as well ;)
16:25:23 <nirik> yep
16:25:26 <samyak-jn> #topic Monitoring discussion [nirik]
16:25:31 <samyak-jn> #info https://nagios.fedoraproject.org/nagios
16:25:37 <samyak-jn> #info Go over existing out items and fix
16:25:47 <samyak-jn> nirik, stage is yours :P
16:25:49 <nirik> so, again not much new here.
16:26:06 <nirik> there's a few things we could try and clean up, but haven't yet...
16:26:35 <nirik> no real new alerts tho, so I guess just move on for now
16:26:39 <samyak-jn> Ok thanks, so we can move forward then? If that's okay?
16:26:42 <samyak-jn> Sure!
16:26:49 <samyak-jn> #topic Learning topic
16:27:00 <siddharthvipul> I am excited for upcoming monitoring changes
16:27:02 <samyak-jn> #info ansible how it works and how to contribute [nirik] on 2021-05-13
16:27:03 <siddharthvipul> (tools)
16:27:24 <siddharthvipul> oh nice topic
16:27:31 <samyak-jn> Here you go again, I'm excited for today! nirik
16:27:59 <nirik> cool.
16:28:16 <nirik> So, we have been using ansible for many years (since it's very earliest days)
16:28:49 <nirik> We have a git repo with all our public ansible playbooks/roles/files/etc
16:29:00 <nirik> thats at https://pagure.io/fedora-infra/ansible
16:29:26 <nirik> We also have a private git repo that has passwords and sensitive info in it. Thats only on batcave01 (our ansible control host)
16:30:09 <nirik> the public repo syncs from pagure.io to batcave01 where it's merged with the private repo and thats where playbooks run
16:30:28 <dtometzki> can we use the public ansible repos or we need more permissions ?
16:30:39 <nirik> We have a small wrapper called 'rbac-playbook' that lets us delegate permissions to groups to run playbooks.
16:31:19 <nirik> The public ansible repo is committable by anyone in the list for that repo on pagure.io... but it's mostly the 'sysadmin' account group.
16:31:30 <nirik> anyone can submit Pull requests to the public repo.
16:32:05 <nirik> only folks in a sysadmin-* group can login to batcave01, and only specific groups can run specific playbooks.
16:32:21 <nirik> for example, sysadmin-badges can run badges related playbooks there
16:32:59 <nirik> In addition to running playbooks manually, we have a daily job that runs over all playbooks, but with --check and --diff... so it doesnt make changes, it just reports if changes would be made.
16:33:27 <nirik> and also often I run the 'main' playbook, which you can see in the repo on the top level... it just includes all the other playbooks.
16:34:10 <nirik> batcave01 (our control host) can reach all our other hosts to configure them via ansible...
16:34:36 <samyak-jn> <nirik "and also often I run the 'main' "> https://pagure.io/fedora-infra/ansible/blob/main/f/main.yml this one correct?
16:34:44 <nirik> Looking at the ansible repo, we have a README that tries to explain the layout...
16:34:53 <nirik> samyak-jn: yep. exactly.
16:35:12 <samyak-jn> nice.., please go on.
16:35:21 <nirik> So the main areas to look at are:
16:36:08 <nirik> inventory/ this is the list of all our hosts. It's split out into several files... but you can see if a host exists and what groups it's in there. groups are [groupname] and hosts are under each group. Hosts can be in multiple groups.
16:36:40 <dtometzki> do we have a staging area where people can develop and test ?
16:36:42 <nirik> inventory/host_vars and inventory/group_vars: These are variable assosciated with groups and hosts...
16:37:20 <nirik> dtometzki: great question. we do have a staging env, but it uses the same ansible repo and such as production. This is to allow them to be very close to the same all the time.
16:37:43 <nirik> for just general poking around, it's easy to spin up a container or vm locally and play with ansible.
16:38:25 <nirik> group vars are less specific than host vars and host vars override if something is defined in both places...
16:38:34 <dtometzki> yeah that is now my way
16:38:53 <nirik> The other two really commonly used/looked at directories are:
16:39:09 <nirik> playbooks - the actual playbooks you run... these include roles and tasks.
16:39:41 <nirik> roles - this is where most of the heavy lifting happens. These contain the config files/templates and tasks to setup things.
16:40:41 <nirik> In addition, ansible now has 'collections' which are a higher than roles level, generic way to distribute things. We have a few of those installed, in particular linux-system-roles and ansible-collections-community-general
16:41:39 <nirik> so, if we have time (which I think we do), lets look at a playbook?
16:41:47 <samyak-jn> Can you bit elaborate the `collections` part?
16:41:54 <nirik> playbooks/groups/batcave.yml is the playbook that sets up batcave01 (the control host itself. ;)
16:41:59 <samyak-jn> <nirik "so, if we have time (which I thi"> I'm inn! yes please
16:43:02 <nirik> samyak-jn: sure! collections are just a way to distribute ansible code so that others can easily reuse it. For example, the ansible-collections-community-general has a ton of things in it, but to pick one:
16:43:25 <nirik> /usr/share/ansible/collections/ansible_collections/community/general/plugins/modules/pam_limits.py
16:43:39 <nirik> this is a ansible module that lets you adjust/manage the pam_limits files.
16:44:01 <nirik> oh, and I left out something super handy!
16:44:11 <samyak-jn> Oh gotcha, i understood it now
16:44:29 <nirik> ansible includes docs for things pretty well... 'ansible-doc pam_limits' will give you detailed docs on how to use that module
16:44:52 <nirik> I use ansible-doc all the time.
16:45:10 <nirik> anyhow, back to the batcave01 playbook.
16:45:23 <nirik> If you look at the top the first thing we do is import _another_ playbook. ;)
16:45:52 <nirik> that playbook ( virt-create) gets passed a variable called 'myhosts' set to 'batcave'
16:46:24 <nirik> that playbook then runs and looks to make sure the virtual host exists. If it doesn't it uses host and group variables to create it with virt-install.
16:46:48 <nirik> after that (which gets skipped anytime the host actually already exists)
16:47:13 <nirik> you can see we have a play that sets up some variable files and then starts calling roles.
16:47:43 <nirik> down below we have 'pre_tasks' and 'tasks' and 'handlers'
16:48:03 <nirik> tasks are usually something quick / easy that it wasn't worth making a role out of
16:48:25 <nirik> in this case there's one to setup yum repos and one to setup the motd (message of the day that you see when you login)
16:48:54 <nirik> handlers are things that trigger on changes. You can have a handler that restarts something if it's config file changes, etc.
16:49:13 <nirik> you can see in roles, that we can also pass roles variables
16:49:43 <nirik> ansible uses jinja2 for templating, so variables are of the form "{{ variablename }}"
16:50:15 <nirik> so any questions on that?
16:50:18 <dtometzki> do you have a good link to jinja2 ?
16:50:22 <t0xic0der> Templating is, hands down, what makes the playbooks convenient!
16:51:02 <dtometzki> tutorial or examples ?
16:51:37 <nirik> https://ttl255.com/jinja2-tutorial-part-1-introduction-and-variable-substitution/ is a pretty reasonable tutorial
16:51:48 <samyak-jn> So just to be clear once, is something is being changed then handler gets into account and performs the desired role for example here: restart_services will run?
16:51:58 <samyak-jn> s/is/if
16:52:16 <t0xic0der> The official documentation is good too, https://docs.ansible.com/ansible/latest/user_guide/playbooks_templating.html but lesser examples here. :/
16:52:33 <nirik> samyak-jn: yeah, so if you look at the handlers file there they are defined with a name then a command to run on change...
16:52:46 <samyak-jn> yes
16:54:45 <nirik> so, in addition to playbooks and using ansible to configure things, we also use ansible as a ad-hock controller... that does require you to be in the sysadmin-main group to have that access, but it's handy to run commands over groups of hosts easily.
16:55:45 <nirik> you may want to play around with that on your home networks/machines/containers. ;)
16:56:17 <nirik> ok, so I think thats a pretty high level overview. :) Happy to answer further questions on the list/out of meeting/etc.
16:56:28 <samyak-jn> Yesss, so one ore questions, are there new comers task right now that I can look after the meeting to contribute to?
16:56:58 <samyak-jn> I just learned using ansible in couple of last week only so still learning :)
16:57:12 <nirik> samyak-jn: yeah, I think there's a few marked easyfix in the infra tickets list...
16:57:22 <nirik> you can submit PR's for them
16:57:54 <samyak-jn> nirik, I'll take a look and ping on the group for help ;) Thanks for today's topic it was very much awesome!
16:58:04 <mobrien> One thing to mention is that we have some low precedent vars set in places like group_vars/all.yml and there is one for staging too. If there is ever a variable that is not as you expect they may be the culprit
16:58:42 <bodanel> samyak-j: you can help me with my ticket
16:58:42 <mobrien> I mention it because it confused me last week :)
16:59:05 <bodanel> there are over 200 servers to migrate to rhel-system-roles
16:59:08 <bodanel> plenty of work
16:59:30 <samyak-jn> bodanel: Oh will ping you after the meeting, thanks for this, it will help me brush up!
17:00:05 <samyak-jn> If there are no further questions can we move on to next topic?
17:00:11 <samyak-jn> nirik++
17:00:16 <siddharthvipul> we are over time :)
17:00:21 <samyak-jn> Thanks for the session.
17:00:34 <samyak-jn> siddharthvipul: yea :3
17:00:37 <siddharthvipul> I think we can obviously continue in -admin :)
17:00:41 <dtometzki> thanks nirik
17:00:48 <dtometzki> nirik++
17:00:48 <zodbot> dtometzki: Karma for kevin changed to 8 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:00:54 <t0xic0der> nirik++
17:00:54 <zodbot> t0xic0der: Karma for kevin changed to 9 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:00:55 <siddharthvipul> nirik++ ftw
17:00:57 <zodbot> siddharthvipul: Karma for kevin changed to 10 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:01:03 <nirik> no problem. :)
17:01:25 <bodanel> nirik++
17:01:25 <zodbot> bodanel: Karma for kevin changed to 11 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:01:26 * nirik has to run to another meeting, thanks everyon
17:01:31 <siddharthvipul> nirik: \o
17:01:40 <siddharthvipul> bye everyone
17:01:54 <samyak-jn> #endmeeting