16:01:29 <eddiejenningsjr> #startmeeting Infrastructure (2023-07-27)
16:01:29 <zodbot> Meeting started Thu Jul 27 16:01:29 2023 UTC.
16:01:29 <zodbot> This meeting is logged and archived in a public location.
16:01:29 <zodbot> The chair is eddiejenningsjr. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:01:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:01:29 <zodbot> The meeting name has been set to 'infrastructure_(2023-07-27)'
16:01:38 <eddiejenningsjr> #meetingname infrastructure
16:01:38 <zodbot> The meeting name has been set to 'infrastructure'
16:01:38 <eddiejenningsjr> #chair nirik zlopez nb bodanel dtometzki jnsamyak lenkaseg
16:01:38 <eddiejenningsjr> #info Agenda is at: https://board.net/p/fedora-infra
16:01:38 <zodbot> Current chairs: bodanel dtometzki eddiejenningsjr jnsamyak lenkaseg nb nirik zlopez
16:01:38 <eddiejenningsjr> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:01:38 <eddiejenningsjr> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:01:39 <eddiejenningsjr> #topic greetings!
16:01:44 <eddiejenningsjr> .hello eddiejennings
16:01:45 <zodbot> eddiejenningsjr: Something blew up, please try again
16:01:48 <zodbot> eddiejenningsjr: An error has occurred and has been logged. Please contact this bot's administrator for more information.
16:01:50 <nirik> morning
16:03:22 <nirik> zodbot: reload Fedora
16:03:22 <eddiejenningsjr> Oh noes!  I broke Zod!
16:03:22 <pcreech[m]> mornin
16:03:22 <nirik> nope, long standing plugin bug...
16:03:22 <nirik> it will be back working in a min
16:03:22 <eddiejenningsjr> Time appropriate greetings, all! :)
16:03:22 <mkonecny> .hello zlopez
16:03:25 <zodbot> nirik: Kneel before zod!
16:03:28 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@pacse.eu>
16:03:32 <darknao> .hi
16:03:34 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:04:10 <eddiejenningsjr> Welcome to the weekly Fedora Infra meeting!  I assure you it will be 100% more enjoyable than troubleshooting SuSE update problems.
16:04:58 <pcreech[m]> Is that garuntee on the box?
16:05:06 <eddiejenningsjr> Let's start the party!
16:05:07 <phsmoura> .hi
16:05:08 <zodbot> phsmoura: phsmoura 'Pedro Moura' <pmoura@redhat.com>
16:05:14 <eddiejenningsjr> #topic New folks introductions
16:05:14 <eddiejenningsjr> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:05:14 <eddiejenningsjr> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:05:55 <eddiejenningsjr> pcreech: Yes.  Also in the Fedora ToS somewhere:  meetings are 100% more enjoyable than troubleshooting SuSE ;)
16:06:17 <eddiejenningsjr> Do we have anyone new with us today?  If so, let yourself be known!
16:06:26 <eddiejenningsjr> We won't bite!  We're a friendly bunch!
16:08:42 <eddiejenningsjr> Seeing none, let's move along.
16:08:58 <eddiejenningsjr> #topic Next chair
16:08:58 <eddiejenningsjr> #info magic eight ball says:
16:08:58 <eddiejenningsjr> #info chair 2023-07-27 - eddiejennings
16:08:58 <eddiejenningsjr> #info chair 2023-08-03 - cancelled for flock
16:08:58 <eddiejenningsjr> #info chair 2023-08-10 - pcreech
16:09:53 <eddiejenningsjr> Do we have any volunteers for 2023-08-17?
16:10:24 <eddiejenningsjr> It's a bit far out there date wise, but we are skipping next week.
16:10:44 <phsmoura> me
16:10:46 * nirik might not be around that day... its my b-day. ;)
16:10:49 <eddiejenningsjr> Chairing is a fun way to be involved with the Infra Team.  You get to run the show for this meeting :D
16:11:20 <eddiejenningsjr> Ah, nirik is a summer child.  I'm a winter kid :D
16:11:28 <eddiejenningsjr> and sold to phsmoura !
16:12:05 <eddiejenningsjr> #info chair 2023-08-17 phsmoura
16:12:32 <eddiejenningsjr> Anyone for 2023-08-24?  If not, we're set for a few weeks.
16:13:44 <eddiejenningsjr> Seeing none, let's move along!  I'm not on IRC today.  How's the bridge looking?
16:14:22 <nirik> its rusted, but staying up...
16:15:05 <eddiejenningsjr> Excellent. . . I mean, good. :)
16:15:17 <eddiejenningsjr> #topic announcements and information
16:15:17 <eddiejenningsjr> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 0730 UTC in #centos-meeting
16:15:17 <eddiejenningsjr> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:15:17 <eddiejenningsjr> #info flock to fedora coming up in early aug (2nd-4th in cork, ie) https://flocktofedora.org
16:15:17 <eddiejenningsjr> #info This meeting is cancelled during Flock
16:15:36 <eddiejenningsjr> And one other announcement, congrats to our new sysadmin-main! :D
16:16:46 <nirik> ;) yeah! congrats...
16:16:55 <mkonecny> Congrats!
16:17:03 <darknao> \o/ thanks!
16:17:08 <eddiejenningsjr> darknao: I won't call for a speech :D
16:17:22 <eddiejenningsjr> Any other announcements?
16:17:41 <darknao> I'm trying very hard to not break anything :)
16:18:11 <nirik> so are we all. ;)
16:19:06 <eddiejenningsjr> Looks like there are no other announcements.  Let's keep going.
16:19:24 <eddiejenningsjr> #topic Oncall
16:19:24 <eddiejenningsjr> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:19:24 <eddiejenningsjr> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:19:24 <eddiejenningsjr> ## .oncalltakeeu .oncalltakeus
16:19:24 <eddiejenningsjr> #info eddijenningsjr is on call from 2023-07-21 to 2023-07-27
16:19:24 <eddiejenningsjr> #info darknao is on call from 2023-07-28 to 2023-08-03
16:19:52 <darknao> .oncalltakeeu
16:19:52 <zodbot> darknao: Kneel before zod!
16:21:03 <eddiejenningsjr> On-call is another fun way to be involved with the Infra team.  You'll be the front line to acknowledge issues and either resolve it yourself (unlikely if an apprentice like me), ping a sysadmin-main person (unlikely unless the world's on fire), or tell the summoner to put in a ticket and we'll get to the issue soon (most likely).
16:21:51 <eddiejenningsjr> Do we have a volunteer for 2023-08-03 to 2023-08-10?
16:22:30 <eddiejenningsjr> I hear there may or may not be a bonus for being on-call during Flock ;)
16:24:07 <eddiejenningsjr> If no volunteers, I'll be willing to do another week, since I won't be traveling to Flock.
16:24:28 <pcreech[m]> My brain is screaming no... Which means I should probably say maybe/yes?
16:24:33 <nb> .hi
16:24:34 <zodbot> nb: nb 'Nick Bebout' <nick@bebout.net>
16:24:36 <nirik> ha
16:24:44 * nb is not volunteering - just saying that I am here
16:24:44 <darknao> I'm not going either so I can keep the phone for one extra week
16:24:52 <eddiejenningsjr> It is far less intimidating than it sounds.
16:25:22 <eddiejenningsjr> Matrix makes it convenient, since you can get a mobile notification if you username is mentioned.  So you don't have to be on IRC all the time
16:26:31 <eddiejenningsjr> I'll take it.  New sysadmin-main needs a week off ;)
16:26:31 <pcreech[m]> Put me down, why not
16:26:39 <eddiejenningsjr> sold to pcreech
16:27:09 <eddiejenningsjr> #info pcreech is on call from 2023-08-03 to 2023-08-10
16:27:22 <eddiejenningsjr> You'll do fine :)
16:27:29 <eddiejenningsjr> #info Summary of last week: (from current oncall )
16:28:25 <eddiejenningsjr> So I was summoned twice last week.
16:28:25 <eddiejenningsjr> 1.  Issue related to the Apache config we discussed last week, that was resolved by nirik rolling back the change.
16:28:25 <eddiejenningsjr> 2.  Storage issue that happened outside of my hours, but looks like was resolved by nirik.
16:29:28 <nirik> cool.
16:29:32 <eddiejenningsjr> pcreech to give you some context, my previous two on-calls had zero pings and one ping respectively.
16:30:03 <eddiejenningsjr> And nirik, the floor is your's.
16:30:05 <eddiejenningsjr> #topic Monitoring discussion [nirik]
16:30:05 <eddiejenningsjr> #info https://nagios.fedoraproject.org/nagios
16:30:05 <eddiejenningsjr> #info Go over existing out items and fix
16:31:24 <nirik> not much new here I don't think...lets see
16:31:58 <nirik> yeah, about the same really.
16:32:05 <nirik> One thing to note...
16:32:27 <nirik> I realized/saw that noc02 is actually a rhel7 instance. ;( noc01 is rhel8. I will probibly redo them to 9 soon.
16:32:43 <nirik> (although zabbix is coming)
16:33:14 <eddiejenningsjr> At least rhel 7 isn't end of life yet ;)
16:33:47 <nirik> yet.
16:33:49 <mkonecny> The zabbix is currently on hold, because David Kirwan is busy with CentOS
16:34:22 <nirik> sure, but its the eventual plan
16:35:19 <nirik> it's hard to retire infrastructure... see the recent article about IMAX movies using a emulated palm pilot to run the film loading.
16:35:55 <eddiejenningsjr> Yep.  I laughed aloud when I saw said article.
16:36:03 <nirik> anyhow, nothing more on monitoring that I can think of off hand
16:36:39 <eddiejenningsjr> Noted, and thank you for keeping watch over the Infra server flock :)
16:36:49 <eddiejenningsjr> #topic Learning topic
16:36:59 <eddiejenningsjr> #info Considerations and tips regarding PyPI vs Distribution Repo installs of Ansible [eddiejenningsjr] on 2023-07-27
16:37:52 <eddiejenningsjr> So this learning topic will be examining a case study, and questions during the discussion are welcome.
16:38:18 <eddiejenningsjr> And the study will be related to some challenges I've dealt with at my own work regarding Ansible.
16:39:37 <eddiejenningsjr> As the title suggests, AWX and Ansible Automation Platform are outside of the scope.  Maybe one day we'll use those tools, but when introducing Ansible to an environment and getting folks on board with the "Ansible way" of using Git and doing infra changes only through Ansible, I encourage beginning with Ansible Engine.
16:41:08 <eddiejenningsjr> So a brief recap.  Ansible Engine (CLI-only Ansible) is available as ansible-core (base install with minimal modules and no frills) and ansible (base installe + some community curated commonly used modules).
16:41:49 <eddiejenningsjr> Personally, I tend to use ansible-core and add modules as needed.  Both are valid ways of consuming Ansible.
16:42:44 <eddiejenningsjr> Most distributions maintain packages for both ansible-core and ansible in their repositories.  So dnf install ansible-core and you're good to go.
16:43:21 <eddiejenningsjr> For most server infrastructure uses of Ansible, using the distribution's package is 100% fine.
16:44:16 <eddiejenningsjr> One thing to keep in mind is you may not always get the most recent version of those packages.  Since the release is based on the repo maintainers, instead of the Ansible team.
16:44:52 <eddiejenningsjr> Generally this isn't a problem, but it's something to keep in mind depending on your requirements of how old you allow your packages to be.
16:45:27 <eddiejenningsjr> Alternatively, you can install ansible and ansible-core from PyPI, which would be using the Python pip command for installs.
16:46:26 <eddiejenningsjr> This method grants you much greater control of which version of Ansible you get, but comes at the tradeoff of you can no longer maintain the packages through your system's package manager.
16:47:15 <eddiejenningsjr> Also, you will want to utilize Python Virtual Environments when installing Ansible in this way, so there is a small learning curve of understanding how to utilize Python Virtual Environments.
16:48:18 <eddiejenningsjr> So what was the problem that sparked this debate?  In a few words. . . NETWORK TEAM
16:48:35 <eddiejenningsjr> It's always the network team, right?  Until it isn't ;)
16:48:57 <nirik> I thought it was always DNS. ;)
16:49:01 <eddiejenningsjr> Ha!
16:49:34 <eddiejenningsjr> My previous experiences with Ansible did not include managing switches and routers; so, this is new territory for me.
16:50:09 <eddiejenningsjr> And I learned the new-and-soon-to-be-THE-way way for Ansible to do its ssh connection to network devices is to use pylibssh.
16:50:35 <eddiejenningsjr> Well, this presents a problem -- at least for RHEL 8 users.
16:50:58 <eddiejenningsjr> To my knowledge, there is no pylibssh package available.
16:51:20 <eddiejenningsjr> Not to worry though!  Red Hat documentation tells us to just install it from PyPI.  Problem solved!
16:51:30 <eddiejenningsjr> Not exactly.
16:52:33 <eddiejenningsjr> When Ansible makes its ssh connection, even if you have PyPI libraries and such, it still looks to the System python libraries, so the result is if you're trying to use something that needs pylibssh, Ansible will fail saying it cannot find pylibssh.
16:53:10 <eddiejenningsjr> The only solution I've found for this is to not install Ansible via the distro repo, but rather install it from PyPI into a Python Virtual environment.
16:53:30 <mkonecny> Does adding new folders to PYTHONPATH helps?
16:53:55 <eddiejenningsjr> Then and only then will it look to the Python libraries in the virtual environment, and successfully make the pylibssh connections.
16:53:59 <nirik> someone should package that. (I'm not sure I have cycles to)
16:54:26 <eddiejenningsjr> I did not think to try PYTHONPATH.  I did try telling the Ansible interperter variable where to look for python, but that did not help.
16:55:08 <eddiejenningsjr> So when working with the Python virtual environment install of Ansible, the question becomes, how can we make this easy for users to use without always having to activate the virtual environment all the time.
16:56:12 <eddiejenningsjr> I got an idea from a Reddit answer to my query about this situation.  Within /usr/local/bin I have symlinks for the common Ansible commands pointing to the executables within the virtual environment.
16:57:04 <eddiejenningsjr> So far that has worked beautifully.  My network buddies just run ansible-playbook and don't worry that what's being run is /path/to/python/venv/ansible-playbook
16:57:17 <eddiejenningsjr> But there is one gotcha!
16:58:23 <eddiejenningsjr> For we systems people, when we use become, and the ansible.builtin.shell or the command module, we must use the full patch to the Ansible executable, if we wish to run an Ansible command.
16:59:09 <eddiejenningsjr> That's is because by default, sudo does not include usr/local/bin in its path, and I judge for how rarely I need to do the above, it's not worth changing the default path used by sudo.
17:00:02 <eddiejenningsjr> So for our final minute.  Both installs from distro and PyPI are valid.  If you use the PyPI way heed my experience above concerning gotchas.  AND if your network team is using Ansible Enging, keep in mind how you will deal with pylibssh (or the older paramiko packages).
17:01:29 <eddiejenningsjr> I feel like I'm out of breath, and we're a minute over.  But thanks for listening to the chat :).  Feel free to reach out to me in other channels or individually for questions.  I'm happy to share what little wisdom I have
17:02:11 <mkonecny> Thanks, this was nice guide
17:02:16 <mkonecny> Worth a blog post
17:02:26 * nirik has to go to fesco meeting. thanks Eddie Jennings, Jr.!
17:02:50 <eddiejenningsjr> #endmeeting