16:00:29 <eddiejenningsjr> #startmeeting Infrastructure (2023-04-13)
16:00:29 <zodbot> Meeting started Thu Apr 13 16:00:29 2023 UTC.
16:00:29 <zodbot> This meeting is logged and archived in a public location.
16:00:29 <zodbot> The chair is eddiejenningsjr. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:29 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:29 <zodbot> The meeting name has been set to 'infrastructure_(2023-04-13)'
16:00:29 <eddiejenningsjr> #meetingname infrastructure
16:00:29 <zodbot> The meeting name has been set to 'infrastructure'
16:00:29 <eddiejenningsjr> #chair nirik zlopez nb bodanel dtometzki jnsamyak
16:00:29 <zodbot> Current chairs: bodanel dtometzki eddiejenningsjr jnsamyak nb nirik zlopez
16:00:29 <eddiejenningsjr> #info Agenda is at: https://board.net/p/fedora-infra
16:00:30 <eddiejenningsjr> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:30 <eddiejenningsjr> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:00:30 <eddiejenningsjr> #topic greetings!
16:00:48 <eddiejenningsjr> .hello eddiejennings
16:00:49 <zodbot> eddiejenningsjr: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:00:51 <nirik> morning
16:00:57 <eddiejenningsjr> Good morning all!
16:01:17 <dherrera> hi!
16:01:22 <aheath1992> .hi
16:01:24 <zodbot> aheath1992: aheath1992 'Andrew Heath' <aheath1992@gmail.com>
16:01:26 <jnsamyak> .hello2 jnsamyak
16:01:27 <zodbot> jnsamyak: jnsamyak 'Samyak Jain' <samyak.jn11@gmail.com>
16:01:44 * eddiejenningsjr takes a break from testing Ansible at work to chair the meeting.
16:01:45 <eddiejenningsjr> :D
16:02:37 <patrikp[m]> Hello.
16:02:59 <eddiejenningsjr> How's everyone today?
16:03:34 <aheath1992> So far so good, Ansible & NBDE work
16:04:13 <eddiejenningsjr> #topic New folks introductions
16:04:13 <eddiejenningsjr> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:04:13 <eddiejenningsjr> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:14 <nils> .hello nphilipp
16:04:15 <zodbot> nils: nphilipp 'Nils Philippsen' <nphilipp@redhat.com>
16:04:34 <eddiejenningsjr> Do we have any new folks with us today?  If so, make yourself be known and tell us something about you :)
16:04:38 <eddiejenningsjr> We won't bite.  I promise :D
16:05:31 <eddiejenningsjr> I thought I saw some E-mails from new folk, but that may have been last week.
16:07:21 <eddiejenningsjr> Seems like we don't have any new folk today.  So let's move on.
16:07:38 <eddiejenningsjr> #topic Next chair
16:07:38 <eddiejenningsjr> #info magic eight ball says:
16:07:38 <eddiejenningsjr> #info chair 2023-04-13 - eddiejenningsjr
16:07:39 <eddiejenningsjr> #info chair 2023-04-20 - jnsamyak
16:07:39 <eddiejenningsjr> #info chair 2023-04-27 - ???
16:07:55 <eddiejenningsjr> We're set for next week.  Do we have a volunteer for 2023-04-27?
16:08:24 <eddiejenningsjr> For anyone here who hasn't chaired before, it's a low stress job and a fun way to be involved -- even if you're like me and have little time.
16:09:43 <eddiejenningsjr> I'm not in IRC today.  Is the Matrix chat in sync?
16:09:57 <phsmoura> I can take it
16:10:02 <nils> eddiejenningsjr, I’m on IRC and I can read you
16:10:13 <eddiejenningsjr> @nils Thanks :D
16:10:19 <nils> phsmoura++
16:10:19 <zodbot> nils: Karma for phsmoura changed to 1 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:10:30 <eddiejenningsjr> and excellent, sold to phsmoura
16:10:51 <eddiejenningsjr> #info chair 2023-04-27 - phsmoura
16:11:11 <eddiejenningsjr> Anyone interested in 2023-05-04, Star Wars day?
16:11:18 <eddiejenningsjr> Otherwise, we're set for a couple of weeks.
16:12:58 <eddiejenningsjr> Looks like we'll handle May later on.  :)  Moving along
16:13:24 <eddiejenningsjr> #topic announcements and information
16:13:24 <eddiejenningsjr> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:13:24 <eddiejenningsjr> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:13:24 <eddiejenningsjr> #info Fedora 38 final freeze is in effect
16:13:35 <eddiejenningsjr> I removed the Easter announcement.  Any new announcements?
16:14:05 <nirik> #info f38 first go/nogo is right after this meeting. ;)
16:14:54 <eddiejenningsjr> Added.  Any others?
16:16:24 <eddiejenningsjr> Hearing none, we'll trek onward
16:17:02 <eddiejenningsjr> And now for everyone's favorite topic. . .
16:17:04 <nils> [Star Wars reference ✅, Star Trek refernce ✅]
16:17:08 <eddiejenningsjr> #topic Oncall
16:17:08 <eddiejenningsjr> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:17:08 <eddiejenningsjr> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:17:08 <eddiejenningsjr> ## .oncalltakeeu .oncalltakeus
16:17:08 <eddiejenningsjr> #info nirik is on call from 2023-04-06 to 2023-04-13
16:17:08 <eddiejenningsjr> #info eddiejenningsjr is on call from 2023-04-14 to 2023-04-20
16:17:08 <eddiejenningsjr> #info jednorozec is on call from 2023-04-21 to 2023-04-27
16:17:14 <eddiejenningsjr> .oncalltakeus
16:17:14 <zodbot> eddiejenningsjr: Kneel before zod!
16:18:27 <eddiejenningsjr> On-call is another great way to be involved.  For an apprentice like me, I cannot do administrative actions on systems, but my function for on-call is to see if my summoning is something that needs to have someone like Nirik pinged, or simply have a ticket put in
16:18:59 <eddiejenningsjr> Also, if there is an event involving the sysadmins, the on-call person placates the masses in chat by informing them that "X is being worked on"
16:19:47 <eddiejenningsjr> That being said, is anyone interested in in 2023-04-28 to 2023-05-04?
16:20:54 <eddiejenningsjr> If you use Matrix, on-call is quite conventient, as you can get a push on a mobile device should you happen to be pinged.
16:22:24 <eddiejenningsjr> No takers for this week it seems.  Be thinking about it.  If you're "on the fence," give it a try :)
16:22:38 <eddiejenningsjr> And now a word from this past week's oncall :)
16:22:40 <eddiejenningsjr> #info Summary of last week: (from current oncall )
16:23:16 <nirik> So there were 3 oncall pings...
16:23:42 <nirik> 1 was an auth outage... the ping was when I was asleep, but I fixed it as soon as I got up. A ticket was filed on it.
16:23:58 <nirik> 2. was a proxy/network issue... it cleared up before I got up. no ticket filed.
16:24:03 <nirik> sorry, thats 2.
16:24:33 <eddiejenningsjr> Nirik is the master sysadmin.  He fixes stuff while alseep :D
16:24:36 <nirik> 3. was zodbot not being able to do fas / hi / hellows again... Saffronique fixed it. no ticket.
16:24:47 <nirik> man, that was 3... what the heck
16:25:04 <nirik> 3. Is matrix doing this, or can I just not type?
16:25:10 <nirik> yes, it's matrix. ;)
16:25:40 <eddiejenningsjr> I though it was  reference to the three most common problems in IT:  naming conventions and off-by-one errors.
16:25:41 <nirik> it's displaying '1' for each of those.
16:25:49 <smooge> yeah md mode
16:25:57 <nirik> (but irc sees the real number over the bridge. ok, whatever)
16:26:10 <smooge> it will block all those 1.s into one line unless you put all that in a ``` ``` set
16:26:20 <eddiejenningsjr> Good to know
16:26:36 <nirik> it's not doing one line... I am sending "3. something" and it's displaying "1. something"
16:26:54 * eddiejenningsjr shrugs.
16:26:59 <nirik> but we can poke at that out of meeting
16:27:18 <nils> > yeah md mode
16:27:24 <nils> tools trying to be clever, eh?
16:27:29 <eddiejenningsjr> Thank you for the on-call report :).  And continuing with you. . .
16:27:31 <eddiejenningsjr> #topic Monitoring discussion [nirik]
16:27:31 <eddiejenningsjr> #info https://nagios.fedoraproject.org/nagios
16:27:31 <eddiejenningsjr> #info Go over existing out items and fix
16:27:39 <nirik> nils: seems so.
16:28:13 <nirik> so, nothing new here from last week, aside that aheath1992 submitted a PR to fix/drop one of the checks we no longer need. I'm not sure if he's run the playbook to deploy it yet.
16:28:30 <aheath1992> I have and the check i gone
16:28:32 <nirik> thanks for looking into those alerts aheath1992
16:28:39 <nirik> cool!
16:29:20 <nirik> thats all I had for this section this week.
16:29:44 <eddiejenningsjr> Excellent.
16:30:09 <eddiejenningsjr> This week is a learning topic week.  We'll return to the ticket log next week.
16:30:53 <eddiejenningsjr> Before I get started on this week's topic.  Do we have any volunteers for the next learning topic, that would be on 2023-04-27?
16:31:20 <nirik> FYI, I won't be at the meeting next week...
16:31:50 <eddiejenningsjr> We're all doomed!  . . .  I mean, that's fine.  Enjoy the meeting time off :)
16:32:39 <eddiejenningsjr> I know the idea was floated a little while back to just have a "sysadmin war stories" sharing time instead of a topic.
16:32:39 <nirik> well, it's jury duty, so... not sure how much I will enjoy it.
16:32:53 <eddiejenningsjr> I'm 0/3 for jury duty selection :(.  I want to be on one.
16:33:31 <nirik> I think a sysadmin war stories/times when things blew up might be nice...
16:34:10 <eddiejenningsjr> Any objections from the audience for sysadmin story time on 2023-04-27?
16:34:13 <aheath1992> +1 love a good syadmin war story
16:34:55 <eddiejenningsjr> It's settled. :)
16:34:56 <eddiejenningsjr> #topic Sysadmin War Story Discussion [all_of_us] on 2023-04-27
16:35:13 <eddiejenningsjr> And now for #topic Wireguard basics [eddiejenningsjr] on 2023-04-13
16:36:24 <eddiejenningsjr> It won't be possible for me to go over everything there is to know about Wireguard (as I don't know everything there is to know ;).  So I'm starting with some useful links.
16:36:24 <eddiejenningsjr> https://www.wireguard.com/
16:36:24 <eddiejenningsjr> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/assembly_setting-up-a-wireguard-vpn_configuring-and-managing-networking
16:37:08 <eddiejenningsjr> Wireguard is a resource efficient, highly performant VPN solution that's baked right into the Linux kernel.
16:38:55 <eddiejenningsjr> It is possible to create several VPN solutions using Wireguard without the need to install agents on Linux systems.  Also, it does have clients for use on Windows, Mac, and other OSes.  And many modern hardware and software routers support it.
16:38:55 <eddiejenningsjr> https://www.wireguard.com/install/
16:40:29 <eddiejenningsjr> I'm going to describe the basics of getting two nodes connected with Wireguard (point-to-point), and then describe some other possible VPN architectures.
16:41:25 <eddiejenningsjr> Although Wireguard is part of the kernel, you will want to install the wireguard-tools package, which gives you access to the userspace command "wg", which you'll use.
16:42:47 <eddiejenningsjr> To get started, first and foremost, each participating node must generate a public/private keypair for Wireguard.
16:43:35 <eddiejenningsjr> In the Wireguard documentation, there is information about "cryptokey routing," which is the special sauce that Wireguard uses to know where to send traffic it receives, and it's based on the peers you configure and their public keys.
16:44:02 <eddiejenningsjr> The method I like to use to generate these keys is the following command wg genkey | tee private.key | wg pubkey > public.key
16:44:44 <eddiejenningsjr> wg genkey creates a private key, which is then piped to tee so it can be output to a file, and the stdout is piped to wg pubkey so the other half of the keypair is created, which is finally output to a file.
16:46:40 <eddiejenningsjr> Now from here there are two ways to actually create the Wireguard interface, which is basically a network interface that has the following properties:
16:46:40 <eddiejenningsjr> - your private key
16:46:40 <eddiejenningsjr> - a peer's public key
16:46:40 <eddiejenningsjr> - a peer's non-VPN IP address
16:46:40 <eddiejenningsjr> - a list of allowed IPs and subnets for traffic to traverse this interface
16:48:03 <eddiejenningsjr> Many folks use the wg-quick command and its associated configuration file (part of the wireguard-tools package) to do this.  wg-quick basically does just-in-time configurations using the ip route2 commands to create and tear-down an interface / connection and configure your firewall.
16:49:02 <eddiejenningsjr> When I learned, since I'm on Fedora, I used nmcli and Network Manager interface configuration files.  See my shameless plug for those steps:  https://youtu.be/oJRhqHZb8eM
16:50:32 <eddiejenningsjr> Either way you choose, the result is the same.  1. Create the wireguard interface with the above properties (and specify the port that Wireguard will be listening on). 2. Configure the firewall to allow traffic.  3.  If needed configure routing.
16:50:51 <eddiejenningsjr> I did want to touch on two "gotchas" before I finish, from my experience in testing.
16:51:18 <eddiejenningsjr> First is specifying "allowed ips" on your Wireguard interfaces.
16:52:14 <eddiejenningsjr> Let's say you have wg0 as your interface.  For traffic leaving that interface, the destination IP must match whatever is in "allowed-ips."  For traffic entering that interface (coming from your peer, or elsewhere), teh source IP must match "allowed-ips"
16:52:33 <eddiejenningsjr> Making a mistake with allowed-ips has caused me some heartache.
16:54:35 <eddiejenningsjr> The last "gotcha" is default routing.  From what I've seen NetworkManager and wg-quick will handle this for you when you bring up the interface.  The gotcha comes in with running ip route and not seeing anything for Wireguard.  The routing for Wireguard to your actual default gateway is handled by policy based routing, which is beyond my knowledge to explain.
16:55:06 <eddiejenningsjr> I know that's a ton of information, but it's enough to get you started and show you where to look for answers. :)
16:55:24 <eddiejenningsjr> In our last 5 minutes are there any questions?
16:55:44 <nils> first thanks for all the info, eddiejenningsjr++
16:56:07 <nirik> yeah, good stuff.
16:56:41 <nirik> worth noting that we use openvpn in fedora infra, but switching to wireguard might be something we do when the last of the rhel7/8 instances are gone. :)
16:57:33 <eddiejenningsjr> I use it in two ways:  On my laptop, when I turn on the interface all Internet bound traffic goes through the tunnel (to a Linode VPS).  Also on that same Linode VPS, I have a tunnel to an nginx VM at home acting as a reverse proxy, so I can use the Linode public IP to expose home stuff to the Internet if I want.
16:58:12 <eddiejenningsjr> I've never used openVPN, but from everything I've heard / read, it seems like Wireguard is more lightweight and better performing.
16:59:12 <eddiejenningsjr> Any other questions, comments in our final two minutes? :)
16:59:28 <nirik> wireguard is likely... but it does need a kernel module. ;)
16:59:54 <eddiejenningsjr> Yeah, I think RHEL 9 is the first to have it baked in
17:00:18 <eddiejenningsjr> We've reached the hour.  Thank you for attending everyone!
17:00:20 <eddiejenningsjr> #endmeeting