16:00:31 <mkonecny> #startmeeting Infrastructure (2023-02-02)
16:00:31 <zodbot> Meeting started Thu Feb  2 16:00:31 2023 UTC.
16:00:31 <zodbot> This meeting is logged and archived in a public location.
16:00:31 <zodbot> The chair is mkonecny. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:31 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:31 <zodbot> The meeting name has been set to 'infrastructure_(2023-02-02)'
16:00:31 <mkonecny> #meetingname infrastructure
16:00:31 <zodbot> The meeting name has been set to 'infrastructure'
16:00:31 <mkonecny> #chair nirik zlopez nb bodanel dtometzki jnsamyak
16:00:31 <mkonecny> #info Agenda is at: https://board.net/p/fedora-infra
16:00:31 <zodbot> Current chairs: bodanel dtometzki jnsamyak mkonecny nb nirik zlopez
16:00:32 <mkonecny> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:32 <mkonecny> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:00:32 <mkonecny> #topic greetings!
16:00:38 <mkonecny> Hi everyone
16:00:55 <nirik[m]> morning
16:00:58 <aheath1992> Morning
16:01:04 <darknao> .hi
16:01:06 <zodbot> darknao: darknao 'Francois Andrieu' <darknao@drkn.ninja>
16:01:26 <mkonecny> Welcome aboard of today's voyage
16:01:45 <mkonecny> I will be our captain for today
16:01:54 <phsmoura> .hi
16:01:55 <zodbot> phsmoura: phsmoura 'Pedro Moura' <pmoura@redhat.com>
16:02:04 <eddiejenningsjr> .hello eddiejennings
16:02:07 <zodbot> eddiejenningsjr: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:03:34 <mkonecny> So let's look if there is anyone new aboard this ship :-)
16:03:35 <mkonecny> #topic New folks introductions
16:03:35 <mkonecny> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:03:36 <mkonecny> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:03:38 <mkonecny> Don't be shy and introduce yourself
16:05:10 <eddiejenningsjr> We won't bite :)
16:05:32 <nirik[m]> we are a pretty friendly bunch. or try to be. :)
16:06:34 <mkonecny> It seems that there is nobody new today
16:06:35 <mkonecny> So let's look for the captain of next voyage
16:06:41 <mkonecny> #topic Next chair
16:06:41 <mkonecny> #info magic eight ball says:
16:06:41 <mkonecny> #info chair 2023-02-02 - mkonecny
16:06:42 <mkonecny> #info chair 2023-02-09 - dtometzki
16:06:42 <mkonecny> #info chair 2023-02-16- ?
16:07:19 <eddiejenningsjr> I can do 2023-02-16 :)
16:07:56 <mkonecny> Hear, hear, the captain for voyage on 2023-02-16 will be Eddie Jennings, Jr.
16:08:29 <mkonecny> #info chair 2023-02-16- eddiejennings
16:09:12 <mkonecny> Let's see if there is any volunteer for next one
16:09:14 <mkonecny> #info chair 2023-02-22 - ???
16:09:24 <mkonecny> Oh, sorry
16:09:27 <mkonecny> #info chair 2023-02-23 - ???
16:10:57 <nirik[m]> I guess I could take it if no one else... or we could just punt and fill it next week?
16:11:26 <mkonecny> I will leave it to next week, we already have captains for next two weeks assigned
16:12:20 <mkonecny> Let's check what the radio operator heard on radio
16:12:24 <mkonecny> #topic announcements and information
16:12:25 <mkonecny> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:12:25 <mkonecny> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:12:27 <mkonecny> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:12:27 <mkonecny> #info oncall should also handling #fedora-releng pings if possible
16:12:27 <mkonecny> #info F38 branching from rawhide on  2023-02-07
16:12:36 <mkonecny> Anything he missed?
16:13:24 <smooge> #info DNS problems with mail have been tracked and fixed ?
16:13:50 <smooge> mainly in case someone wants to know where their @fedoraproject.org email may have gone
16:14:25 <nirik[m]> #info bodhi is now refreshing critpath every day
16:14:56 <nirik[m]> #info Test monitoring/alerting in staging openshift cluster. Coming soon to more apps near you
16:17:03 <mkonecny> This seems that the radio operator really missed few messages
16:17:29 <nirik[m]> I think we missed sending them before hand. ;)
16:17:54 <smooge> yeah
16:18:08 <smooge> I had to come via a runner boat with this message
16:18:30 <mkonecny> Thanks for delivering them
16:19:00 <mkonecny> And now let's check who is appointed to the lookout post
16:19:09 <mkonecny> #topic Oncall
16:19:09 <mkonecny> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:19:09 <mkonecny> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:19:20 <mkonecny> #info darknao is on call from 2023-01-26 to 2023-02-02
16:19:20 <mkonecny> #info jednorozec is on call from 2023-02-02 to 2023-02-09
16:19:21 <mkonecny> #info mkonecny is on call from 2023-02-09 to 2023-02-16
16:19:40 <mkonecny> It seems that we are pretty full for next few weeks
16:20:03 <mkonecny> #info ??? is on call from 2023-02-17 to 2023-02-23
16:20:22 <mkonecny> Could we find a volunteer for the following week?
16:21:24 <mkonecny> If not, no worries we can fill this post in upcoming weeks
16:21:52 <dtometzki> .hi
16:21:53 <zodbot> dtometzki: dtometzki 'Damian Tometzki' <damian@riscv.tometzki.de>
16:22:04 <eddiejenningsjr> Alas, I'll be out of town much of that week.
16:22:08 <dtometzki> a little bit late :-(
16:22:42 <mkonecny> Let's keep it empty for now and fill the post in upcoming weeks
16:23:00 <mkonecny> And now what the current lookout saw in the distance
16:23:01 <mkonecny> #info Summary of last week: (from current oncall )
16:23:46 <mkonecny> Do we have darknao on board?
16:23:51 <darknao> only one ping last week
16:23:57 <darknao> Fri 01-27: login timeouts from id.fp-o. Was solved by rebooting ipsilon01.
16:24:03 <darknao> and that's it
16:24:47 <mkonecny> OK and now for the chief engineer to hear about the status of our engines
16:24:57 <mkonecny> #topic Monitoring discussion [nirik]
16:24:57 <mkonecny> #info https://nagios.fedoraproject.org/nagios
16:24:57 <mkonecny> #info Go over existing out items and fix
16:25:10 <nirik[m]> aye aye captain
16:25:36 <nirik[m]> nothing really new to report here... same alerts for the most part.
16:25:47 <nirik[m]> There's a ticket to fix some of them if anyone wants to look into them.
16:26:01 <nirik[m]> otherwise the engines are churning along.
16:26:31 <mkonecny> Good to hear chief engineer
16:27:15 <mkonecny> And now for our VIP guest Eddie Jennings, Jr. , who will talk about BTRFS
16:27:22 <mkonecny> #topic Learning topic
16:27:22 <mkonecny> #info 2023-02-02 Using Btrbk for BTRFS snapshot backups [eddiejennings]
16:27:58 <eddiejenningsjr> :D
16:28:26 <eddiejenningsjr> As always this comes with the preface of I'm no master of BTRFS.  I'm learning along with the rest of us :)
16:29:11 <mkonecny> There are no masters of BTRFS, just those that know more than others :-D
16:29:11 <eddiejenningsjr> Also, I'm at work right now, so hopefully I won't be summoned in the next 30 minutes.
16:33:15 <eddiejenningsjr> If you're wanting to use BTRFS's snapshot capability for backups, this is the basic overview of how that would work.
16:33:15 <eddiejenningsjr> 1. Create a BTRFS subvolume such as /.snapshots to store your snapshots locally
16:33:15 <eddiejenningsjr> 2. Use btrfs subvolume snapshot -r to make a read-only snapshot of the BTRFS volume you want to backup
16:33:15 <eddiejenningsjr> 3. Use btrfs send | brtfs receive to send the snapshot to a different medium or offsite.
16:34:17 <eddiejenningsjr> The key when doing this method is the -r when making the snapshot as this will make it read-only (immutable).  Note it can be deleted, but not altered.
16:35:07 <eddiejenningsjr> So it is entirely possible to write a BASH script to do this and run it as a cron job or with systemd timers; however. . .
16:36:11 <eddiejenningsjr> While I do enjoy writing my own stuff, sometimes its best to not reinvent the wheel.  That's where btrbk comes in.  https://github.com/digint/btrbk
16:37:03 <eddiejenningsjr> Before I talk about btrbk, are there any questions about the general backup concept of, take a snapshot, have it stored locally, then send it to a remote place?
16:39:10 <eddiejenningsjr> Seeing none, I'll keep going.
16:39:23 <nirik[m]> does send/receive use ssh? or it's own transport? or ?
16:41:16 <eddiejenningsjr> Yes :)  If you send remotely, you'll likely be using ssh.  If you're sending to say an external hard drive, you'd just have btrfs send your_snapshot | btrfs receive /path/to/your/device/dir
16:42:08 <eddiejenningsjr> You can install btrbk via Fedora repos (dnf install btrbk).
16:43:28 <nirik[m]> nice. I didn't know you could do that locally... makes sense though.
16:44:10 <eddiejenningsjr> The general setup for btrbk is that you edit its configuration file.  Then schedule /usr/bin/btrbk run.
16:44:26 <eddiejenningsjr> It really is that simple.
16:45:04 <eddiejenningsjr> Now, you can create as much complexity as you want.  I'll describe what I tested and describe the success.
16:46:27 <eddiejenningsjr> btrbk provides a btrbk.conf.example file, which is useful for building your own config file.  What you'll probably do is just take the example file and simply alter it to your needs.  I don't have my file in a public repo, else I'd share it here (might make it public one day).
16:48:06 <eddiejenningsjr> For sending snapshots over SSH, the easiest setup for that is to setup ssh key authentication to the root account on the receiving server, and just connecting via root.  I didn't want to do that, as I do try to not use root when I don't have to for things.
16:49:40 <eddiejenningsjr> So what I did was create a different user account for btrbk to connect as.  However, doing this required me to grant this user sudo privileges, since btrfs commands require elevated privileges.
16:50:40 <eddiejenningsjr> At first I thought all I needed to do was give it permission to use sudo with btrfs-receive; however, I learned that there's a bit more happening with btrbk's connection (because of the features brtbk provides).
16:50:59 <eddiejenningsjr> What I ended up using for the account was ALL=(ALL) NOPASSWD: /usr/sbin/btrfs*,/usr/bin/readlink*
16:51:31 <eddiejenningsjr> allowing it to use any command starting with btrfs (which includes options and such) and readlink.
16:52:55 <eddiejenningsjr> So while the account won't own the entire server, it can do what it wants with btrfs.  The other safety measure I have is that this account has no password; thus, it can only authenticate with ssh keys.  Should someone get console access to my backup server, they wouldn't be able to login as my backup service user.
16:55:18 <eddiejenningsjr> Once I got the authentication and sudo figured out, snapshots started flowing well.  I had btrbk keep hourly snapshots for a day, and daily snapshots for a week.  You can configure it to do retention on the target server, but for my tests I chose not to do that.
16:55:39 <eddiejenningsjr> Now, there is a gotcha with btrfs sending to a remote.  In my experience it CANNOT handle a connection interruption.  So if your ssh connection dies, you have to start over, else you are left with an incomplete snapshot.
16:56:16 <eddiejenningsjr> So for your very first snapshot send, it may take a while.  But subsequent sends are diffs, so they are quick.
16:57:43 <mkonecny> We are almost at end of this meeting
16:57:48 <eddiejenningsjr> The way I handled potential incomplete snapshots was I scheduled a script that ran btrbk run, as well as btrbk clean, which deleted incomplete snapshots on the server side.  That way I didn't have to worry about keeping incomplete snapshots around, and since I was snapshotting hourly missing a few wasn't a problem.
16:57:53 <eddiejenningsjr> Alas, yes we are :(
16:58:12 <eddiejenningsjr> I thought I'd have less to say :P
16:58:52 <eddiejenningsjr> I'd be happy to do a part 2 of btrbk and take questions and go through my setup more at a late rmeeting
16:59:12 <eddiejenningsjr> and go over restoration of backups
16:59:24 <mkonecny> It was plenty of information :-)
16:59:59 <phsmoura> eddiejenningsjr++
17:00:28 <darknao> eddiejenningsjr++
17:00:36 <mkonecny> Thank you everyone for joining our voyage today
17:00:37 <mkonecny> Let's see you next time
17:00:47 <darknao> eddiejennings++
17:00:47 <zodbot> darknao: Karma for eddiejennings changed to 7 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:00:52 <eddiejenningsjr> Thanks for steering the ship zlopez :)
17:00:57 <eddiejenningsjr> zlopez++
17:01:01 <mkonecny> #endmeeting