16:00:33 <mkonecny> #startmeeting Infrastructure (2022-09-08)
16:00:33 <zodbot> Meeting started Thu Sep  8 16:00:33 2022 UTC.
16:00:33 <zodbot> This meeting is logged and archived in a public location.
16:00:33 <zodbot> The chair is mkonecny. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:33 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:33 <zodbot> The meeting name has been set to 'infrastructure_(2022-09-08)'
16:00:33 <mkonecny> #meetingname infrastructure
16:00:33 <zodbot> The meeting name has been set to 'infrastructure'
16:00:33 <mkonecny> #chair nirik mobrien zlopez bodanel dtometzki jnsamyak computerkid
16:00:33 <zodbot> Current chairs: bodanel computerkid dtometzki jnsamyak mkonecny mobrien nirik zlopez
16:00:34 <mkonecny> #info Agenda is at: https://board.net/p/fedora-infra
16:00:34 <mkonecny> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:34 <mkonecny> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:00:34 <mkonecny> #topic greetings!
16:00:48 <nirik> morning
16:01:10 <mkonecny> Hi everyone
16:01:16 <mkonecny> Welcome to today show :-)
16:01:34 <mkonecny> I will be your guide for today
16:01:34 <mkonecny> .hello zlopez
16:01:35 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@psmail.xyz>
16:01:46 <eddiejenningsjr> .hello eddiejennings
16:01:47 <zodbot> eddiejenningsjr: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:02:30 <mkonecny> Let's go to the first part of this show
16:02:45 <mkonecny> Anyone new wants to say hello?
16:02:46 <mkonecny> #topic New folks introductions
16:02:46 <mkonecny> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
16:02:46 <mkonecny> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:04:29 <mkonecny> It seems like no new folks today
16:05:02 <mkonecny> #topic Next chair
16:05:02 <mkonecny> #info magic eight ball says:
16:05:02 <mkonecny> #info chair 2022-09-15 - dtometzki
16:05:09 <mkonecny> #info chair 2022-09-22 - phsmoura
16:05:09 <mkonecny> #info chair 2022-09-29 - ???
16:05:27 <mkonecny> Anybody wants to be seated in the chair for 2022-09-29?
16:06:01 <eddiejenningsjr> I should be able to do 2022-09-29
16:06:11 <mkonecny> It's yours
16:06:36 <mkonecny> #info chair 2022-09-29 - eddiejenningsjr
16:07:05 <mkonecny> I think we are prepared for rest of the month :-)
16:07:13 <mkonecny> Let's see what else is awaiting us today
16:07:19 <mkonecny> #topic announcements and information
16:07:19 <mkonecny> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:07:19 <mkonecny> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:07:24 <mkonecny> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:07:24 <mkonecny> #info please help us with improving contribution to fedora infra https://discussion.fedoraproject.org/t/improving-contribution-to-fedora-infrastructure/38294/8
16:07:24 <mkonecny> #info oncall should also handling #fedora-releng pings if possible
16:07:45 <mkonecny> #info Fedora 37 Beta freeze now in effect!
16:07:46 <mkonecny> #info Fedora 37 Beta Go/No Go meeting happening today 1700 UTC in #fedora-meeting
16:07:50 <mkonecny> Anything else to announce?
16:08:27 <nirik> #info Fedora is awesome
16:08:31 <nirik> Thats all I have
16:08:46 * mkonecny agrees
16:08:56 <mobrien> I think that one is a known fact rather than announcement nirik
16:09:07 <nirik> ah yeah, probibly so.
16:09:36 <eddiejenningsjr> Apparently Fedora (maybe the kernel itself) isn't playing nice with the trackpad on the Clevo model used for the latest System76 Lemur Pro, but other than a minor issue, I concur, Fedora = awesome :D
16:10:26 <mkonecny> It seems that there is nothing else to announce
16:10:38 <mkonecny> Let's look at the oncall
16:10:43 <mkonecny> #topic Oncall
16:10:44 <mkonecny> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:10:44 <mkonecny> #info https://docs.fedoraproject.org/en-US/cpe/day_to_day_fedora/
16:10:49 <mkonecny> #info phsmoura is on call from 2022-09-02 to 2022-09-08
16:10:49 <mkonecny> #info mobrien is on call from 2022-09-09 to 2022-09-15
16:10:49 <mkonecny> #info dtometzski is on call from 2022-09-16 to 2022-09-22
16:11:01 <mkonecny> #info ??? is on call from 2022-09-23 to 2022-09-29
16:11:05 <mobrien> .oncalltakeeu
16:11:07 <zodbot> mobrien: Kneel before zod!
16:11:11 <mkonecny> We are well prepared for the oncall
16:11:24 <eddiejenningsjr> I'll take on-call for 2022-09-23 to 2022-09-29.  I like ending on-call with a meeting chair day :)
16:11:34 <nirik> yeah, that often works well...
16:11:42 <mkonecny> Eddie Jennings, Jr.: It's yours
16:12:02 <mkonecny> #info eddiejenningsjris on call from 2022-09-23 to 2022-09-29
16:12:24 <mkonecny> Sorry missing space
16:12:41 <mkonecny> #info Summary of last week: (from current oncall )
16:12:45 <phsmoura> I didnt receive any pings
16:13:00 <mkonecny> So quiet week
16:13:11 <mkonecny> #topic Monitoring discussion [nirik]
16:13:11 <mkonecny> #info https://nagios.fedoraproject.org/nagios
16:13:11 <mkonecny> #info Go over existing out items and fix
16:13:39 <nirik> nothing really new here I don't think... let me check
16:14:08 <mkonecny> Maybe replacing nagios with zabbix :-D
16:14:18 <nirik> I've noticed the f35-test and f36-test instances bounce on and off a lot
16:14:28 <nirik> if someone wants to go look at those that would be nice.
16:14:39 <mkonecny> What they are used for?
16:14:47 <nirik> maintainer-test machines.
16:15:05 <nirik> https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintainers
16:15:18 <nirik> they have to be on our vpn due to IPA.
16:15:49 <nirik> otherwise we continue to get badges-backend alerts. I suppose if someone is bored, they could look at that when it happens and see whats causing it to get stuck..
16:16:14 <nirik> otherwise same stuff, we can move on...
16:16:40 <mkonecny> Ok
16:16:53 <mobrien> I did have a quick look at f36 maintainer test the other day, seems to be a slow connection from what I can see with the vpn
16:16:57 <mobrien> I will look into it more though
16:17:06 <mkonecny> We actually have a learning topic for today
16:17:12 <mkonecny> #topic Learning topic
16:17:12 <mkonecny> #info 2022-09-08 BTRFS #2 [eddiejennings]
16:17:17 <eddiejenningsjr> :D
16:17:25 <nirik> excellent.
16:18:14 <eddiejenningsjr> As always, I don't claim to be a master, as I'm learning myself, and of course I couldn't finish some of my testing this weekend due to $life.
16:18:59 <eddiejenningsjr> Today, I'm going to talk about one way to leverage BTRFS for file backups.
16:20:03 <eddiejenningsjr> First, some inspiration.  This Fedora Magazine article from September 2020 is what got me thinking about this and provided some initial technical guidance.  https://fedoramagazine.org/btrfs-snapshots-backup-incremental/
16:21:35 <eddiejenningsjr> As with anything related to learning, it is best (in my opinion) to try to apply what you're trying to do to the real world; thus, here is what I set out to do.
16:23:47 <eddiejenningsjr> Both my and my wife's laptop are running Fedora 36 using default storage settings, so `/home` is using BTRFS as its file system.  I want to achieve the following.
16:23:48 <eddiejenningsjr> 1. Take regular backups of `/home`
16:23:48 <eddiejenningsjr> 2. Keep a few backups on the laptop
16:23:48 <eddiejenningsjr> 3. Keep most externally (on a VM in my home lab)
16:23:48 <eddiejenningsjr> 4. Do some cold archiving offsite (likely BackBlaze B2)
16:24:23 <eddiejenningsjr> The key is, for better or worse, I'm wanting to build the solution myself and leverage BTRFS features.
16:25:03 <eddiejenningsjr> For those unfamiliar with BTRFS, what feature am I talking about?  I'm talking about BTRFS's snapshot feature!
16:26:48 <eddiejenningsjr> What is a snapshot, you ask?  It is effectively a copy of a particular BTRFS subvolume (in fact it's a subvolume itself).
16:27:40 <eddiejenningsjr> Those more knowledgeable about BTRFS than I will likely cringe at my word copy, as it's not exactly a copy as in doing `cp file1 file2`, but in terms of function, that is how a snapshot is going to behave in this context.
16:28:25 <eddiejenningsjr> If you want to view the subvolumes on your system, you can run `sudo btrfs subvolume list /`
16:30:26 <eddiejenningsjr> So for my scenario, I decided that I wanted to make a hidden directory named `/.snapshots` and create all of my snapshots of `/home` there.
16:31:07 <eddiejenningsjr> You can create a snapshot nearly anywhere, BUT, the destination of the snapshot must be a BTRFS file system.
16:32:13 <eddiejenningsjr> Let's say that `/.snapshots` was its own mount point on an XFS filesystem (rather than being a part of `/`, which is BTRFS for me), saving BTRFS snapshots there would not work -- to the best of my knowledge.
16:33:04 <eddiejenningsjr> Ok, Eddie, how do we actually make these snapshots, and how would they be considered a backup.
16:34:53 <eddiejenningsjr> To make a snapshot that is read-only (explanation coming soon), you would run `sudo btrfs subvolume snapshot -r /home /.snapshots/snapshotname`
16:35:30 <eddiejenningsjr> `/home` is the BTRFS subvolume you're wanting to snapshot and `/.snapshots/snapshotname` is the directory that will be your snapshot.
16:36:46 <eddiejenningsjr> So if you ran that command (assuming `/.snapshots` exists on your system and it's BTRFS formatted), followed by `ls /.snapshots`, you will see a directory named `snapshotname`
16:37:00 <eddiejenningsjr> within this directory will be the entire contents of the `/home` directory
16:37:54 <eddiejenningsjr> Also everything is readonly (that's what the `-r` does), so data cannot be changed; thus, you have an immutable backup of `/home` now at `/.snapshots/snapshotname`
16:40:02 <eddiejenningsjr> If you were to delete `/home/foo.txt`.  You can easily restore the file by doing `cp /.snapshots/snapshotname/foo.txt /home/`
16:41:22 <eddiejenningsjr> The beauty of this is these snapshots (which are actually BTRFS subvolumes themselves) don't take much (any that I've seen so far, but I'm sure it's taking up something) space.  So it's feasible to take a snapshot say hourly, which is what I'll be doing.
16:41:59 <eddiejenningsjr> Earlier, I mentioned readonly snapshots, and that I would do some explaining.
16:42:03 <petebuffon> is the read-only feature using permissions?
16:43:43 <nirik> I think it's enforced at the filesystem level... and files have their own perms if you look at them...
16:43:45 <eddiejenningsjr> Good question.  From what I can see the read-only aspect does not come from traditional octal permissions
16:43:49 <nirik> (but I'm not sure)
16:44:07 <mkonecny> I agree with nirik
16:44:16 <eddiejenningsjr> But rather, it's a part of BTRFS filesystem attributes
16:45:35 <eddiejenningsjr> If you were to do `sudo btrfs subvolume show /path/to/your/snapshot` you'll see in flags that's readonly (if you made the snapshot using `-r`
16:46:04 <cmurf> Correct, it's enforced by btrfs kernel code, not even root can modify the contents of read-only subvolumes.
16:46:16 <petebuffon> got it
16:46:23 <eddiejenningsjr> By default snapshots are not readonly, you must pass the `-r` flag.
16:47:04 <eddiejenningsjr> Why would you want non-readonly snapshots?  Having a non-readonly snapshots make doing a large revert-in-place function possible.
16:47:57 * nirik thinks cmurf has a highlight for btrfs. ;)
16:48:28 <eddiejenningsjr> This isn't something I have messed with, but I want to in the future.  For my project, I am trying to emulate "traditional" backups of you have copies which are immutable, and you restore files by copying from this immutable copy.
16:49:40 <eddiejenningsjr> So for the project, I've covered items 1 and 2.  Life got in the way for testing btrfs send / receive, which would be the next step with getting these snapshots off of the local machine (snapshot doesn't do you much good when the laptop itself is on fire) and onto an external server.
16:50:36 <eddiejenningsjr> Now cmurf is far more knowledge able than I in things btrfs, but what I can provide is the newb's perspective and some basics of how to get started working with snapshots as backups :)
16:51:13 <eddiejenningsjr> So, questions / comments / snarky remarks? :)
16:51:22 <TheExorcist[m]> Hi \o
16:51:27 <cmurf> You're doing great, keep going 🙂
16:51:37 <mkonecny> I had the backups set similarly, but I was forced to disable them, because of the space the snapshot took and deleting files didn't help, because they are still in the snapshot
16:52:07 <eddiejenningsjr> This is the script I will be using to creating the snapshots on my and my wife's laptop.
16:52:07 <eddiejenningsjr> https://gitlab.com/EddieJennings/home-lab-ansible/-/blob/create_backup_server/templates/btrfs-backup.sh.j2
16:52:10 <TheExorcist[m]> Can anybody get me some learning stuff on, why we need to migrate from EXT file systems to BTRFS?
16:52:56 <eddiejenningsjr> I'm on BTRFS solely due to defaults for Fedora Workstation, and the snapshots were intriguing to me
16:53:36 <eddiejenningsjr> Said script is being launched from a systemd timer that goes off every hour.  If I find that's taking too much space, I'll adjust.
16:53:55 <nirik> TheExorcist[m]: well, a bit beyond the scope here... but look at the fedora change for it and the mailing list discussion. Basically more features with btrfs
16:53:55 <TheExorcist[m]> OK, BTRFS is something like Home Stuff for RedHat and Fedora?
16:54:14 <petebuffon> BTRFS is a modern copy-on-write filesystem with features including snapshots, file integrity, compression etc.
16:54:48 <petebuffon> Supposedly also including native encryption in the future?
16:55:24 <eddiejenningsjr> Once I get my btrfs send /receive going with my backup server, I'll do another learning session detailing the what / how / gotcha from my experience :)
16:55:25 <nirik> being worked on yeah...
16:55:27 * eddiejenningsjr hands back to zlopez
16:57:54 <mkonecny> We have last few minutes of the meeting
16:58:00 <mkonecny> So I'm switching to open floor
16:58:01 <mkonecny> #topic Open Floor
16:58:11 <mkonecny> Anything else to discuss?
16:59:03 <TheExorcist[m]> I had more issues with BTRFS, than EXT file systems, so far. in my experience, BTRFS is still very unstable.
16:59:18 <TheExorcist[m]> and unreliable.
16:59:30 <cmurf> Have you filed bugs?
16:59:38 <TheExorcist[m]> I filed one.
16:59:44 <nirik> I had issues with it many years ago... but recently it's been fine here
16:59:58 <cmurf> Cool, either cc me on it or paste URL here please
17:00:22 <mkonecny> We are out of time
17:00:24 <mkonecny> #endmeeting