16:00:17 <mobrien> #startmeeting Infrastructure (2022-05-26)
16:00:17 <zodbot> Meeting started Thu May 26 16:00:17 2022 UTC.
16:00:17 <zodbot> This meeting is logged and archived in a public location.
16:00:17 <zodbot> The chair is mobrien. Information about MeetBot at https://fedoraproject.org/wiki/Zodbot#Meeting_Functions.
16:00:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:17 <zodbot> The meeting name has been set to 'infrastructure_(2022-05-26)'
16:00:23 <mobrien> #meetingname infrastructure
16:00:23 <zodbot> The meeting name has been set to 'infrastructure'
16:00:29 <mobrien> #chair nirik siddharthvipul mobrien zlopez bodanel dtometzki jnsamyak computerkid
16:00:29 <zodbot> Current chairs: bodanel computerkid dtometzki jnsamyak mobrien nirik siddharthvipul zlopez
16:00:36 <mobrien> #info Agenda is at: https://board.net/p/fedora-infra
16:00:45 <mobrien> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
16:00:50 <nirik> morning
16:00:51 <mobrien> #info Fedora Infra documentation: https://docs.fedoraproject.org/en-US/infra
16:00:51 <mobrien> #topic greetings!
16:01:15 <eddiejennings> .hi
16:01:16 <zodbot> eddiejennings: eddiejennings 'Eddie Jennings' <eddie@eddiejennings.net>
16:01:17 <mobrien> howdy everyone!
16:01:54 <nikolay__> hi
16:02:51 <austinpowered> .hi
16:02:52 <zodbot> austinpowered: austinpowered 'T.C. Williams' <fedoraproject@wootenwilliams.com>
16:02:56 <petebuffon> .hello petebuffon
16:02:57 <zodbot> petebuffon: petebuffon 'Peter Buffon' <pabuffon@gmail.com>
16:03:31 <mkonecny> .hello zlopez
16:03:32 <zodbot> mkonecny: zlopez 'Michal Konecny' <michal.konecny@psmail.xyz>
16:03:45 <mobrien> #topic New folks introductions
16:03:52 <mobrien> Any new folks here today?
16:04:01 <nikolay__> yes
16:04:09 <nikolay__> What's up? My name is Nikolay. Essentially It is not my first introduction. The first one was two years ago when I was just a networking admin. So I felt here useless:) I want to try again!
16:04:09 <nikolay__> Today, I am a junior Linux Admin(only two years). Mostly skilled in python/ansible/bash, okd4/3, k8s, KVM, maintaining data-warehouse, and regular sysadmin staff. Mostly I am working offline without cloud, internet, etc. So I am here to contribute, learn and practice new technologies and to level up my skills.
16:04:12 <mobrien> This is a chance to introduce yourself.
16:04:39 <eddiejennings> Welcome, nikolay__
16:04:53 <mobrien> Welcome nikolay__
16:05:02 <nirik> welcome!
16:05:03 <mobrien> I hope it goes better this time
16:05:07 <nikolay__> thanks:)
16:05:17 <mobrien> I'm sure there is plenty around here for you
16:05:42 <mobrien> Forgot this
16:05:42 <mobrien> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
16:05:49 <petebuffon> welcome :)
16:06:13 <eddiejennings> Worry not nikolay__, I'm only slightly above useless and I haven't been banished :D
16:06:41 <nikolay__> ok :D
16:07:21 <mobrien> Ok lets move on to the next section
16:07:24 <mobrien> #topic Next chair
16:07:29 <mobrien> #info chair 2022-05-26 - mobrien
16:07:34 <mobrien> #info chair 2022-06-02 - eddiejennings
16:07:41 <mobrien> chair 2022-06-09 - ???
16:07:47 <nirik> everyone is useful. ;) we are all in this together!
16:07:50 <mobrien> any volunteers
16:07:56 <mobrien> +1
16:08:02 <eddiejennings> I'm still good for 2022-06-02
16:08:13 <mobrien> Thanks eddiejennings
16:10:21 <nirik> running meetings is fun and a good way to get involved. ;)
16:10:48 <eddiejennings> nirik has the truth of it.  It *is* fun. :)
16:11:11 <mobrien> It's quite simple to do. It's just copy/paste and you get to know the people around
16:11:28 <nirik> if no one else wants it I can take it...
16:11:57 <eddiejennings> I'm unavailable on 2022-06-09, else I'd do it again.
16:12:05 <mobrien> Going once....
16:12:21 <mobrien> Going twice....
16:12:28 <petebuffon> i'll take it
16:12:45 <mobrien> and petebuffon takes it at the last!
16:13:18 <mobrien> for extra dramatic effect 🙂
16:13:33 <mobrien> #info chair 2022-06-09 - petebuffon
16:13:57 <petebuffon> hah you know me
16:14:25 <mobrien> ok now that we have that, lets take the journey forward ....
16:14:29 <mobrien> #topic announcements and information
16:14:34 <mobrien> #info CPE Infra&Releng EU-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1030 Europe/paris in #centos-meeting
16:14:39 <mobrien> #info CPE Infra&Releng NA-hours team has a Monday through Thursday 30 minute meeting going through tickets at 1800 UTC in #fedora-meeting-3
16:14:46 <mobrien> #info If your team wants support from the Fedora Program Management Team, file an isssue: https://pagure.io/fedora-pgm/pgm_team/issues?template=support_request
16:14:52 <mobrien> #info please help us with improving contribution to fedora infra https://discussion.fedoraproject.org/t/improving-contribution-to-fedora-infrastructure/38294/8
16:14:59 <mobrien> #info oncall should also handling #fedora-releng pings if possible
16:15:38 <mobrien> Does anyone have any other announcements or tidbits they want to put out to the world?
16:16:37 <nirik> #info new ipsilon version rolled out... seperate otp field is live!
16:17:08 <mobrien> Wohoo! 🎉
16:17:39 <eddiejennings> Yay!
16:19:54 <mobrien> Ok seems no more announcements, let me continue
16:19:55 * nirik closes those 2 tickets
16:19:55 <mobrien> #topic Oncall
16:20:03 <mobrien> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
16:20:09 <mobrien> ## .oncalltakeeu .oncalltakeus
16:20:15 <mobrien> #info eddiejennings on call from 2022-05-20 to 2022-05-26
16:20:20 <mobrien> #info darknao on call from 2022-05-27 to 2022-06-02
16:20:33 <mobrien> ??? on call from 2022-06-03 to 2022-06-10
16:20:55 <mobrien> Is anyone willing and able to take those dates?
16:21:10 <nirik> I'm happy to take the last one if no one else wants it. ;)
16:21:53 <mobrien> Lets see if there will be late drama again
16:21:59 <mobrien> Going once ....
16:22:30 <mobrien> Going twice ....
16:23:27 <mobrien> huh there is no gavel emoji
16:24:16 <mobrien> Oh wait there is a judge with a gavel, lets pretend they are an auctioneer
16:24:27 <mobrien> gone to nirik 🧑‍⚖️
16:24:44 <mobrien> #info nirik on call from 2022-06-03 to 2022-06-09
16:25:04 <mobrien> also if darknao is around feel free to take it for this week
16:25:39 <mobrien> ok lets move on
16:25:42 <mobrien> #info Summary of last week: (from current oncall )
16:25:47 <mobrien> over to you eddiejennings
16:26:52 <eddiejennings> I did not recall seeing any pings for the oncall this week; though, I did not regularly monitor the releng channel.  I need to add that to my default channels in Hexchat.
16:27:13 <eddiejennings> I know there were a couple of things that came up, but the regulars were online at the time.
16:28:21 <eddiejennings> That is all. :)
16:28:27 <mobrien> I just did a search and no oncall in releng for 10 days so you didn't miss any 🙂
16:28:47 <eddiejennings> Phew. :D
16:29:09 <mobrien> Ok lets move to ..
16:29:09 <mobrien> #topic Monitoring discussion [nirik]
16:29:16 <mobrien> #info https://nagios.fedoraproject.org/nagios
16:29:21 <mobrien> #info Go over existing out items and fix
16:29:26 <mobrien> over to you nirik
16:29:43 <nirik> I cleaned up some more things the other day...
16:30:01 <nirik> we have 1 down mgmt interface (actually there's another one down that isn't in nagios perhaps?)
16:30:53 <nirik> there's some busgateway ones that need cleaning up... we repointed it to the new datanommerdb and some of it's checks are... not needed anymore.
16:31:15 <nirik> Also, in related news I got zabbix in staging up the other week...
16:31:26 <nirik> haven't had a chance to add more hosts to it yet, but will try when I get time.
16:31:28 <mobrien> I had a look at proxy09 briefly this morning on one of my breaks from training. I cleared some of the alarms but need to look at the others
16:31:53 <petebuffon> how do you like zabbix vs. nagios?
16:31:54 <nirik> I just fixed 2 more there... selinux labels were off.
16:32:29 <nirik> petebuffon: well, a bit early to tell. zabbix has a much nicer web interface I think... want to see if it's default monitoring handles most of what we want...
16:32:39 <mobrien> ah, thanks nrirk
16:32:42 <mobrien> nirik++
16:32:42 <zodbot> mobrien: Karma for kevin changed to 5 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
16:32:51 <petebuffon> nirik: okay thanks :)
16:33:02 <mobrien> I'm amazed I hadn't given you cookies yet
16:33:26 <nirik> ha. nom nom nom nom
16:33:34 <nirik> oh, looking at noc02....
16:33:57 <nirik> there's a bunch of copr dev machines listed there. we should either stop checking them there or figure out why it's showing them down
16:35:00 <nirik> thats it for monitoring.
16:35:10 <mobrien> noc02 is our external nagios server for those who may be unaware https://nagios-external.fedoraproject.org/nagios/
16:35:45 <nirik> yeah. it just monitors a subset of external things...
16:36:35 <mobrien> Ok moving on, we do have a learning topic for today but before we get to it. Is there any other requests/suggestions/volunteers for new ones for the next few weeks?
16:36:37 <nikolay__> ok i will take a look.
16:36:44 <mobrien> Todays is the last on the list
16:38:00 <nirik> I might be able to do something on zabbix week after next...
16:38:06 <mobrien> If there is a topic people wish to learn about and I can talk on it I will, just not sure what people would like to hear about
16:38:18 <eddiejennings> I can't say when I'd be ready, but I'll want to do another one on using btrfs
16:39:23 <mobrien> I can put down the topic with no date against it and you can add a date when you are ready
16:39:38 <eddiejennings> Sounds good.
16:40:20 <eddiejennings> Shall I take the floor? :)
16:40:55 <mobrien> yes, let me just change the topic
16:40:57 <mobrien> #topic Learning topic
16:41:01 <mobrien> all yours eddiejennings
16:41:44 <eddiejennings> Ok.  If you folks have come seeking wisdom from a great master of btrfs, you will be disappointed.  If you've come to hear some real-world takes from a newb who's learning himself, then you're at the right place :)
16:42:41 <eddiejennings> So I focused on becoming more familiar with replication and snapshots with btrfs, which truly became more of an experiment with snapshots.
16:43:11 <eddiejennings> First, some links.  These were helpful as references for commands and ideas.
16:43:52 <eddiejennings> https://www.fosslinux.com/46315/btrfs-file-system-fedora-linux.htm  -- Fairly ad-heavy page, but has some good fundamentals
16:44:21 <eddiejennings> https://fedoramagazine.org/btrfs-snapshots-backup-incremental/ -- The article that sparked my interest in btrfs in general
16:45:17 <eddiejennings> If you're running recent versions of Fedora workstation (and you installed fresh, rather than upgrading), you're already using btrfs. :)
16:46:03 <eddiejennings> Of the many features of btrfs, the idea of taking snapshots is quite useful for making some easy backups.
16:46:45 <eddiejennings> I'll be using my own workstation as an example for the new few minutes.
16:47:21 <eddiejennings> The /home directory is a btrfs subvolume; thus, it is eligible for having snapshots taken.
16:48:48 <eddiejennings> Let's say I wanted to take a backup of my home directory in a hidden directory named /.snapshots
16:49:40 <eddiejennings> To achieve this, I would use the following command sudo btrfs subvolume snapshot -r /home /.snapshots/name_of_my_snapshot
16:50:29 <eddiejennings> Once I run that command, there is now a read-only copy of my entire /home volume within /.snapshots/name_of_my_snapshot.
16:51:00 <eddiejennings> The -r flag is important, since for backups, you really don't want to be able to alter that data.  Afterall it's a backup, not a working copy.
16:53:08 <eddiejennings> Should you need to recover a file from that backup, you would simply cp the file from within the snapshot back to its original location: cp /.snapshots/name_of_my_snapshot/eddie/foo /home/eddie/foo
16:53:37 <eddiejennings> At a high, basic level that's all there is to it. :D
16:54:12 <nikolay__> nice!
16:54:15 <eddiejennings> "But wait!" you say.  Storing a backup on the device you're backing up isn't wise!  We need to get this onto external storage
16:54:46 <eddiejennings> That is where the btrfs send and receive commands come into play.
16:55:20 <eddiejennings> I only have a couple of minutes, but here's an overview of a way to get this to a remote backup server.
16:56:04 <eddiejennings> The caveat:  To the best of my knowlege, you can only send a snapshot to another btrfs filesystem.
16:56:37 * nirik has another meeting at the top of the hour, will slip out for that... but thanks for the talk!
16:56:51 <eddiejennings> So, let's say I wanted to send my snapshot to the /mnt/backups directory of a remote server named backup.  Prior to this, I would've already created /mnt/backups as a btrfs subvolume
16:57:30 <eddiejennings> you would do this:  sudo btrfs send /.snapshots/name_of_my_snapshot | ssh root@backup btrfs receive /mnt/backups
16:58:06 <eddiejennings> The pipe is important.  I think of it as the "output" of a btrfs send command flows through the pipe into the btrfs receive command, which places it into the desired directory.
16:58:48 <eddiejennings> To recover a file, you could use scp to fetch it from that /mnt/backups/name_of_my_snapshot/. . . on your backup server.
16:59:29 <eddiejennings> You can probably imagine some strategies where you keep one snapshot on your local system but send historical ones to your backup server.  And there's also doing incremental snapshots, but that's beyond the scope of my experiments.
16:59:47 <eddiejennings> Thus, you have the basics of using snapshots as backups for btrfs subvolumes :D
17:01:11 <nikolay__> nice summary, thanks. i always use lvm snapshots, but it looks better...
17:01:25 <eddiejennings> Thank you.  mobrien back to you
17:01:26 <nikolay__> i have to learn btrfs now
17:01:44 <mobrien> Thanks eddiejennings that was very informative
17:01:58 <mobrien> unfortunately thats all we have time for
17:02:04 <mobrien> Thanks all for coming
17:02:08 <mobrien> #endmeeting