15:00:01 <nirik> #startmeeting Infrastructure (2020-08-27)
15:00:01 <zodbot> Meeting started Thu Aug 27 15:00:01 2020 UTC.
15:00:01 <zodbot> This meeting is logged and archived in a public location.
15:00:01 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:01 <zodbot> The meeting name has been set to 'infrastructure_(2020-08-27)'
15:00:01 <nirik> #meetingname infrastructure
15:00:01 <nirik> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul mobrien
15:00:01 <nirik> #info Agenda is at: https://board.net/p/fedora-infra
15:00:01 <zodbot> The meeting name has been set to 'infrastructure'
15:00:01 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny mobrien nirik pingou siddharthvipul smooge
15:00:02 <nirik> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:00:03 <nirik> #topic aloha
15:00:14 <smooge> hello!
15:00:28 <mkonecny> .hello zlopez
15:00:28 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@psmail.xyz>
15:00:30 <seddik> hello
15:00:37 <mobrien> .hello mobrien
15:00:38 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
15:00:46 <hyperreal> .hello hyperreal
15:00:47 <zodbot> hyperreal: hyperreal 'None' <hyperreal64@pm.me>
15:00:49 <siddharthvipul> .hello siddharthvipul1
15:00:51 <zodbot> siddharthvipul: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com>
15:01:11 <seddik> .hello seddik
15:01:16 <zodbot> seddik: seddik 'seddik alaoui ismaili' <seddikalaouiismaili@gmail.com>
15:01:20 <nirik> good morning everyone.
15:01:24 <nirik> #topic New folks introductions
15:01:24 <nirik> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:01:24 <nirik> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:01:32 <nirik> any new folks like to introduce themselves?
15:01:39 <hyperreal> I will
15:02:11 <nirik> welcome hyperreal.
15:03:10 <mobrien> welcome hyperreal
15:03:26 <nirik> are you interested in development work or operations or both ?
15:03:36 <hyperreal> I'm Jeff Serio. Have a Bachelor's degree in CS. I'm interested in system administration, DevOps, containers, automation with Ansible, OpenShift, etc.
15:03:55 <hyperreal> nirik: for Fedora Infra, I'm interested in both
15:04:45 <nirik> awesome. Do take a look at the Getting started link and see us after the meeting in #fedora-admin and we can get you added to our apprentice group...
15:05:06 <nirik> any other newcomers today?
15:05:18 <hyperreal> nirik: I'm already in the apprentice group :) I was new last week but missed the meeting
15:05:24 <TomAPC> Yup, finally got time to attend.
15:05:25 <nirik> ah ha. cool.
15:05:38 <nirik> hey TomAPC. welcome.
15:05:54 <hyperreal> Hi TomAPC
15:06:35 <mboddu> nirik: hyperreal actually worked on the playbook to copy content between different hosts - https://pagure.io/fedora-infrastructure/issue/9236
15:06:48 <hyperreal> hi mboddu :)
15:07:03 <mboddu> Hi hyperreal :)
15:07:15 <TomAPC> Hey folks, I'm Tom, SysAdmin for an AI firm in the UK. Looking to expand my SysAdmin knowledge outside of my very in-house tooling we use now. Looking into how the 'real world' works with Devops, general SysAdmin operations as well as things like containers (currently use Docker+K8s, interested to see how things like podman work) as well as various other tasks. I promise to sent an introduction to the mailing list soon :)
15:07:21 <TomAPC> Thanks all for the welcome!
15:07:37 <hyperreal> That sounds awesome TomAPC
15:07:39 <nirik> cool. welcome again...
15:08:06 <nirik> Do feel free to ask questions, look around for tasks to work on, etc. :)
15:08:48 <TomAPC> Thanks nirik, will also follow up about the apprentice group also :)
15:08:54 <nirik> #topic Next chair
15:08:54 <nirik> #info magic eight ball says:
15:08:54 <nirik> #info 2020-08-27 - nirik
15:08:54 <nirik> #info 2020-09-03 - pingou
15:08:54 <nirik> #info 2020-09-10 - ?
15:08:59 <nirik> TomAPC: sounds good!
15:09:08 <nirik> anyone want to step up for the 09-10 meeting?
15:09:19 <smooge> actually i was going to see if I could run next weeks meeting and have pingou run the one after that.. but he is out
15:09:22 <mobrien> I can do that one
15:09:28 <smooge> but that works
15:09:47 <nirik> mobrien: thanks.
15:09:50 <hyperreal> Last I checked there weren't any open easyfix issues. Is there another way to find issues suitable for newcomers?
15:10:20 <nirik> hyperreal: it's a difficult problem, which we haven't really come up with a good solution to...
15:10:35 <hyperreal> I see
15:11:20 <nirik> it's a balance because often easyfix tickets are something we could fix in a minute... so we have to refrain from fixing it, file a ticket and hope someone wants to fix it...
15:11:29 <nirik> then they get stae
15:11:32 <nirik> stale
15:11:42 <hyperreal> yeah I was thinking that
15:12:07 <nirik> but I would say: listen in channels and chime in when you want to help out on something people are discussing, and look at normal tickets for something that looks easy...
15:12:20 <nirik> and we should make another effort to file some more easyfix ones.
15:12:39 <hyperreal> indeed. I was also thinking finding something more challenging and working with someone more experienced
15:13:19 <nirik> yeah. +1
15:13:27 <nirik> #topic announcements and information
15:13:27 <nirik> #info CPE Sustaining EU-hours team has a Monday through Friday 30 minute meeting going through tickets at 0830 UTC in #centos-meeting
15:13:27 <nirik> #info CPE Sustaining NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin
15:13:27 <nirik> #info Datacenter move is over, but some items still need to be done: see https://fedoraproject.org/wiki/Infrastructure/2020-post-datacenter-move-known-issues
15:13:33 <nirik> any other announcements?
15:13:54 <mobrien> I have one
15:14:03 <mobrien> Probably should have added it there
15:14:36 <nirik> just #info it now. ;)
15:14:36 <mobrien> We have a docs hackfest next Friday 4th September if anyone would like to join they are more than welcome
15:15:10 <mobrien> #info Fedora Infra docs fest Friday 4th September 13:00 - 16:00 UTC
15:15:12 <nirik> Excellent. I am looking forward to it.
15:15:16 <TomAPC> Sounds like a good first task :)
15:15:22 <hyperreal> mobrien,m the hackfest sounds good
15:15:33 <nirik> might not make the start, but I will be there. ;)
15:15:42 <mboddu> Count me in, I am very much interested
15:15:48 <nirik> mobrien: is there a particular place? irc?
15:16:12 <mobrien> If anyone wants to add to the agenda it is here https://board.net/p/infra-doc-hackfest
15:16:20 <mboddu> Probably we can reserve one of the #fedora-meeting channels
15:16:27 <seddik> okey
15:16:29 <mboddu> Provided if they are available
15:16:44 <mobrien> nirik I sent an email to the fedora infra lit that has agoogle meet link but if people want irc we can use that
15:16:57 <smooge> #info Smoogen will be on PTO 2020-09-05->2020-09-20
15:16:58 <mobrien> s/lit/list/
15:17:29 <nirik> mobrien: sure, I think things are pretty light for meetings on fridays...
15:17:46 <nirik> (so meeting channels should be pretty open)
15:17:50 <mobrien> The reason for the video link is that pbokoc is doing a demo on antora and docs.fp.o at the start
15:17:57 * nirik nods.
15:18:30 <nirik> thanks for organizing this mobrien!
15:18:37 <mboddu> mobrien++
15:18:37 <nirik> smooge: enjoy your time away. :)
15:18:38 <zodbot> mboddu: Karma for mobrien changed to 10 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:18:46 <mobrien> I will add an irc channel as well and we can branch out to that after the initial video
15:18:57 <nirik> sounds good.
15:19:14 <mboddu> smooge: Enjoy your time off, hopefully you will get us more builders/storage when you get back :P
15:19:26 <nirik> #topic Oncall
15:19:26 <nirik> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:19:26 <nirik> #info mobrien is oncall for 2020-08-20 to 2020-08-27
15:19:26 <nirik> #info smooge is oncall for 2020-08-27 to 2020-09-03
15:19:27 <nirik> #info ? is oncall for 2020-09-03 to 2020-09-10
15:19:41 <nirik> someone want oncall for 09-03 -> 09-10?
15:19:48 <mboddu> If no one, I can take it
15:19:54 <mboddu> If no one interested*
15:19:56 <nirik> thanks mboddu!
15:20:20 <nirik> #info Summary of last week: (from current oncall )
15:20:31 <nirik> I guess thats mobrien ? any oncall pings?
15:20:33 <mobrien> 2 pings this week
15:20:37 <mobrien> 1. nils asked about backup/restore for FreeIPA
15:20:49 <mobrien> I suggested that a playbook for each be created, potentially a toddler/cron for the back up
15:20:58 <mobrien> Less privileges needed for back up as not a disruptive change
15:21:03 <mobrien> High privilege needed for restore as potentially destructive
15:21:15 <mobrien> 2. Asked to run bastion playbook to update aliases
15:21:22 <mobrien> nirik beat me to it as it was out of hours
15:21:24 <nils> re 1: https://pagure.io/fedora-infra/ansible/pull-request/218 :)
15:22:18 <mobrien> There was also a question this morning about sigul which wasn't technically an on call ping
15:22:20 <nirik> cool.
15:22:58 <mobrien> it was just about versions and ended up being an issue on the end user side
15:23:57 <nirik> sounds good. Things seem to be kinda quiet lately. :) Which is just fine with me...
15:24:07 <mobrien> nirik should we have mentioned the beta freeze in the announcements, just thought of it
15:24:08 <seddik> someone can review thhis PR => https://pagure.io/fedora-infra/ansible/pull-request/178  while pingou is away ?
15:24:46 <nirik> mobrien: oh yeah. ;)
15:24:52 <nirik> seddik: I can try and take a look
15:24:58 <nirik> #topic Monitoring discussion [nirik]
15:24:58 <nirik> #info https://nagios.fedoraproject.org/nagios
15:24:58 <nirik> #info Go over existing out items and fix
15:25:03 <nirik> so, nagios...
15:25:13 <seddik> nirik thanks
15:25:39 <nirik> down hosts are mustangs... need some serious poking with power cycle and serial.
15:25:56 <nirik> retrace01 is in there, perhaps that should be removed now that retrace03 is up...
15:26:21 <nirik> proxy35 and proxy36 are being created by mobrien...
15:27:07 <nirik> bugzilla2fedmsg is broken. I guess I should file a ticket on it to track fixing it... we are connecting, but getting no messages.
15:27:50 <nirik> pagure01 is getting really low on disk space. we should really move it to rhel8 and give it more space soon.
15:28:34 <seddik> and sundries01.iad2.fedoraproject.org
15:28:53 <nirik> yeah, that needs some investigation, or just making it bigger I guess.
15:28:58 <smooge> nirik, I power cycled all the mustangs last week and got the ones I could online
15:28:59 <nirik> although we are in freeze right now.
15:29:12 <smooge> it lasted all of a day and they oops to death
15:29:31 <nirik> smooge: thanks. yeah, I think perhaps some of them need fresh installing? not sure.
15:30:12 <smooge> probably.. I have never successfully gotten a mustang to install without a dd so not sure
15:30:23 <nirik> so lots of things to fix... but most of them we can wait until after freeze.
15:32:41 <nirik> smooge: or perhaps we just replace them with emags and let them go quietly into the night
15:32:41 <nirik> anyhow, thats about all the nagios poking today I think.
15:32:41 <nirik> #topic Open Floor
15:33:00 <nirik> and... irc disconencted for me. wheee
15:33:08 <nils> You're still here.
15:33:24 <nils> Anyway, Ryan and I asked ourselves if changes in staging are affected by the beta freeze.
15:33:27 <mobrien> irc has been misbehaving all day for me
15:33:40 <nils> We thought probably not, because it's not really working ATM anyway.
15:33:52 <nirik> nope... staging is never frozen
15:33:58 <nils> 👍
15:34:04 <nirik> so hack away. ;)
15:34:06 <nils> Thanks for clarifying!
15:34:19 <nirik> I did start on deploying koji01.stg and db-koji01.stg yesterday.
15:34:24 <nils> Ach, let's wait for the proxies and firewalls first :o)
15:34:34 <nirik> I might do some more today... but yeah, we need proxies to work.
15:35:10 <nirik> smooge: any news on that? or still in holding pattern?
15:35:29 <mobrien> Is it just existing things that are frozen or does it mean no new stuff as well?
15:35:29 <nils> nirik, just looked into the ticket, no changes there
15:36:48 <nirik> mobrien: well, anything that affects frozen hosts is frozen... so new proxies aren't frozen, but you might need/want a freeze break if/when you add them into production... ie, adding vpn ccd files or the like
15:36:58 <nirik> nils: ;(
15:37:16 <nils> nirik, *shrug* it is as it is ;)
15:37:22 <nirik> but you can do a PR/freeze break... we are usually pretty open on them especially near the start of a freeze.
15:37:49 <nils> nirik, we're hammering away at local ipsilon testing for the time being
15:38:27 <nirik> cool.
15:38:33 <smooge> firewall work is sidelined with other work. trying to get it out today
15:38:43 <nirik> Anyone have anything else for open floor?
15:40:38 <nirik> ok then... everyone gets 20min of their day back! :)
15:40:42 <nirik> #endmeeting