19:00:23 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
19:00:23 <zodbot> Meeting started Mon Feb 15 19:00:23 2021 UTC.
19:00:23 <zodbot> This meeting is logged and archived in a public location.
19:00:23 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:23 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:23 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:00:23 <nirik> #chair mboddu nirik smooge pingou mobrien
19:00:23 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting
19:00:23 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops
19:00:23 <zodbot> Current chairs: mboddu mobrien nirik pingou smooge
19:00:23 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:00:24 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily
19:00:25 <nirik> #info reminder: speak up if you want to work on a ticket!
19:00:26 <nirik> #topic Tickets needing review
19:00:28 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
19:00:40 <nirik> smooge: you around?
19:00:41 <smooge> hello
19:00:44 <smooge> lots of tickets
19:01:19 <nirik> yeah. you ok to mod?
19:01:25 <nirik> .ticket 9645
19:01:26 <zodbot> nirik: Issue #9645: IAD2: Rack/stack new servers - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9645
19:01:34 <nirik> med/med/ops ?
19:01:58 <smooge> still trying to get firefox to work.. 1 sec
19:02:04 <nirik> I guess all of them are...
19:02:21 <nirik> .ticket 9662
19:02:22 <zodbot> nirik: Issue #9662: Please disable the fedmsg-stg-bot in #fedora-neuro - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9662
19:02:43 <nirik> that one needs investigating why it's getting the prod stream. med/med/ops seems fine for it too.
19:06:49 <nirik> BRB, morecoffeeneeded
19:06:49 <smooge> looking at it.. its like all the filters are gone
19:07:18 <nirik> value01.stg is definitely getting production messages. Something got messed up in groups or something I bet.
19:09:26 <smooge> so what i was wondering was why we were getting: pagure.issue.comment.added -- computerkid commented on ticket Fedora-Council/tickets#353: "Refreshing the Fedora Podcast" https://pagure.io/Fedora-Council/tickets/issue/353#comment-715526
19:11:10 <nirik> because:
19:11:12 <nirik> body=[
19:11:12 <nirik> "^((?!(fedora-infrastructure)).)*$",
19:11:12 <nirik> ],
19:11:28 <nirik> any message that has "fedora-infrastructure" in the body of it, will match
19:12:16 <nirik> I revamped the releng one not long ago, we could do the infra one too.
19:12:23 <nirik> but thats a seperate issue from the stg thing...
19:14:06 <nirik> .ticket 9664
19:14:07 <zodbot> nirik: Issue #9664: Koschei: not all neuro-sig packages are listed in the neuro-sig group now - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9664
19:14:14 <nirik> is probibly low/low/ops?
19:14:24 <nirik> we need to figure out why the sync script isn't running.
19:16:22 <nirik> any other ones to discuss?
19:17:20 <copperi> Where are physical servers / racks at ?
19:17:35 <smooge> multiple locations
19:18:08 <smooge> some are in a DC in Virginia, USA, others in NC, USA and others are at some universities and isps
19:20:05 * nirik nods. all over the place. ;)
19:20:20 <copperi> ok, ticket didn't say where
19:20:58 <nirik> the ones with IAD2: are the ashburn, va datacenter...
19:21:07 <smooge> Ah ok, IAD2 is the code name for the datacenter.
19:21:15 <copperi> ok
19:21:23 <smooge> sorry for the confusion.. using a TLA without explanation
19:22:10 <nirik> smooge: so, I was pondering on updates/reboots: How about I apply updates all around, then reboot what I can, and if something gets 'squirrley' for the updates, we just reboot it...
19:22:38 <nirik> and we can do a proper cycle between beta and final.
19:23:23 <smooge> sounds good. I have been doing hand updates on a couple of boxes which take a long time
19:23:45 <smooge> so log01 should only have a short update this time versus the hour it took when I did it last week
19:24:19 <nirik> cool. I will plan on that later today.
19:24:28 <nirik> otherwise I still have piles of emails to get thru.
19:24:36 <nirik> and rawhide compose to unbreak
19:25:40 <nirik> smooge: you have any plans for the rest of today/tomorrow?
19:26:04 <nirik> oh, and I am out friday again (this is the last one I planned I think)
19:26:09 <smooge> so far my day has been nearly back to back meetings
19:26:33 <smooge> so my next thing is to see what I wrote down as deliverables from those meetings and try to completely them
19:26:41 <nirik> and wed I might need to drop a cat at the vet early and pick up later... or we might cancel that. should know tomorrow.
19:26:59 <smooge> tomorrow will be whatever is needed to help Matt. I should also call IBM and Lenovo and see if I can get hardware repairs sent
19:27:25 <nirik> so I hope he can power cycle things for us at least. :)
19:27:32 <smooge> me too
19:28:04 <nirik> I did have a question on the stg power8... were we going to have him cycle it and see if it came up at all? or just try and move another in it's place? moving another in would require a bunch of vlan changes...
19:28:10 <smooge> I am going to have him remove the dead mustangs from the racks. [there are 3 which were never recovered. Then I was going to have the ones in 101 moved to the empty slots in 105
19:28:30 <smooge> i was going to move another and list this one as dead
19:28:41 <smooge> then get it and the other 2 dead ones sent back to IBM
19:28:56 <nirik> I mean, we don't really know what happened to it. It just dropped off... it could just need a cycle...
19:28:57 <smooge> which is what I need to add to my list of finding out how that is done
19:29:21 <smooge> i thought you did power cycle it
19:29:34 <smooge> but at this point my brain is mush..
19:29:42 <nirik> I'm not sure if I did or if we didn't have access to pdu's at that point.
19:29:55 <smooge> that one is in 104 which hasn't had pdu changes yet
19:30:20 <nirik> oh? I thought they all changed. ok. I can try power cycling it/finding it's serial then.
19:30:23 * nirik adds to list
19:30:23 <smooge> but these are pdu's ... if you can get them to stay on the net for a day you are perfomring miracles
19:30:42 <smooge> nirik, no 101, 102 and one pdu in 103 got changed. this week is 103/104/105
19:30:57 <smooge> depending on when electricians and stuff show up
19:31:18 <smooge> or at least that is what I got from Sunday's dm
19:31:25 <nirik> ok. I just want access so we cna cycle things. ;)
19:32:09 <smooge> understood.
19:32:20 <nirik> ok, anything else?
19:32:26 <nirik> #topic open floor
19:32:29 <dtometzki> what are pdu´s
19:32:31 <dtometzki> ?
19:32:37 <smooge> power distribution units
19:32:41 <nirik> #info nirik out on friday on 'vacation'
19:32:59 <smooge> the long strips of power on the back of a cabinet that rack mounted systems are plugged into
19:33:00 <nirik> yeah, basically power strips that are networked...
19:33:15 <smooge> good question and another TLA
19:33:19 <nirik> so you can get an interface to them to turn power on or off on particular sockets. ;)
19:34:04 <smooge> if its working.. the chips in these things are weaker than a 2 dollar cell phone so we end up resetting them as much as the sockets
19:34:15 <nirik> :)
19:34:23 <smooge> but when it works.. we can fix a lot of hard hung systems
19:34:29 <nirik> great question dtometzki... thanks for being here and asking. :)
19:34:41 <nirik> ok, if nothing else will end in a minute
19:34:42 <dtometzki> :-)
19:34:47 <smooge> thanks copperi also
19:34:50 <smooge> copperi++
19:34:50 <zodbot> smooge: Karma for copperi changed to 3 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
19:34:55 <smooge> dtometzki++
19:34:55 <zodbot> smooge: Karma for dtometzki changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
19:35:03 <smooge> and wit that.. I am off to coffee land
19:35:03 <nirik> thanks all.
19:35:08 <nirik> #endmeeting