19:00:23 #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 19:00:23 Meeting started Mon Feb 15 19:00:23 2021 UTC. 19:00:23 This meeting is logged and archived in a public location. 19:00:23 The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:23 Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:23 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 19:00:23 #chair mboddu nirik smooge pingou mobrien 19:00:23 #meetingname fedora_infrastructure_ops_daily_standup_meeting 19:00:23 #info meeting is 30 minutes MAX. At the end of 30, its stops 19:00:23 Current chairs: mboddu mobrien nirik pingou smooge 19:00:23 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 19:00:24 #info agenda is at https://board.net/p/fedora-infra-daily 19:00:25 #info reminder: speak up if you want to work on a ticket! 19:00:26 #topic Tickets needing review 19:00:28 #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 19:00:40 smooge: you around? 19:00:41 hello 19:00:44 lots of tickets 19:01:19 yeah. you ok to mod? 19:01:25 .ticket 9645 19:01:26 nirik: Issue #9645: IAD2: Rack/stack new servers - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9645 19:01:34 med/med/ops ? 19:01:58 still trying to get firefox to work.. 1 sec 19:02:04 I guess all of them are... 19:02:21 .ticket 9662 19:02:22 nirik: Issue #9662: Please disable the fedmsg-stg-bot in #fedora-neuro - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9662 19:02:43 that one needs investigating why it's getting the prod stream. med/med/ops seems fine for it too. 19:06:49 BRB, morecoffeeneeded 19:06:49 looking at it.. its like all the filters are gone 19:07:18 value01.stg is definitely getting production messages. Something got messed up in groups or something I bet. 19:09:26 so what i was wondering was why we were getting: pagure.issue.comment.added -- computerkid commented on ticket Fedora-Council/tickets#353: "Refreshing the Fedora Podcast" https://pagure.io/Fedora-Council/tickets/issue/353#comment-715526 19:11:10 because: 19:11:12 body=[ 19:11:12 "^((?!(fedora-infrastructure)).)*$", 19:11:12 ], 19:11:28 any message that has "fedora-infrastructure" in the body of it, will match 19:12:16 I revamped the releng one not long ago, we could do the infra one too. 19:12:23 but thats a seperate issue from the stg thing... 19:14:06 .ticket 9664 19:14:07 nirik: Issue #9664: Koschei: not all neuro-sig packages are listed in the neuro-sig group now - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9664 19:14:14 is probibly low/low/ops? 19:14:24 we need to figure out why the sync script isn't running. 19:16:22 any other ones to discuss? 19:17:20 Where are physical servers / racks at ? 19:17:35 multiple locations 19:18:08 some are in a DC in Virginia, USA, others in NC, USA and others are at some universities and isps 19:20:05 * nirik nods. all over the place. ;) 19:20:20 ok, ticket didn't say where 19:20:58 the ones with IAD2: are the ashburn, va datacenter... 19:21:07 Ah ok, IAD2 is the code name for the datacenter. 19:21:15 ok 19:21:23 sorry for the confusion.. using a TLA without explanation 19:22:10 smooge: so, I was pondering on updates/reboots: How about I apply updates all around, then reboot what I can, and if something gets 'squirrley' for the updates, we just reboot it... 19:22:38 and we can do a proper cycle between beta and final. 19:23:23 sounds good. I have been doing hand updates on a couple of boxes which take a long time 19:23:45 so log01 should only have a short update this time versus the hour it took when I did it last week 19:24:19 cool. I will plan on that later today. 19:24:28 otherwise I still have piles of emails to get thru. 19:24:36 and rawhide compose to unbreak 19:25:40 smooge: you have any plans for the rest of today/tomorrow? 19:26:04 oh, and I am out friday again (this is the last one I planned I think) 19:26:09 so far my day has been nearly back to back meetings 19:26:33 so my next thing is to see what I wrote down as deliverables from those meetings and try to completely them 19:26:41 and wed I might need to drop a cat at the vet early and pick up later... or we might cancel that. should know tomorrow. 19:26:59 tomorrow will be whatever is needed to help Matt. I should also call IBM and Lenovo and see if I can get hardware repairs sent 19:27:25 so I hope he can power cycle things for us at least. :) 19:27:32 me too 19:28:04 I did have a question on the stg power8... were we going to have him cycle it and see if it came up at all? or just try and move another in it's place? moving another in would require a bunch of vlan changes... 19:28:10 I am going to have him remove the dead mustangs from the racks. [there are 3 which were never recovered. Then I was going to have the ones in 101 moved to the empty slots in 105 19:28:30 i was going to move another and list this one as dead 19:28:41 then get it and the other 2 dead ones sent back to IBM 19:28:56 I mean, we don't really know what happened to it. It just dropped off... it could just need a cycle... 19:28:57 which is what I need to add to my list of finding out how that is done 19:29:21 i thought you did power cycle it 19:29:34 but at this point my brain is mush.. 19:29:42 I'm not sure if I did or if we didn't have access to pdu's at that point. 19:29:55 that one is in 104 which hasn't had pdu changes yet 19:30:20 oh? I thought they all changed. ok. I can try power cycling it/finding it's serial then. 19:30:23 * nirik adds to list 19:30:23 but these are pdu's ... if you can get them to stay on the net for a day you are perfomring miracles 19:30:42 nirik, no 101, 102 and one pdu in 103 got changed. this week is 103/104/105 19:30:57 depending on when electricians and stuff show up 19:31:18 or at least that is what I got from Sunday's dm 19:31:25 ok. I just want access so we cna cycle things. ;) 19:32:09 understood. 19:32:20 ok, anything else? 19:32:26 #topic open floor 19:32:29 what are pdu´s 19:32:31 ? 19:32:37 power distribution units 19:32:41 #info nirik out on friday on 'vacation' 19:32:59 the long strips of power on the back of a cabinet that rack mounted systems are plugged into 19:33:00 yeah, basically power strips that are networked... 19:33:15 good question and another TLA 19:33:19 so you can get an interface to them to turn power on or off on particular sockets. ;) 19:34:04 if its working.. the chips in these things are weaker than a 2 dollar cell phone so we end up resetting them as much as the sockets 19:34:15 :) 19:34:23 but when it works.. we can fix a lot of hard hung systems 19:34:29 great question dtometzki... thanks for being here and asking. :) 19:34:41 ok, if nothing else will end in a minute 19:34:42 :-) 19:34:47 thanks copperi also 19:34:50 copperi++ 19:34:50 smooge: Karma for copperi changed to 3 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 19:34:55 dtometzki++ 19:34:55 smooge: Karma for dtometzki changed to 2 (for the current release cycle): https://badges.fedoraproject.org/tags/cookie/any 19:35:03 and wit that.. I am off to coffee land 19:35:03 thanks all. 19:35:08 #endmeeting