17:59:34 #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 17:59:34 Meeting started Wed Sep 2 17:59:34 2020 UTC. 17:59:34 This meeting is logged and archived in a public location. 17:59:34 The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:59:34 Useful Commands: #action #agreed #halp #info #idea #link #topic. 17:59:34 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 17:59:34 #chair mboddu nirik smooge pingou mobrien 17:59:34 #meetingname fedora_infrastructure_ops_daily_standup_meeting 17:59:34 #info meeting is 30 minutes MAX. At the end of 30, its stops 17:59:34 Current chairs: mboddu mobrien nirik pingou smooge 17:59:34 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 17:59:34 #info agenda is at https://board.net/p/fedora-infra-daily 17:59:35 #topic Tickets needing review 17:59:37 #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 17:59:41 oops... I am early 17:59:43 Hello 17:59:59 Well, we can close early :D 18:00:05 hello 18:01:00 hey. sure... someone want to mod tickets? 18:01:49 or I can and someone can list them? whatever works. :) 18:02:06 I can do it 18:02:17 ^ Updating tickets 18:02:20 .ticket 9292 18:02:21 nirik: Issue #9292: Slow transfer speeds from fedoraproject servers - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9292 18:02:32 I think smooge was working on this... not sure how much we can really do here. 18:02:47 Put everything in cloud :D 18:03:08 nirik, at this point I think it is close can't fix but I was wanting to see if anyone had any ideas 18:03:10 no, no, then you still have to get to them... instead, it will be the next new thing: 18:03:16 _networkless!_ 18:03:25 lol 18:03:46 pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#9292: high-trouble, medium-gain, and ops https://pagure.io/fedora-infrastructure/issue/9292 18:03:47 pagure.issue.assigned.added -- smooge assigned ticket fedora-infrastructure#9292 to smooge https://pagure.io/fedora-infrastructure/issue/9292 18:03:48 pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#9292 https://pagure.io/fedora-infrastructure/issue/9292 18:04:07 Thanks smooge 18:04:09 nirik, mboddu do you guys have any ideas? 18:04:45 No idea, but I am trying whats are the speeds on my end 18:04:47 I don't really. I guess asking networking folks to see if theres any known upstream issues, or if they can advertise us differently to go over another provider 18:05:17 nirik, we have 2 network links. one is via cogentco and the other is via the one which is at 3/4 use 18:06:06 I asked upstream and they looked at the network link and the cogentco is pretty much full. The other one is close. 18:06:13 ugh. 18:06:29 so we don't have the same BW as we did in phx2? 18:06:34 we have more 18:06:41 * nirik is going over the cogentco link. 18:06:48 as its dedicated to us 18:07:17 or at least that is what I think we have.. 18:07:22 huh, so we must be using more? I wonder where.... 18:07:55 or maybe we have less 18:08:06 I don't know at this point.. and should just ask 18:08:25 yeah, I'd say lets ask... can't hurt. 18:08:31 and would be good to know exactly 18:09:22 so, high trouble / high gain / ops/ 18:09:24 ? 18:09:33 +1 18:09:41 FWIW, I am hitting 20+ MBPS 18:09:53 you must not be using the cogentco link 18:10:13 my downloads start at 3-5MB/sec, then drop down to about 500k 18:11:03 anyhow... 18:11:16 .ticket 9293 18:11:17 nirik: Issue #9293: Permission for Noggin/AAA team members to auth services in staging - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9293 18:11:36 I'd say close this with pingou willing to get them access/run playbooks... 18:11:57 otherwise we can add people to groups to allow various playbooks as we know which ones they are 18:12:25 pagure.issue.edit -- smooge edited the close_status and status fields of ticket fedora-infrastructure#9293 https://pagure.io/fedora-infrastructure/issue/9293 18:12:26 pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#9293: "Permission for Noggin/AAA team members to auth services in staging" https://pagure.io/fedora-infrastructure/issue/9293#comment-675368 18:12:33 +1 18:12:48 done.. [sorry was writing an email to ask about bandwidth] 18:12:51 I am hitting cognentco 18:12:53 there's one private ticket, but we will just handle that in ticket 18:13:05 8.|-- be3472.ccr42.jfk02.atlas.cogentco.com 0.0% 50 17.0 17.9 14.9 27.9 2.4 18:13:05 9.|-- be2807.ccr42.dca01.atlas.cogentco.com 0.0% 50 29.9 25.0 22.1 32.0 2.3 18:13:05 10.|-- be3084.ccr41.iad02.atlas.cogentco.com 0.0% 50 25.0 23.9 21.8 35.1 2.1 18:13:18 Anyway, we passed ahead of it 18:13:36 huh, ok. 18:13:44 any releng tickets? 18:14:02 well in that case I am expecting it is a link firther out thant be2807.ccr42.dca01.atlas.cogentco.com which is the bottleneck 18:14:17 There is an unretirement ticket, low,low,ops. 18:14:37 +1 18:14:43 .fasinfo andreamanzi 18:14:44 mboddu: User: andreamanzi, Name: Andrea Manzi, email: andrea.manzi@gmail.com, Creation: 2012-12-02, IRC Nick: None, Timezone: UTC, Locale: en, GPG key ID: None, Status: active 18:14:47 mboddu: Approved Groups: cla_done cla_fpca packager fedorabugs 18:15:20 I thought it might not be even low :D 18:15:43 Thats all I have 18:16:37 cool. Any other PR's, freeze breaks, issues, tickets or poems? 18:16:50 .ticket 9250 18:16:53 smooge: Issue #9250: Mails from copr-fe-dev.aws.fedoraproject.org being marked as spam - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9250 18:17:27 ok so it looks like the way spf is set up needs to be updated 18:17:33 yeah, I do not want to add all of amazon to our stuff. 18:17:50 or all of google 18:18:19 I wonder how it changes (if any) if they just sent it direct instead of via us? 18:18:51 if they send it direct they are not neutral anymore in the spf because we do not authorize that zone 18:19:09 at the moment they are 'neutral' and anything not sending via bastion is 'bad' 18:19:14 or whatever the term is 18:19:45 I am thinking that what might be needed is putting an spf record in the aws.fedoraproject.org dns zone 18:20:05 but I am going to need to read up on it and figure it out.. 18:20:14 or for that host specifically. 18:20:15 and in 48 hours I turn into a pumpkin 18:20:18 not the entire thing. 18:20:44 I don't see how we need google in there at all. 18:21:03 me either since we don't do anything with google 18:22:14 ok, yeah, I guess it does need to be for the domain... aws.fedoraproject.org... adding that host as a permitted sender 18:22:44 anyway I am going to put some changes in today.. and then I am going to drop the ticket so someone else can work on it while I am on PTO 18:23:40 we could also I suppose masquerade that host... ie, they send to bastion and we re-write it to be from 'copr@fedoraproject.org' or something... which might have a higher reputation (or not?) 18:23:46 sounds good 18:24:19 ok spf does not inherit.. so I need to duplicate that record in each of our subzones 18:24:35 so that may help part 18:25:01 aka aws.fedoraproject.org needs its own spf record even if the mail goes through bastion which is ok for fedoraproject.org 18:25:02 I'd say just add it to aws for now 18:25:06 yep 18:25:12 because SPF is horrible. ;) 18:25:44 ok, anything else before we close out? 18:27:08 not from me 18:27:13 #endmeeting