15:00:52 <mkonecny> #startmeeting Infrastructure (2020-09-24)
15:00:52 <zodbot> Meeting started Thu Sep 24 15:00:52 2020 UTC.
15:00:52 <zodbot> This meeting is logged and archived in a public location.
15:00:52 <zodbot> The chair is mkonecny. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:52 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:52 <zodbot> The meeting name has been set to 'infrastructure_(2020-09-24)'
15:00:52 <mkonecny> #meetingname infrastructure
15:00:52 <zodbot> The meeting name has been set to 'infrastructure'
15:00:52 <mkonecny> #chair nirik pingou smooge cverna mizdebsk mkonecny abompard siddharthvipul mobrien
15:00:52 <mkonecny> #info Agenda is at: https://board.net/p/fedora-infra
15:00:52 <mkonecny> #info About our team: https://docs.fedoraproject.org/en-US/cpe/
15:00:52 <zodbot> Current chairs: abompard cverna mizdebsk mkonecny mobrien nirik pingou siddharthvipul smooge
15:00:53 <mkonecny> #topic aloha
15:01:05 <mkonecny> .hello zlopez
15:01:05 <nirik> good morning everyone.
15:01:05 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@psmail.xyz>
15:01:09 <smooge> hrllo
15:01:11 <siddharthvipul> Goood morning \o
15:01:21 <mkonecny> Hi everyone, I hope you have a good day
15:01:50 <mobrien> .hello2
15:01:51 <zodbot> mobrien: mobrien 'Mark O'Brien' <markobri@redhat.com>
15:02:18 <siddharthvipul> .hello siddharthvipul1
15:02:19 <zodbot> siddharthvipul: siddharthvipul1 'Vipul Siddharth' <siddharthvipul1@gmail.com>
15:02:42 <mkonecny> Is everybody eager to see what today meeting has for us?
15:02:49 <nirik> you bet!
15:02:55 <siddharthvipul> OHH YEAHH
15:03:25 <mkonecny> So let's go to the first topic
15:03:51 <mkonecny> Who is here regularly already knows what will follow
15:03:54 <mkonecny> #topic New folks introductions
15:03:54 <mkonecny> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:03:54 <mkonecny> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:04:17 <mkonecny> And it is here, so anybody new wants to say hello?
15:05:38 <mkonecny> It looks like nobody new today
15:06:07 <mkonecny> Let's move to the next topic
15:06:41 <mkonecny> It's on fire and you have fun if you sit on it. What it is?
15:06:59 <mkonecny> #topic Next chair
15:07:00 <mkonecny> #info magic eight ball says:
15:07:00 <mkonecny> #info 2020-09-24 - mkonecny
15:07:00 <mkonecny> #info 2020-10-01 - nirik
15:07:00 <mkonecny> #info 2020-10-08 - mobrien
15:07:01 <mkonecny> #info 2020-10-15 - siddharthvipul
15:07:13 * nirik nods.
15:07:27 <mkonecny> As I see it's pretty full for next few weeks
15:07:31 <siddharthvipul> we are quite ahead here.. if we don't get some volunteer who is not a regular for this, I would recommend skipping
15:07:36 <siddharthvipul> (for next chair)
15:08:02 <mkonecny> siddharthvipul: Good idea, I was thinking the same
15:08:14 <siddharthvipul> :D
15:08:52 <mkonecny> So next topic will be something that will make your brain explode
15:09:06 <mkonecny> Some of you may already know what I'm talking about
15:09:17 <mkonecny> Let's go for it
15:09:23 <mkonecny> #topic announcements and information
15:09:23 <mkonecny> #info CPE Sustaining EU-hours team has a Monday through Friday 30 minute meeting going through tickets at 0830 UTC in #centos-meeting
15:09:23 <mkonecny> #info CPE Sustaining NA-hours team has a Monday through Friday 30 minute meeting going through tickets at 1800 UTC in #fedora-admin
15:09:23 <mkonecny> #info Datacenter move is over, but some items still need to be done: see https://fedoraproject.org/wiki/Infrastructure/2020-post-datacenter-move-known-issues
15:09:25 <mkonecny> #info F33 Beta freeze is in effect
15:10:09 <mkonecny> Do we want to announce anything else or is your brain already exploded :-D
15:11:02 <nirik> not much more to announce currently... oh...
15:11:11 <mkonecny> Don't be shy, just announce something
15:11:13 <nirik> #info F33 beta go/no-go is later this morning
15:11:45 <mkonecny> #info F33 beta go/no-go at 17:00 UTC
15:12:10 <smooge> #info smooge is back from PTO and causing problems
15:12:40 <mobrien> not problems, learning opportunities
15:12:58 <siddharthvipul> mobrien - the man of marketing :P
15:13:01 <siddharthvipul> haha
15:13:10 <mkonecny> We are looking forwards for the problems caused by smooge, last time the datacenter got moved :-)
15:14:12 <mkonecny> Forward to the next topic
15:14:36 <mkonecny> This one is the scary one, when nobody wants to volunteer
15:14:48 <mkonecny> Let's welcome
15:14:49 <mkonecny> #topic Oncall
15:14:49 <mkonecny> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:14:50 <mkonecny> #info mobrien is oncall for 2020-09-17 to 2020-09-24
15:14:50 <mkonecny> #info nirik is oncall for 2020-09-24 to 2020-10-01
15:14:50 <mkonecny> #info ??? is oncall for 2020-10-01 to 2020-10-08
15:14:52 * nirik plays halloween music
15:15:03 <nirik> .oncalltakeus
15:15:03 <zodbot> nirik: Kneel before zod!
15:15:14 <mkonecny> nirik: You are the next victim
15:15:26 <mkonecny> And who wants to go after him?
15:15:37 <mkonecny> Who is brave enough to take the oncall?
15:15:50 <siddharthvipul> mkonecny, I can
15:16:09 <mkonecny> It's all yours
15:16:19 <mkonecny> #info siddharthvipul is oncall for 2020-10-01 to 2020-10-08
15:16:52 <mkonecny> Let's look at what horrible things the current oncall saw
15:16:54 <mkonecny> #info Summary of last week: (from current oncall )
15:17:06 <mkonecny> mobrien: The floor is yours
15:17:10 <mobrien> Quiet week this week
15:17:19 <mobrien> just one thing really
15:17:34 <mobrien> ipsilon was giving 500 when trying to log in to AWS
15:17:55 <mobrien> a clearing of the browser cache solved it in this case
15:18:04 <mkonecny> Oh, this sounds scary
15:18:05 <mobrien> it is an intermittent problem
15:18:32 <nirik> I think it's a fas issue actually... it returns bad/incomplete data to ipsilon...
15:18:53 <mobrien> hopefully noggin will solve this then
15:19:20 <mobrien> Thats about it from on call
15:19:22 <nirik> yep
15:20:37 <mkonecny> Next on your meeting you can see horrible things, like servers getting hammered or unexpected outages
15:20:44 <mkonecny> But now
15:20:46 <mkonecny> #topic Monitoring discussion [nirik]
15:20:46 <mkonecny> #info https://nagios.fedoraproject.org/nagios
15:20:46 <mkonecny> #info Go over existing out items and fix
15:21:24 <nirik> fun. ;)
15:21:26 <nirik> lets see.
15:21:51 <nirik> so, we have some down aarch64 machines still... and some machines we need to get firewall rules changed for
15:22:19 <nirik> one question/discussion item that came up:
15:22:55 <nirik> Do we want to monitor staging from our production nagios? or do we want to make a staging one? or do we just want to wait until we redo monitoring and start in staging...
15:23:31 <mobrien> If we are definitely redoing monitoring then it makes sense to start in staging
15:23:33 <mkonecny> +1 for experimenting with new monitoring in staging
15:24:11 <nirik> yeah... just not sure when we are going to be doing it...
15:24:53 <nirik> anyhow, most of the rest are ones we have seen, just haven't fixed yet.
15:24:58 <nirik> although there is one...
15:25:08 <nirik> "Check proxies for oversubscription"
15:25:30 <nirik> I think this is monitoring fedmsg-relay?
15:25:51 <nirik> it seems to be alerting on all the new proxies (including some of the iad2 ones)
15:26:05 <nirik> would be nice to fix that or remove the check.
15:26:13 <mobrien> I think this may be related to the script being written in python2
15:26:26 <mobrien> could be wrong though ..
15:26:34 <nirik> oh wait... it's not fedmsg
15:26:36 <nirik> it's haproxy
15:26:49 <nirik> from proxy01 (where it is green): HAPROXY SUBS OK: 0.56% subscribed. 115 current of 20480 maxconn.
15:27:00 <nirik> yeah, could be indeed.
15:27:20 <nirik> mobrien: can you dig into it? I'm happy to help out...
15:27:34 <mobrien> Ya, sure
15:27:50 <nirik> cool.
15:28:10 <nirik> I'm happy to move on now, unless there's any alerts or monitoring anyone wants to discuss...
15:28:16 <mobrien> I'll have a look and may ping you later this evening for help/advice if needed
15:28:32 <smooge> advise: burn it with fire
15:28:53 <smooge> help: a can of kerosene, a box of matches, and some gun cotton
15:29:11 * nirik has come back from PTO spoiling for a fight. ;)
15:29:21 <nirik> sorry, that should be: smooge has...
15:29:30 <smooge> that will be you in a couple of weeks
15:29:46 <mobrien> just put if 1==1 everywhere in the script, should be fine
15:30:20 <smooge> and if it doesn't.. we really need to look at that system
15:30:22 <nirik> while true { exit 0; };
15:31:10 <mkonecny> If there isn't anything else, I would gladly take you on a journey
15:31:11 <smooge> ok I have nothing more.. my plan for staging monitoring was just to put in a rule for various templates that if a system was in the staging group.. skip it
15:31:34 <smooge> then the new monitoring can be built around what is in that group
15:32:09 <mkonecny> Journey to the depths of the infrastructure, the fearful place of proxies
15:32:21 <mkonecny> Let's give the floor to our guides
15:32:28 <mkonecny> # topic learning topic
15:32:29 <mkonecny> #info proxies [nirik/mobrien] for 2020-09-24
15:32:48 <mkonecny> #topic learning topic
15:32:48 <mkonecny> #info proxies [nirik/mobrien] for 2020-09-24
15:32:53 <mobrien> ok I can start here and nirik can fill in the blanks or correct me
15:33:02 <nirik> sounds good! go for it.
15:33:18 <mobrien> A number of the proxies in AWS (The none NA ones) are not yet in use, I have them up and configured they just need some DNS configuration to be put in place. I was hoping to have that done this morning but realised I missed something so got side tracked. They should be done soon.
15:33:49 <mobrien> so this explanation is assuming that work as done
15:33:57 <mobrien> We have proxies all over the world, a large number in our DC's in North America as well as some in APAC, EU, SA and AFR
15:34:25 <mobrien> When someone tries to hit any of our webapps covered by the proxies the will get directed to a proxy in their region which is determined by GeoIP and then will be either served static content from the proxy or redirected
15:35:13 <nirik> (or just proxied into a application via a vpn connection from the proxy back to our main datacenter)
15:35:23 <mobrien> Incoming traffic is dealt with by apache, there is some static web content synced to proxies at regular intervals (not sure what the interval are) from back end servers which is served or haproxy is used to direct any other requests
15:35:45 <mobrien> as nirik says those redirects are done over vpn
15:36:20 <mobrien> We use varnish to cache content as well
15:37:44 <mobrien> We have this which shows the status of backend services for each of our proxies https://admin.fedoraproject.org/haproxy/proxy01
15:37:59 <mobrien> just change the proxy number in the url to the desired proxy
15:38:40 <smooge> looks like we need to make a notifs-web02 and a mbs-frontend02 ?
15:39:01 <nirik> smooge: we could yeah.
15:39:12 <smooge> or take them out of haproxy?
15:39:22 <smooge> sorry.. I meant that for #fedora-noc
15:39:28 <smooge> mobrien, my apologies
15:39:33 <mobrien> no prob
15:39:43 <nirik> note on the haproxy page: we get complaints from time to time that we have a open http port there (not https). But I am not sure what we want to do about it or care...
15:40:33 <mobrien> I was wondering is all traffic between the backend servers an the proxies https?
15:40:58 <mobrien> If its over vpn it may not need to be
15:41:05 <nirik> not in all cases, nope. It's http over vpn
15:41:38 <mobrien> ok so I guess the proxies work as a TLS termination point for a lot of our backend servers too then?
15:42:03 <nirik> most of the time we terminate ssl on the proxies. Sometimes we do have https from proxy -> backend... like openshift apps do that
15:42:10 <nirik> yep
15:42:38 <mobrien> ok cool, was wondering that. I can't think of anything else off the top of my head
15:42:47 <mobrien> Anyone have any questions about it?
15:42:57 <mkonecny> #topic Learning topic discussion
15:42:58 <nirik> oh, one other thing: we also run on proxies our mirrorlist servers.
15:43:23 <nirik> it's a rust app. It gets new data every hour from mm-backend01 (pushed from there)
15:43:39 <smooge> it was a container.. then they made it better
15:44:12 * smooge is really looking for a fight today it would seem
15:45:24 <mkonecny> smooge is scarying plenty of people right now
15:45:47 <smooge> halloween (my favourite holiday is in 37 days)
15:46:10 <smooge> so I should put the scaring to next month
15:46:19 <mkonecny> Let's go to the final topic of today. The one you are looking forward to
15:46:22 <smooge> anyway.. back to looking at opendkim
15:46:27 <nirik> we have a even scarier thing in 39 days in the us, but I digress.
15:46:28 <mkonecny> #topic Open Floor
15:46:54 <mkonecny> There is one thing from siddharthvipul for the Open Floor
15:47:20 <mkonecny> #info Call for projects to participate in Hacktoberfest [siddharthvipul]
15:47:54 <nirik> that just involves marking the ticket/issue with #hacktoberfest? or ?
15:48:28 <siddharthvipul> nirik, yeah, pretty much! there is also an event that you can organise on October 1 to invite people and tell them more about the org
15:48:47 <siddharthvipul> that pulls contributors as well
15:49:00 <nirik> nice
15:49:49 <siddharthvipul> one issue I am thinking is that all the projects that we have on github are pretty mature and are not really beginner friendly..
15:49:50 <mkonecny> I will go through the issues for Anitya and see if I can mark something
15:49:58 <siddharthvipul> mkonecny, thank you
15:50:16 <nirik> does it have to be on github?
15:50:51 <siddharthvipul> yeah
15:50:54 <siddharthvipul> that's the issue
15:51:01 <nirik> ok.
15:51:17 <nirik> there was a ticket to update mote... might be in line with this
15:51:44 <nirik> https://github.com/fedora-infra/mote
15:51:59 <siddharthvipul> nirik, oh yes, that sounds great
15:52:03 <nirik> .ticket 9317
15:52:04 <zodbot> nirik: Issue #9317: Pull Request review and update deployment for Mote (IRC meeting log web app) - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9317
15:54:04 <siddharthvipul> so, request is to please tag some issues and if you can volunteer to talk about the Fedora project for 30 minutes on OCt 1, even better :)
15:54:54 * nirik nods
15:57:26 <siddharthvipul> that's all from me
15:57:32 <mkonecny> We are near the time that we have for this meeting
15:58:04 <mkonecny> I hope everybody had fun this time and I'm looking forward to see you in the future
15:58:18 <mkonecny> #endmeeting