18:00:23 <smooge> #startmeeting Infrastructure (2018-02-22)
18:00:23 <zodbot> Meeting started Thu Feb 22 18:00:23 2018 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:23 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:23 <zodbot> The meeting name has been set to 'infrastructure_(2018-02-22)'
18:00:23 <smooge> #meetingname infrastructure
18:00:23 <smooge> #topic aloha
18:00:23 <smooge> #chair smooge relrod nirik dgilmore threebean pingou puiterwijk pbrobinson maxamillion
18:00:23 <zodbot> The meeting name has been set to 'infrastructure'
18:00:23 <zodbot> Current chairs: dgilmore maxamillion nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:32 <puiterwijk> .hello2
18:00:33 <zodbot> puiterwijk: puiterwijk 'Patrick "マルタインアンドレアス" Uiterwijk' <puiterwijk@redhat.com>
18:00:45 <tflink> .hello2
18:00:46 <zodbot> tflink: tflink 'Tim Flink' <tflink@redhat.com>
18:00:57 <pingou> .hello pingou
18:00:58 <zodbot> pingou: pingou 'Pierre-YvesChibon' <pingou@pingoured.fr>
18:00:59 <puiterwijk> ay Tim!
18:01:17 <smooge> .hello2
18:01:18 <zodbot> smooge: smooge 'Stephen J Smoogen' <smooge@gmail.com>
18:01:24 <keitellf> .hello2
18:01:26 <zodbot> keitellf: keitellf 'Lee Keitel' <lee@onesimussystems.com>
18:01:27 <cverna> hello o/
18:02:07 <keitellf> How's everyone?
18:02:35 <smooge> hello
18:02:42 <smooge> ok nirik is running a bit late
18:02:44 * relrod waves
18:02:51 * sayan is here
18:03:06 <smooge> #topic New folks introductions
18:03:06 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
18:03:55 <smooge> hello to anyone new
18:04:26 <smooge> #chair tflink
18:04:26 <zodbot> Current chairs: dgilmore maxamillion nirik pbrobinson pingou puiterwijk relrod smooge tflink threebean
18:06:20 <smooge> ok next up then
18:06:41 <smooge> #topic announcements and information
18:06:41 <smooge> #info Infrastructure hackfest in april: https://fedoraproject.org/wiki/Infrastructure_Hackathon_2018
18:06:41 <smooge> #info Debugging 503's on mirrormanager dropped from 80,000 to 10,000 day (thanks nirik)
18:06:41 <smooge> #info Rebuild mirrormanager container to be more resilient
18:06:54 <smooge> There will be a hackfest in April
18:09:25 <smooge> We have a problem with mirrormanager giving out 503's under heavy load. Out of 135 million requests we were seeing 80-100k failures
18:09:26 <lorbus> .hello2
18:09:27 <zodbot> lorbus: lorbus 'Christian Glombek' <c@petersen-glombek.de>
18:09:45 <smooge> we are looking at how to deal with this a bit better
18:10:05 <keitellf> do we know what's actually causing the errors?
18:10:09 <puiterwijk> I think that the mirrormanager stuff was related to how we restart things, and with some changes we got it significantly down, right?
18:10:53 <smooge> keitellf, so the errors occur usually on the start of the hour from :00 -> :15 after
18:11:17 <smooge> this is also when 50% of all mirror requests come in per day
18:11:36 <keitellf> That makes sense, cronjobs.
18:12:17 <smooge> as puiterwijk said part of the problem was that it was related to how we restart things, and part of it was the number of dockers we had at a time.
18:12:50 <smooge> changing those helped bring it down, but we have 2 hosts which consistently show multiple thousands during the 05:00 -> 05:15 UTC time
18:13:44 <smooge> that is the busiest hour of the day
18:14:18 <smooge> so next steps are to look at how many threads the containers have and how many are running at that time frame
18:14:30 <smooge> and moving one of the proxies to a hopefully faster box
18:15:23 <smooge> ok any other announcements for the last week?
18:16:15 <smooge> #topic Apprentice Open office minutes
18:16:33 <smooge> #info A place where apprentices can get help on any tickets or problems they are having
18:17:53 <smooge> Any questions here?
18:18:44 <smooge> Alright then..
18:18:47 <smooge> #topic Open Floor
18:19:14 <smooge> OK I would like to get people to send stuff they would like talked about or updated to the list for next meeting
18:20:16 <smooge> Anything else ?
18:20:31 <smooge> .. is never sure if he is in a netsplit or everyone is being quiet
18:21:12 <smooge> ok I am going to say "thank you all for coming today"
18:21:14 <keitellf> ..crickets
18:21:25 <smooge> and I hope we have more next meeting
18:21:28 <smooge> #endmeeting