18:01:37 <smooge> #startmeeting Infrastructure
18:01:37 <zodbot> Meeting started Thu Sep 18 18:01:37 2014 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:01:37 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:01:49 <smooge> #meetingname fedora-infrastructure
18:01:49 <zodbot> The meeting name has been set to 'fedora-infrastructure'
18:02:02 <smooge> #chairs nirik smooge
18:02:13 * nirik waves from his slow slow network connection
18:02:18 <smooge> #topic Please say hi
18:02:24 * bochecha waves
18:02:34 <smooge> Sorry for the delay.. one of the Dell dracs is not responding to any passwords
18:03:35 <smooge> #topic  New folks introductions and Apprentice tasks.
18:03:53 * oddshocks here
18:04:03 <Neldogz> here
18:04:07 * relrod here
18:04:11 <Neldogz> neldogz is here
18:04:12 <smooge> Any new people wanting to become apprentices and such?
18:04:15 * tflink here
18:04:18 <smooge> hi Neldogz
18:05:02 <smooge> Any apprentices with problems they need to discuss?
18:06:33 <smooge> ok so next topic
18:06:41 <smooge> #topic Freeze reminder
18:07:27 <smooge> we are still in Alpha freeze. All changes on frozen machines should go to infrastructure@lists.fedoraproject.org and get two +1.
18:08:14 * nirik nods.
18:08:23 <nirik> hopefully we will be leaving freeze next week, but we will see
18:08:41 <smooge> use the freezelist script in ansible to determine if the machine is frozen
18:08:57 <smooge> the decision for shipping an alpha is in #fedora-meeting-2 at the moment
18:09:20 <smooge> #topic Applications status / discussion
18:10:01 <nirik> many of our app developers are out this week. ;)
18:10:09 <smooge> okie dokie
18:10:21 <smooge> #topic Sysadmin status / discussion
18:10:41 <smooge> nirik is on travel and connecting via tin cans in New Hampshire
18:11:03 <smooge> I have been cleaning up various files on lockbox01 so we could have enough disk space
18:11:41 <smooge> I am currently working on getting fed-cloud10 and fed-cloud11 up. fed-cloud10 has decided it doesn't like any of our passwords so I need to get a hands on to fix it.
18:11:50 <nirik> Our new cloud is coming along nicely thanks to msuchy.
18:11:58 <smooge> +1
18:11:58 <nirik> hopefully soon we will have it operational
18:12:13 * threebean arrives late
18:12:24 <smooge> I also want to say that puiterwijk did a LOT of after hours work this week to get a dead download server back up.
18:12:31 <smooge> threebean, any application items?
18:12:39 * tflink is still seeing a large number of failures in taskotron due to bodhi issues
18:12:55 <threebean> new fmn release went out earlier this week as promised last week
18:12:57 <threebean> http://threebean.org/blog/new-fmn-release/
18:13:02 <nirik> yeah, great work puiterwijk. ;)
18:13:13 <tflink> still don't know what's causing it, we're planning to implement a better error handling and retry mecahnism to work around the read-only issues to reduce the number of failed jobs
18:13:19 <nirik> tflink: thats really weird. did it seem like all of them were on bodhi02?
18:13:20 <threebean> ran into some OOM issues.  I reduced the # of threads for now but maybe after freeze we can increase the ram on that box.
18:14:03 <smooge> #info various people are on travel
18:14:06 <tflink> nirik: not sure, the server isn't logged in errors and I didn't correlate the errors with access on bodhi01/02
18:14:12 <smooge> #info OOM issues on fmn
18:14:41 <nirik> tflink: ok. .
18:14:41 <smooge> threebean, actually I am an idiot. what is fmn exactly fedora messaging network?
18:15:12 <threebean> that sounds good :P
18:15:29 <threebean> I think the idea originally was FedMsg Notifications
18:15:45 <threebean> but I've been calling it just "Fedora Notifications" (which doesn't map to fmn...)
18:15:48 <tflink> nirik: I'll look into it after the meeting
18:16:12 <smooge> threebean, no problem I knew of Fedora Notifications and thought it was something that ran on fmn.
18:17:04 <threebean> ah, cool.  yeah, it is just this thing https://apps.fedoraproject.org/notifications
18:17:23 <smooge> oh and another application item is that there are some mirrormanager items for release
18:19:57 <threebean> sorry, what does that mean?
18:20:02 <threebean> is that something that needs to be picked up?
18:22:46 <smooge> threebean, there were some items dgilmore was working on in mirrormanager. I don't know anything beyond that.
18:23:12 <smooge> since mirrormanager is an app and frozen I figured I should note it.
18:23:20 <threebean> ok
18:25:21 <smooge> #topic nagios/alerts recap
18:25:28 <smooge> ok we have quite a few of these.
18:26:51 <smooge> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=8&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=8&eday=28&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=3&hoststates=7&servicestates=120&limit=25
18:26:51 <zodbot> smooge: http://tinyurl.com/m3v3a5x
18:27:15 <smooge> sorry my paste kept putting in the wrong thing..
18:28:33 <threebean> huh, I hadn't realized that datagrepper-db-query was complaining.
18:28:51 <smooge> I think they went with the bodhi item but I am not sure
18:29:54 * threebean will look into increasing the timeout on that
18:30:06 <smooge> ok
18:30:41 <smooge> thanks
18:30:46 <threebean> yeah, np :)
18:30:52 <smooge> #topic Upcoming Tasks/Items
18:31:08 <smooge> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:32:46 <smooge> I guess I will need to edit this in a bit to add in that we look to have an Alpha for next week
18:32:59 <smooge> nirik, that was the outcome correct?
18:34:06 <smooge> I am guessing CDMA ate his link again :)
18:34:23 <smooge> #topic Open Floor
18:34:31 <smooge> Ok anything for the open floor?
18:35:20 <bochecha> I have a few ansible patches attached to https://fedorahosted.org/fedora-infrastructure/ticket/4452 , need someone to review/commit :)
18:36:05 <bochecha> other than that, I probably won't be able to make the meetings any more, starting from next week :(
18:36:16 <smooge> oh dear
18:36:20 <smooge> everything ok?
18:36:46 <smooge> bochecha, I will review the items after/while I deal with the misbehaving idrac
18:36:51 <smooge> thanks for the ticket number
18:36:53 <bochecha> sure, it's just that it's at 8pm here, which won't be a good time any more starting next week ;)
18:37:03 <smooge> ah I understand.
18:37:27 <smooge> well keep in touch when you can
18:37:47 <threebean> bochecha: :)
18:38:00 <bochecha> sure :)
18:38:24 * nirik is back online again
18:40:18 <smooge> nirik, so the only item was it looks like we ship alpha next week from the other meeting but wasn't sure.
18:40:24 <smooge> after that I was going to endmeeting
18:40:36 * nirik nods.
18:40:43 <smooge> ok I am going to end meeting in 1 minute.
18:41:35 <smooge> #endmeeting