18:00:11 <nirik> #startmeeting Infrastructure (2014-10-23)
18:00:11 <zodbot> Meeting started Thu Oct 23 18:00:11 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:11 <nirik> #meetingname infrastructure
18:00:11 <zodbot> The meeting name has been set to 'infrastructure'
18:00:11 <nirik> #topic aloha
18:00:11 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:11 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:14 * pingou 
18:00:19 * lmacken 
18:00:19 * lanica is here.
18:00:47 * threebean is here
18:01:35 * michel_slm is here
18:02:55 * relrod here
18:03:05 <nirik> #topic New folks introductions and Apprentice tasks
18:03:17 <nirik> any new folks like to say hi? or apprentices with questions or comments.
18:05:01 <pingou> I've seen a couple of people passing by asking for help to get started
18:05:13 <nirik> pingou: yeah, saw another one the other day too...
18:05:18 <pingou> but I've not mentionned the meeting, my bad :/
18:05:38 <nirik> well, hopefully they will find us.
18:05:43 <nirik> #topic Freeze reminder
18:05:51 <nirik> just a reminder we are still frozen... ;)
18:05:59 <nirik> #topic Applications status / discussion
18:06:04 <nirik> any applications news this week?
18:06:13 <pingou> well, anitya got few releases :)
18:06:32 <threebean> cool :)
18:06:46 <pingou> most things were in order to find out which project's are failing to retrieve a version
18:06:52 <lmacken> I've been working on a fedmsg-driven atomic ostree composer https://github.com/fedora-infra/fedmsg-atomic-composer
18:06:53 <pingou> which is now available at https://release-monitoring.org/projects/updates/failed
18:06:58 <pingou> lmacken++
18:07:08 <nirik> lmacken: cool.
18:07:12 * oddshocks pops in
18:08:00 <nirik> pingou: nice.
18:08:24 <pingou> the menu "projects" at the top gives you all the filters in place atm
18:08:40 <lmacken> pingou: very nice
18:08:50 <pingou> using these pages (and the keyboard shortcuts added), we went down from 2500+ projects failing to these 440 :)
18:09:06 <nirik> good progress.
18:09:09 <pingou> nirik: and the cron is now correctly set-up to use the lock :)
18:09:15 <oddshocks> lmacken: are we planning on using that compose machine to test the atomic consumer + MM1 setup?
18:09:32 <lmacken> oddshocks: potentially, I think threebean said nothing is using it right now
18:09:44 <lmacken> I'm almost ready to start composing trees in stg
18:09:48 <oddshocks> seems to make sense to me
18:09:52 <lmacken> yeah
18:10:03 <nirik> should be fine
18:10:21 <lmacken> +1 for installing MM1 on composer.stg for testing this week
18:10:26 <tflink> we found a problem with the libtaskotron build currently in production (botched merge left out a bunch of code) but we've fixed the build and it's currently running in dev and stg. may be requesting a freeze break in the next week to update the production clients
18:10:27 <lmacken> then we can blow it away later
18:10:31 * pingou looks forward the MM FAD
18:10:55 <nirik> tflink: sounds good.
18:11:12 <nirik> pingou: me too. ;)
18:11:42 <oddshocks> lmacken: do I need to request access to that box?
18:12:21 <lmacken> oddshocks: the only way to get in is sshing via root from lockbox at the moment. we probably want to change that for now.
18:12:36 <nirik> sure.
18:12:39 <oddshocks> lmacken: cool.
18:12:47 <pingou> oddshocks: do you have root on lockbox? :)
18:13:09 <oddshocks> pingou: I'm not sure, but I don't think so. I have used it to connect to other machines before though obviously
18:13:24 <threebean> yeah, we should open up just the staging one.  add a fas group..
18:13:37 * nirik nods.
18:13:40 <pingou> sysadmin?
18:13:41 <oddshocks> works for me
18:14:10 <nirik> well, sysadmin is a tracking group, not a shell group... so sysadmin-something...
18:14:52 <nirik> we can sort that out of meeting.
18:14:58 <nirik> anything else application wise?
18:15:02 <oddshocks> sysadmin-ostree or something perhaps, that we could apply to compose01.stg temporarily, and then later use it for any actual ostree/atomic machines we set up?
18:15:04 * oddshocks nods
18:16:04 <nirik> #topic Sysadmin status / discussion
18:16:17 <nirik> not too much in sysadmin land right now... since we are in freeze
18:16:47 <nirik> hopefully we will be done with beta soon.
18:17:45 <nirik> #topic nagios/alerts recap
18:17:49 <relrod> the new host1plus has rhel7 on it, I've started ansiblizing it
18:17:54 <nirik> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=10&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=10&eday=23&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=3&hoststates=7&servicestates=120&limit=25
18:17:56 <zodbot> nirik: http://tinyurl.com/l6fc4bl
18:18:05 <pingou> relrod: nice! :)
18:18:08 <nirik> relrod: oh yeah, thanks much for working on it. ;) it's pretty much set now? or ?
18:18:20 <nirik> ideally I would like to add a proxy and a mirrorlists there.
18:18:23 <nirik> (at least)
18:18:44 <relrod> nirik: pretty much - need to finish ansibilizing it (was waiting for +1's, but got those yesterday), then can set up some VMs on it.
18:19:00 <nirik> great!
18:19:31 <nirik> so, on nagios... I think we wanted to just kill the nuancier messages...
18:19:33 <smooge> I got email that we need to know what boxes we want warranty extensions and which ones need 24x7x4 hour service on. so that is my rest of my day
18:19:39 <smooge> oops sorry
18:19:51 <smooge> so nagios...
18:19:55 <nirik> smooge: yeah good to check everything and see if we aren't missing any
18:20:42 <nirik> there's some bodhi slowness lately, but I am really not sure why... nothing much in logs.
18:21:27 <lmacken> yeah, nothing's changed in that code in a while
18:21:41 <lmacken> so, it's probably with the db, or to fas, since that happens during most requests
18:21:43 <nirik> I guess we could try just rebooting those instances...
18:22:19 <lmacken> there have also been random connection dropouts between taskotron and bodhi
18:23:08 <threebean> +1 on killing the nuancier datanommer check.
18:23:27 <pingou> +1
18:23:28 <threebean> .. wait until after freeze for it?
18:23:49 <nirik> something like smokeping or page load graphing might help us see if it's some time...
18:23:59 <nirik> yeah, probibly fine to just wait now
18:24:07 <smooge> what boxes are we wanting on a smokeping?
18:24:42 <threebean> lmacken: we have a collectd/web-service role in ansible you might be able to use to graph the time to load the bodhi frontpage.
18:24:44 <nirik> some bodhi page... see if it's always slow or gets slow at specific times, etc.
18:25:05 <tflink> lmacken: one of the fixes that was supposed to be in taskotron production was retries for errors on bodhi read-only operations
18:25:12 <lmacken> threebean: that would be real interesting
18:26:00 <lmacken> tflink: that's frustrating
18:26:16 <lmacken> the errors are connection dropouts before the client can finish getting the response body
18:26:25 <nirik> anyhow, more investigation would be good.
18:27:31 <nirik> any other sysadmin side things?
18:28:09 <nirik> #topic Upcoming Tasks/Items
18:28:09 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:28:21 <nirik> anything upcoming anyone would like to schedule or note?
18:29:22 <nirik> we should add the FAD to that calendar ;)
18:29:26 <pingou> +1
18:29:41 <nirik> #topic Open Floor
18:29:49 <pingou> for the FAD, we'll need to plan the hotel room at one point
18:29:54 <nirik> anyone have anything for open floor?
18:30:03 <pingou> and please update the wiki page when you know your travel details :)
18:30:04 <nirik> pingou: oh yeah... also when do we get plane tickets?
18:30:07 <smooge> not me
18:30:21 <pingou> nirik: I stickster_afk is the one that will book them
18:30:44 <pingou> I'll ping Matt to know if he received well my last email
18:30:45 <nirik> ah right. so we need to gather info to send to him...
18:30:46 <smooge> oh a couple of things
18:30:54 <pingou> nirik: likely yes
18:31:33 <smooge> I will be paying for  most of the food at the FAD and some of the hotel rooms
18:31:51 <pingou> smooge: you're my new best friend :D
18:32:20 <smooge> as I have to book my flight seperately and it makes filing those expenses possible
18:32:42 <nirik> cool. Thanks smooge
18:32:48 <smooge> pingou, what is the webpage for the fad again?
18:33:04 <pingou> smooge: https://fedoraproject.org/wiki/FAD_MirrorManager2_ansible-migration_2014
18:35:24 <nirik> smooge: you had another thing? or that was it?
18:39:15 <smooge> that was it
18:39:23 <nirik> cool.
18:39:35 <nirik> anything else from anyone? or shall we call it a meeting?
18:40:15 <pingou> smooge: https://fedoraproject.org/wiki/FAD_MirrorManager2_ansible-migration_2014#Budget_Realized if you already booked your tickets :)
18:40:56 <smooge> will be doing so in a short while. I need to get hardware dealt with first
18:41:00 <pingou> sure
18:41:13 <nirik> alright. Thanks for coming everyone!
18:41:16 <nirik> #endmeeting