18:02:43 <nirik> #startmeeting Infrastructure (2014-09-25)
18:02:43 <zodbot> Meeting started Thu Sep 25 18:02:43 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:02:43 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:02:43 <nirik> #meetingname infrastructure
18:02:43 <nirik> #topic aloha
18:02:43 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:02:43 <zodbot> The meeting name has been set to 'infrastructure'
18:02:43 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:02:51 * lanica is here for the infra meeting.
18:02:55 <threebean> oy
18:02:55 * tflink is here
18:03:00 <janeznemanic> hi
18:03:04 * puiterwijk here
18:03:06 <Neldogz> neldogz is here
18:03:16 <webpigeon> o/
18:03:25 * lmacken 
18:03:30 * pingou here
18:03:32 * bitlord here, listening \o ;-)
18:04:17 * danielbruno here
18:04:24 <nirik> #topic New folks introductions and Apprentice tasks
18:04:29 <nirik> any new folks like to introduce themselves?
18:04:34 <nirik> or apprentices with questions or comments?
18:04:50 * oddshocks arrives enveloped in a cloud of crows
18:04:57 * relrod arrives too
18:05:03 <danielbruno> !
18:05:46 <smooge> >:}
18:06:09 <nirik> danielbruno: feel free to just chime in. we don't use that meeting protocol thing.
18:06:34 <ssf87> no agenda ?
18:06:38 <danielbruno> nirik, I saw yout reply on the ticket about the planet to track broken feeds
18:06:55 <nirik> ssf87: yes there is. It was sent to the mailing list...
18:07:02 <danielbruno> I would ike to know if I need some permission
18:07:17 <danielbruno> to access .planet files
18:07:27 <nirik> danielbruno: you shouldn't. :) everyones /home/fedora/*/.planet files should be readable.
18:07:41 <danielbruno> great!
18:07:44 <nirik> the planet job runs as nobody so if it can read it you should be able to too
18:08:10 <danielbruno> nirik, thank you :)
18:08:22 <nirik> no problem, do let me know if you run into any problems with it.
18:09:18 <nirik> any other new folks or apprentice questions/comments?
18:09:52 <nirik> #topic Applications status / discussion
18:10:06 <nirik> any application news this week? threebean / lmacken / pingou / oddshocks ?
18:10:15 <threebean> we're having issues with fedmsg load on datanommer right now
18:10:16 <threebean> http://threebean.org/fedmsg-health.html
18:10:31 <lmacken> I did a couple of bodhi updates in production this week. Nothing too exciting, mainly EL7 related stuff.
18:10:35 <nirik> threebean: is that db on rhel7 yet? or still 6?
18:10:37 <puiterwijk> I'd like to at least throw in the public again that fedoauth will be deprecated in the future. not yet sure when
18:10:38 <threebean> it's a result I think of us outgrowing our current one-big-postgres-table setup and we're researching alternatives
18:10:46 <pingou> I've been working on a couple of requests from gnokii for nuancier, review pending
18:10:59 <pingou> I also have some reviews pending for pkgdb2, closing some tickets
18:11:12 <nirik> #info fedmsg load issues with datanommer, being worked on.
18:11:25 <nirik> #info some bodhi1 updates in production around epel7 stuff
18:11:28 <pingou> and I added support to send the reminder emails to multiple addresses on fedocal, merged and planned in the next release
18:11:31 <threebean> nirik: still on 6.5
18:11:33 <lmacken> threebean: it'd be real interesting to see those benchmarks that you wrote earlier between RHEL 6 & 7
18:11:43 <nirik> #info fedoauth will be depreciated at some time to be scheduled yet.
18:11:45 <puiterwijk> #link https://lists.fedoraproject.org/pipermail/infrastructure/2014-September/014863.html
18:12:02 <nirik> #info some nuancier patches pending review.
18:12:23 <nirik> threebean: wonder if this would be a good time to move it to 7? or it needs more than just a newer postgres?
18:12:28 * threebean nods
18:12:36 <threebean> let's move it up this afternoon.
18:12:47 <threebean> I can update moksha and the collectd stuff on busgateway01 at the same time.
18:13:15 <relrod> If anyone uses HRF (hrf.cloud.fp.o) for anything, please switch to datagrepper instead. I want to deprecate HRF because datagrepper can do everything HRF was made to do now.
18:13:37 <nirik> so, would we want to spin a new 7 one up and transfer, or just save the db off and destroy the 6 one and recreate it as 7?
18:13:50 <threebean> the first option sounds safest :)
18:14:08 <nirik> #info hrf.cloud.fedoraproject.org depreciated. Please don't use it anymore, use datagrepper.
18:14:24 <nirik> threebean: yeah, just needs allocating ip's, etc... more infra work, but not too big a deal.
18:14:48 <tflink> taskotron production is making progress - initial systems are set up, proxies are configured, seems to be mostly working
18:15:04 <tflink> still need to get monitoring and backups set up
18:15:20 <nirik> tflink: saw the monitoring ticket, but haven't had a chance to do anything with it.
18:15:28 <tflink> would like to see it funcitoning for several days and a couple of fixes in place before turning off autoqa
18:15:28 <nirik> might be a good place for a new person to jump in. ;)
18:15:33 <nirik> yeah.
18:15:48 <tflink> SOPs have been written for taskotron and resultsdb
18:15:57 <nirik> #info taskotron almost in production. Needs monitoring and backups and a few days of good smooth operation.
18:17:20 <nirik> cool. Any other applications news?
18:17:30 <nirik> anything we want to make sure to try and get done before beta freeze?
18:18:06 <pingou> MM2? :D
18:18:08 * pingou ducks
18:18:11 <oddshocks> F21 RC1 AMIs are listed on my fpeople page, still debugging 32 bit base and 64 bit atomic: https://oddshocks.fedorapeople.org/
18:18:18 <oddshocks> but 64 bit base are there
18:18:33 <nirik> oddshocks: nice. Should those get updated in the website?
18:18:42 <nirik> pingou: ha.
18:19:11 <puiterwijk> pingou: I hear you're volunteering to get it done by then? :)
18:19:11 <oddshocks> nirik: I think robyduck is waiting to update the links until i have the other two sets working, but i'm not sure if that's the same thing
18:19:22 <pingou> we might manage to have anitya running before beta
18:19:24 <nirik> oddshocks: ok, sounds reasonable.
18:19:31 <pingou> but that's not quite part of our infra anyway :)
18:19:42 <pingou> puiterwijk: I was merly proposing you :-p
18:19:49 <nirik> also, as a side note, I put f21alpha cloud image into our cloud... so if we need any f21alpha instances it should be ready.
18:20:02 <nirik> we could add a jenkins one perhaps.
18:20:46 <nirik> which reminds me... pingou: what was the conclusion about making jenkins a more supported/supportable service? no go since it's not packaged for rhel? or ?
18:21:11 <pingou> nirik: not packaged for rhel was/is the biggest blocker
18:21:32 <nirik> yeah. wonder if even 7 wouldn't work... it should have new enough stuff I would think
18:21:54 <pingou> depends on jenkins' deps, which I suspect is big
18:22:02 <puiterwijk> yeah, jenkins dep tree is quite big
18:22:04 <nirik> the docs folks were looking at a jenkins plugin for publishing and serving docs.
18:22:18 <puiterwijk> maybe we should get someone to just go ahead and package it all
18:22:28 <pingou> yes, I read the backlog of this
18:22:41 <pingou> puiterwijk: that would imply we are comitted to maintain it
18:22:50 <puiterwijk> pingou: yeah
18:23:10 <nirik> pingou: do you recall who we were talking to about it at flock?
18:23:29 <pingou> nirik: I remember the face, but not the name
18:23:32 <pingou> :/
18:23:34 <nirik> yeah, same here. ;(
18:23:57 <nirik> oh well, we don't need to do anything right now, but something to think on.
18:24:13 <nirik> #topic Sysadmin status / discussion
18:24:24 <smooge> hi
18:24:26 <puiterwijk> #info CVE-2014-6271/CVE-2014-7169 (Bash issues): Patches for 6271 and workaround for 7169 applied on high profile servers
18:24:28 <nirik> so, we left freeze yesterday... just in time for some security update fun. ;)
18:24:34 <smooge> yay!
18:24:40 <nirik> hey smooge
18:24:49 <nirik> puiterwijk: you also updated mediawiki right?
18:25:00 <puiterwijk> nirik: yup. we're now at the latest one, released yesterday
18:25:13 <puiterwijk> #info Mediawiki upgraded to 1.19.19
18:25:38 <smooge> yay!
18:26:15 <webpigeon> \0/
18:26:15 <nirik> cool.
18:26:28 <puiterwijk> and I'm packaging the 1.23 series
18:26:41 <puiterwijk> (since we need to upgrade before May 2015, since that's EOL for 1.19)
18:27:16 <nirik> yep.
18:27:32 <nirik> I've been working thru backlog of tickets that landed when I was traveling...
18:27:43 <nirik> hopefully we will be back to normal on those before too long.
18:27:53 <nirik> I'd also like to look at scheduling a mass reboot cycle next week.
18:27:58 <nirik> Probibly wed or so...
18:29:00 <nirik> #info mass reboot likely next week to catch up on updates
18:29:17 <nirik> #topic nagios/alerts recap
18:29:26 * nirik looks for the url again.
18:30:04 <puiterwijk> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=9&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=9&eday=4&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=3&hoststates=7&servicestates=120&limit=25
18:30:04 <zodbot> puiterwijk: http://tinyurl.com/l3vjae8
18:30:11 <puiterwijk> nirik: ^
18:30:17 <smooge> and again puiterwijk comes through.
18:30:21 <nirik> beat me to it. ;)
18:30:57 <puiterwijk> nirik: or  better: http://da.gd/fednagios
18:30:59 <nirik> so, the datagrepper issues we are aware of
18:31:45 <nirik> not sure about the collab mail queue. I don't recall seeing those?
18:31:50 <nirik> so must have been warnings
18:31:57 * oddshocks read that as http://da.gd/fadingos
18:31:57 <lmacken> packages03 might need some more memory :\ I spent a little bit of time this week poking some unicode issues with the xapian db, but haven't looked at prod
18:32:25 <nirik> lmacken: yeah, it's gotten stuck a lot lately. It also had a issue this week where OOM killed glusterd. ;(
18:32:31 <nirik> we can bump it up some more.
18:33:01 <nirik> 8gb -> 12?
18:33:14 <lmacken> that sounds fine
18:33:47 <lmacken> also, packages.stg can't get to dl.fp.o:80 for some reason. Need to add a firewall rule?
18:33:53 * nirik can do so after the meeting.
18:33:54 <lmacken> it can ping it
18:34:12 <nirik> is it using internal or external ip?
18:34:36 <lmacken> I think it was hardcoded to download03.phx2
18:34:37 <puiterwijk> lmacken: yeah, stg -> prod is denied with internal IP now, thanks to threebean's blanket rule
18:34:56 <lmacken> okay, cool
18:34:57 <puiterwijk> so it'd need to use external IP to have any chance, or a seperate firewall rule
18:35:07 <nirik> yeah.
18:35:19 <puiterwijk> (I vote for external IP, I rather like the stg->prod firewall rule)
18:35:29 <nirik> if external works, sure.
18:37:01 <nirik> also, re: nagios... I am going to kill unbound-telia01 soon... and possibly kill mirrorlist-serverbeach too. (Although I might make it a rhel7 and see if it's any happier that way)
18:37:26 <oddshocks> nirik: update: apparently those 64 bit base amis _are_ on the website, and roby will add the rest once i figure out what's wrong with them
18:37:34 <nirik> oddshocks: great. :)
18:37:47 <nirik> any other nagios related or sysadmin related stuff before we move on?
18:37:56 <oddshocks> HVM/atomic stuff is proving to be a bit of a learning curve for me, still working out what options need to happen for the image to boot
18:38:28 <oddshocks> having to add a bit of fedimg code to accomplish that special stuff, and since certain instance types can and can't be HVM, and some instance types can and can't be 32 bit, it's a bit of a challenge but i'm working through it
18:38:40 * oddshocks done derailing the sysadmin section
18:38:47 <nirik> yeah, I think the atomic stuff is a learning curve for everyone. :)
18:39:24 <smooge> I wonder if I could set up a download01.stg.phx2.fedoraproject.org?
18:39:36 <nirik> we could if needed sure.
18:39:49 <nirik> seems kinda a waste unless it's different any
18:40:01 <smooge> well I figured it doesn't need to be hardware
18:40:26 <smooge> just an virt for stg stuff to have 1:1 parity to
18:40:31 <nirik> sure, but it would just mount the same stuff? it just seems like it wouldn't be too usefull... and would take up memory/cpu/etc
18:41:25 <nirik> but I guess if we can't get things talking to the prod ones we could.
18:41:49 <nirik> #topic Upcoming Tasks/Items
18:41:50 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:42:01 <nirik> anything upcoming anyone would like to note or schedule?
18:42:28 <puiterwijk> hopefully a real fix for CVE-2014-7169, while I'll deploy once it arrives
18:42:36 <nirik> yeah.
18:42:36 <puiterwijk> while=which
18:43:22 * nirik nods
18:43:29 <nirik> #topic Open Flood
18:43:38 * nirik typos, runs with it.
18:43:39 <puiterwijk> nirik: new name for the open floor? :)
18:43:41 <nirik> anything to flood? ;)
18:43:44 * puiterwijk likes it
18:43:46 * pingou heads for dinner, ttyl :)
18:43:58 <nirik> pingou: enjoy
18:43:58 <pingou> will do :)
18:44:12 <nirik> (really this is open floor, so bring up any other topics anyone would like to discuss)
18:44:15 <threebean> lots of work to do until beta freeze... ;)
18:44:24 <nirik> yeah...
18:44:35 <nirik> I'd really like to get more rhel7 and ansible migrations done
18:44:39 <threebean> random solicitation: i've been working on an update to the landing page at https://apps.fedoraproject.org
18:44:57 <threebean> if anyone wants to help fill in the last bits of content, that would be a help -> https://github.com/fedora-infra/apps.fp.o/
18:45:53 <nirik> cool.
18:47:45 <nirik> ok, if nothing else will close out in a minute.
18:48:19 <nirik> Thanks for coming everyone! lets continue in #fedora-admin, #fedora-apps and #fedora-noc.
18:48:21 <nirik> #endmeeting