18:02:43 <nirik> #startmeeting Infrastructure (2014-09-25) 18:02:43 <zodbot> Meeting started Thu Sep 25 18:02:43 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:02:43 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:02:43 <nirik> #meetingname infrastructure 18:02:43 <nirik> #topic aloha 18:02:43 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:02:43 <zodbot> The meeting name has been set to 'infrastructure' 18:02:43 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:02:51 * lanica is here for the infra meeting. 18:02:55 <threebean> oy 18:02:55 * tflink is here 18:03:00 <janeznemanic> hi 18:03:04 * puiterwijk here 18:03:06 <Neldogz> neldogz is here 18:03:16 <webpigeon> o/ 18:03:25 * lmacken 18:03:30 * pingou here 18:03:32 * bitlord here, listening \o ;-) 18:04:17 * danielbruno here 18:04:24 <nirik> #topic New folks introductions and Apprentice tasks 18:04:29 <nirik> any new folks like to introduce themselves? 18:04:34 <nirik> or apprentices with questions or comments? 18:04:50 * oddshocks arrives enveloped in a cloud of crows 18:04:57 * relrod arrives too 18:05:03 <danielbruno> ! 18:05:46 <smooge> >:} 18:06:09 <nirik> danielbruno: feel free to just chime in. we don't use that meeting protocol thing. 18:06:34 <ssf87> no agenda ? 18:06:38 <danielbruno> nirik, I saw yout reply on the ticket about the planet to track broken feeds 18:06:55 <nirik> ssf87: yes there is. It was sent to the mailing list... 18:07:02 <danielbruno> I would ike to know if I need some permission 18:07:17 <danielbruno> to access .planet files 18:07:27 <nirik> danielbruno: you shouldn't. :) everyones /home/fedora/*/.planet files should be readable. 18:07:41 <danielbruno> great! 18:07:44 <nirik> the planet job runs as nobody so if it can read it you should be able to too 18:08:10 <danielbruno> nirik, thank you :) 18:08:22 <nirik> no problem, do let me know if you run into any problems with it. 18:09:18 <nirik> any other new folks or apprentice questions/comments? 18:09:52 <nirik> #topic Applications status / discussion 18:10:06 <nirik> any application news this week? threebean / lmacken / pingou / oddshocks ? 18:10:15 <threebean> we're having issues with fedmsg load on datanommer right now 18:10:16 <threebean> http://threebean.org/fedmsg-health.html 18:10:31 <lmacken> I did a couple of bodhi updates in production this week. Nothing too exciting, mainly EL7 related stuff. 18:10:35 <nirik> threebean: is that db on rhel7 yet? or still 6? 18:10:37 <puiterwijk> I'd like to at least throw in the public again that fedoauth will be deprecated in the future. not yet sure when 18:10:38 <threebean> it's a result I think of us outgrowing our current one-big-postgres-table setup and we're researching alternatives 18:10:46 <pingou> I've been working on a couple of requests from gnokii for nuancier, review pending 18:10:59 <pingou> I also have some reviews pending for pkgdb2, closing some tickets 18:11:12 <nirik> #info fedmsg load issues with datanommer, being worked on. 18:11:25 <nirik> #info some bodhi1 updates in production around epel7 stuff 18:11:28 <pingou> and I added support to send the reminder emails to multiple addresses on fedocal, merged and planned in the next release 18:11:31 <threebean> nirik: still on 6.5 18:11:33 <lmacken> threebean: it'd be real interesting to see those benchmarks that you wrote earlier between RHEL 6 & 7 18:11:43 <nirik> #info fedoauth will be depreciated at some time to be scheduled yet. 18:11:45 <puiterwijk> #link https://lists.fedoraproject.org/pipermail/infrastructure/2014-September/014863.html 18:12:02 <nirik> #info some nuancier patches pending review. 18:12:23 <nirik> threebean: wonder if this would be a good time to move it to 7? or it needs more than just a newer postgres? 18:12:28 * threebean nods 18:12:36 <threebean> let's move it up this afternoon. 18:12:47 <threebean> I can update moksha and the collectd stuff on busgateway01 at the same time. 18:13:15 <relrod> If anyone uses HRF (hrf.cloud.fp.o) for anything, please switch to datagrepper instead. I want to deprecate HRF because datagrepper can do everything HRF was made to do now. 18:13:37 <nirik> so, would we want to spin a new 7 one up and transfer, or just save the db off and destroy the 6 one and recreate it as 7? 18:13:50 <threebean> the first option sounds safest :) 18:14:08 <nirik> #info hrf.cloud.fedoraproject.org depreciated. Please don't use it anymore, use datagrepper. 18:14:24 <nirik> threebean: yeah, just needs allocating ip's, etc... more infra work, but not too big a deal. 18:14:48 <tflink> taskotron production is making progress - initial systems are set up, proxies are configured, seems to be mostly working 18:15:04 <tflink> still need to get monitoring and backups set up 18:15:20 <nirik> tflink: saw the monitoring ticket, but haven't had a chance to do anything with it. 18:15:28 <tflink> would like to see it funcitoning for several days and a couple of fixes in place before turning off autoqa 18:15:28 <nirik> might be a good place for a new person to jump in. ;) 18:15:33 <nirik> yeah. 18:15:48 <tflink> SOPs have been written for taskotron and resultsdb 18:15:57 <nirik> #info taskotron almost in production. Needs monitoring and backups and a few days of good smooth operation. 18:17:20 <nirik> cool. Any other applications news? 18:17:30 <nirik> anything we want to make sure to try and get done before beta freeze? 18:18:06 <pingou> MM2? :D 18:18:08 * pingou ducks 18:18:11 <oddshocks> F21 RC1 AMIs are listed on my fpeople page, still debugging 32 bit base and 64 bit atomic: https://oddshocks.fedorapeople.org/ 18:18:18 <oddshocks> but 64 bit base are there 18:18:33 <nirik> oddshocks: nice. Should those get updated in the website? 18:18:42 <nirik> pingou: ha. 18:19:11 <puiterwijk> pingou: I hear you're volunteering to get it done by then? :) 18:19:11 <oddshocks> nirik: I think robyduck is waiting to update the links until i have the other two sets working, but i'm not sure if that's the same thing 18:19:22 <pingou> we might manage to have anitya running before beta 18:19:24 <nirik> oddshocks: ok, sounds reasonable. 18:19:31 <pingou> but that's not quite part of our infra anyway :) 18:19:42 <pingou> puiterwijk: I was merly proposing you :-p 18:19:49 <nirik> also, as a side note, I put f21alpha cloud image into our cloud... so if we need any f21alpha instances it should be ready. 18:20:02 <nirik> we could add a jenkins one perhaps. 18:20:46 <nirik> which reminds me... pingou: what was the conclusion about making jenkins a more supported/supportable service? no go since it's not packaged for rhel? or ? 18:21:11 <pingou> nirik: not packaged for rhel was/is the biggest blocker 18:21:32 <nirik> yeah. wonder if even 7 wouldn't work... it should have new enough stuff I would think 18:21:54 <pingou> depends on jenkins' deps, which I suspect is big 18:22:02 <puiterwijk> yeah, jenkins dep tree is quite big 18:22:04 <nirik> the docs folks were looking at a jenkins plugin for publishing and serving docs. 18:22:18 <puiterwijk> maybe we should get someone to just go ahead and package it all 18:22:28 <pingou> yes, I read the backlog of this 18:22:41 <pingou> puiterwijk: that would imply we are comitted to maintain it 18:22:50 <puiterwijk> pingou: yeah 18:23:10 <nirik> pingou: do you recall who we were talking to about it at flock? 18:23:29 <pingou> nirik: I remember the face, but not the name 18:23:32 <pingou> :/ 18:23:34 <nirik> yeah, same here. ;( 18:23:57 <nirik> oh well, we don't need to do anything right now, but something to think on. 18:24:13 <nirik> #topic Sysadmin status / discussion 18:24:24 <smooge> hi 18:24:26 <puiterwijk> #info CVE-2014-6271/CVE-2014-7169 (Bash issues): Patches for 6271 and workaround for 7169 applied on high profile servers 18:24:28 <nirik> so, we left freeze yesterday... just in time for some security update fun. ;) 18:24:34 <smooge> yay! 18:24:40 <nirik> hey smooge 18:24:49 <nirik> puiterwijk: you also updated mediawiki right? 18:25:00 <puiterwijk> nirik: yup. we're now at the latest one, released yesterday 18:25:13 <puiterwijk> #info Mediawiki upgraded to 1.19.19 18:25:38 <smooge> yay! 18:26:15 <webpigeon> \0/ 18:26:15 <nirik> cool. 18:26:28 <puiterwijk> and I'm packaging the 1.23 series 18:26:41 <puiterwijk> (since we need to upgrade before May 2015, since that's EOL for 1.19) 18:27:16 <nirik> yep. 18:27:32 <nirik> I've been working thru backlog of tickets that landed when I was traveling... 18:27:43 <nirik> hopefully we will be back to normal on those before too long. 18:27:53 <nirik> I'd also like to look at scheduling a mass reboot cycle next week. 18:27:58 <nirik> Probibly wed or so... 18:29:00 <nirik> #info mass reboot likely next week to catch up on updates 18:29:17 <nirik> #topic nagios/alerts recap 18:29:26 * nirik looks for the url again. 18:30:04 <puiterwijk> .tiny https://admin.fedoraproject.org/nagios/cgi-bin//summary.cgi?report=1&displaytype=3&timeperiod=last7days&smon=9&sday=1&syear=2014&shour=0&smin=0&ssec=0&emon=9&eday=4&eyear=2014&ehour=24&emin=0&esec=0&hostgroup=all&servicegroup=all&host=all&alerttypes=3&statetypes=3&hoststates=7&servicestates=120&limit=25 18:30:04 <zodbot> puiterwijk: http://tinyurl.com/l3vjae8 18:30:11 <puiterwijk> nirik: ^ 18:30:17 <smooge> and again puiterwijk comes through. 18:30:21 <nirik> beat me to it. ;) 18:30:57 <puiterwijk> nirik: or better: http://da.gd/fednagios 18:30:59 <nirik> so, the datagrepper issues we are aware of 18:31:45 <nirik> not sure about the collab mail queue. I don't recall seeing those? 18:31:50 <nirik> so must have been warnings 18:31:57 * oddshocks read that as http://da.gd/fadingos 18:31:57 <lmacken> packages03 might need some more memory :\ I spent a little bit of time this week poking some unicode issues with the xapian db, but haven't looked at prod 18:32:25 <nirik> lmacken: yeah, it's gotten stuck a lot lately. It also had a issue this week where OOM killed glusterd. ;( 18:32:31 <nirik> we can bump it up some more. 18:33:01 <nirik> 8gb -> 12? 18:33:14 <lmacken> that sounds fine 18:33:47 <lmacken> also, packages.stg can't get to dl.fp.o:80 for some reason. Need to add a firewall rule? 18:33:53 * nirik can do so after the meeting. 18:33:54 <lmacken> it can ping it 18:34:12 <nirik> is it using internal or external ip? 18:34:36 <lmacken> I think it was hardcoded to download03.phx2 18:34:37 <puiterwijk> lmacken: yeah, stg -> prod is denied with internal IP now, thanks to threebean's blanket rule 18:34:56 <lmacken> okay, cool 18:34:57 <puiterwijk> so it'd need to use external IP to have any chance, or a seperate firewall rule 18:35:07 <nirik> yeah. 18:35:19 <puiterwijk> (I vote for external IP, I rather like the stg->prod firewall rule) 18:35:29 <nirik> if external works, sure. 18:37:01 <nirik> also, re: nagios... I am going to kill unbound-telia01 soon... and possibly kill mirrorlist-serverbeach too. (Although I might make it a rhel7 and see if it's any happier that way) 18:37:26 <oddshocks> nirik: update: apparently those 64 bit base amis _are_ on the website, and roby will add the rest once i figure out what's wrong with them 18:37:34 <nirik> oddshocks: great. :) 18:37:47 <nirik> any other nagios related or sysadmin related stuff before we move on? 18:37:56 <oddshocks> HVM/atomic stuff is proving to be a bit of a learning curve for me, still working out what options need to happen for the image to boot 18:38:28 <oddshocks> having to add a bit of fedimg code to accomplish that special stuff, and since certain instance types can and can't be HVM, and some instance types can and can't be 32 bit, it's a bit of a challenge but i'm working through it 18:38:40 * oddshocks done derailing the sysadmin section 18:38:47 <nirik> yeah, I think the atomic stuff is a learning curve for everyone. :) 18:39:24 <smooge> I wonder if I could set up a download01.stg.phx2.fedoraproject.org? 18:39:36 <nirik> we could if needed sure. 18:39:49 <nirik> seems kinda a waste unless it's different any 18:40:01 <smooge> well I figured it doesn't need to be hardware 18:40:26 <smooge> just an virt for stg stuff to have 1:1 parity to 18:40:31 <nirik> sure, but it would just mount the same stuff? it just seems like it wouldn't be too usefull... and would take up memory/cpu/etc 18:41:25 <nirik> but I guess if we can't get things talking to the prod ones we could. 18:41:49 <nirik> #topic Upcoming Tasks/Items 18:41:50 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:42:01 <nirik> anything upcoming anyone would like to note or schedule? 18:42:28 <puiterwijk> hopefully a real fix for CVE-2014-7169, while I'll deploy once it arrives 18:42:36 <nirik> yeah. 18:42:36 <puiterwijk> while=which 18:43:22 * nirik nods 18:43:29 <nirik> #topic Open Flood 18:43:38 * nirik typos, runs with it. 18:43:39 <puiterwijk> nirik: new name for the open floor? :) 18:43:41 <nirik> anything to flood? ;) 18:43:44 * puiterwijk likes it 18:43:46 * pingou heads for dinner, ttyl :) 18:43:58 <nirik> pingou: enjoy 18:43:58 <pingou> will do :) 18:44:12 <nirik> (really this is open floor, so bring up any other topics anyone would like to discuss) 18:44:15 <threebean> lots of work to do until beta freeze... ;) 18:44:24 <nirik> yeah... 18:44:35 <nirik> I'd really like to get more rhel7 and ansible migrations done 18:44:39 <threebean> random solicitation: i've been working on an update to the landing page at https://apps.fedoraproject.org 18:44:57 <threebean> if anyone wants to help fill in the last bits of content, that would be a help -> https://github.com/fedora-infra/apps.fp.o/ 18:45:53 <nirik> cool. 18:47:45 <nirik> ok, if nothing else will close out in a minute. 18:48:19 <nirik> Thanks for coming everyone! lets continue in #fedora-admin, #fedora-apps and #fedora-noc. 18:48:21 <nirik> #endmeeting