18:00:01 <nirik> #startmeeting Infrastructure (2012-08-09) 18:00:01 <zodbot> Meeting started Thu Aug 9 18:00:01 2012 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:01 <nirik> #meetingname infrastructure 18:00:01 <zodbot> The meeting name has been set to 'infrastructure' 18:00:01 <nirik> #topic Greets! 18:00:01 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean 18:00:01 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean 18:00:07 * skvidal is here 18:00:09 * relrod here :) 18:00:34 * ctria almost back (i'll be off next week) and then i'm back 18:00:42 * nirik notes threebean is traveling and probibly won't be able to attend today 18:00:58 * puiterwijk is here 18:01:00 * pingou here 18:01:10 * abadger1999 here 18:01:44 * lmacken 18:02:18 <nirik> ok, cool. Lets go ahead and get started... 18:02:27 <nirik> #topic New folks introductions and Apprentice tasks. 18:02:36 <nirik> If any new folks want to give a quick one line bio or any apprentices 18:02:36 <nirik> would like to ask general questions, they can do so in this part of 18:02:36 <nirik> the meeting. Don't be shy! 18:03:28 <ctria> i hope that you remember me. i'm in process of changing job and hopefully will have more time to spare this time. i officially will ask to join the apprentice group after next week 18:03:34 <ctria> that i'm returning from vacations 18:03:42 <nirik> welcome back ctria 18:04:20 <nirik> ok, if no one else new, will move on to the next topic... 18:04:23 <ctria> (but i could work on any tasks could be worked from a sandy beach next week) 18:04:24 <nirik> #topic Applications status / discussion 18:04:37 <nirik> ctria: relax and have a good vacation I'd say. ;) 18:04:53 <nirik> any application news from this last week or upcoming in the next? 18:05:00 <lmacken> I'm doing a new bodhi production release today, which includes fedmsg support :) 18:05:17 <nirik> there was a lot of fedmsg stuff landing in production as well recently. 18:05:18 * lmacken just has to make a few more tweaks so we don't flood the bus 18:05:34 <nirik> I messed up and thought the freeze was this tuesday, and it's not until next tuesday. ;) 18:05:36 <relrod> http://status.fedoraproject.org/ is live (ok, it's not really an app, it's a statically generated html file, but still) 18:05:36 <abadger1999> Next fas release is in stg right now. Thanks to relrod and puiterwijk for putting it together :-) 18:05:44 <relrod> so 18:05:47 <relrod> please test that out 18:05:56 <relrod> and let puiterwijk and myself know if you have any suggestions 18:05:59 <nirik> relrod: it has it's own topic later in the meeting too. 18:06:04 <relrod> nirik: ah, oops 18:06:07 * relrod shuts up then :) 18:06:07 <lmacken> relrod, puiterwijk: nice work with that guys, it's awesome 18:06:11 <nirik> no worries. 18:06:36 <nirik> we have a #fedora-fedmsg channel with fedmsg bus live on irc now too. 18:07:03 <nirik> anything else that could/will land before freeze? 18:07:09 <lmacken> maybe busmon 18:07:17 * lmacken not sure of the state of our websockets stuff 18:07:27 <nirik> not very far along I'm afraid. 18:07:28 <lmacken> but threebean and I will be doing some hacking on it this weekend in person 18:07:42 <nirik> I still need to talk to RHIT about it... will see how that goes over. 18:08:07 <nirik> lmacken: cool. 18:08:25 <nirik> ok, moving along then... 18:08:32 <nirik> #topic Sysadmin status / discussion 18:08:48 <nirik> this section is for news from sysadmin folks... lets see... 18:08:57 * skvidal is just grumpy today 18:09:02 <skvidal> oh - you don't mean about the people 18:09:03 <nirik> We had some issues with the wiki upgrade last weekend... 18:09:03 <skvidal> sorry 18:09:05 <nirik> ha 18:09:21 <nirik> puiterwijk helped us out and tracked those down... so hopefully it's all happy now. 18:09:52 <nirik> also 2 wiki bug fixes... a rewrite where it was double encoding things and redirecting people wrong on login, and a fix to the rss feed to again show actual diffs. 18:10:43 <nirik> skvidal setup a releng01 with ansible... 18:11:22 <nirik> bunch of password resetting on mgmt interfaces, etc. 18:11:35 * relrod still is working on them, slowly but surely 18:11:37 <nirik> anything else folks would like to ask about or note? 18:11:44 <nirik> relrod: it's very appreciated. 18:11:57 <nirik> ok, moving on 18:12:04 <nirik> #topic Security FAD 18:12:10 <nirik> https://fedoraproject.org/wiki/FAD_Infrastructure_Security_2012 18:12:18 <nirik> I've done some more things to move this along... 18:12:27 <nirik> fixed up the wiki page, pinged people, etc. 18:12:40 <nirik> If you wish to attend, please add yourself to the attendees page. 18:12:48 <nirik> Also, we should try and finalize dates. 18:13:05 <nirik> I was thinking at this point that perhaps after f18 is released, but before holidays might work 18:13:31 <nirik> thoughts? rants? alterative suggestions? 18:13:48 <skvidal> just not during sept 18:13:49 <skvidal> kthx 18:14:02 <smooge> nirik, at that point is it close enough to FudCon to do it then? 18:14:02 * nirik looks at calendar 18:14:03 <pingou> skvidal: +1 18:14:04 <skvidal> oh and I'd like to personally request one other time be blacked out 18:14:10 <skvidal> the first week of october 18:14:27 <skvidal> smooge: we won't get anything done colocating with fudcon 18:14:29 <nirik> smooge: well, we could do that yeah... thats jan instead of nov... 18:14:33 <skvidal> we're too often dragged away 18:14:38 <abadger1999> <nod> 18:14:50 <abadger1999> We'd have to say "Week before fudcon" or some such 18:14:52 <pingou> skvidal: could be before or after fudcon no ? 18:14:59 <nirik> right, it would need to be before. 18:15:03 <nirik> come in early 18:15:04 <skvidal> pingou: that's fine 18:15:05 <skvidal> abadger1999: yah 18:15:07 <skvidal> I don 18:15:11 <smooge> actually I wasn't thinking of doing it then. I was worried it would get the "Thats awfully close to Fudcon" checkmark 18:15:11 <skvidal> t have an issue with that 18:15:11 <nirik> 2012-11-06 - f18 release 18:15:14 <skvidal> just not overlappying 18:15:20 <skvidal> and doing it AFTER fudcon is probably not wise 18:15:27 <skvidal> b/c everyone is strungout after 18:15:30 <skvidal> or rather 18:15:33 <skvidal> everyone I ever speak to is 18:15:37 <skvidal> maybe it's just me 18:15:57 <ctria> lol 18:15:58 <smooge> November sounds good 18:16:03 * abadger1999 agrees w/ skvidal 18:16:10 * nirik also 18:16:14 <smooge> and I agree w/ skvidal 18:16:15 <skvidal> abadger1999: you agree that i'm strung out after fudcon? 18:16:16 <nirik> so, its a narrow window there in nov... 18:16:17 <skvidal> nice 18:16:20 <abadger1999> hehe 18:16:36 <nirik> release is the 6th, and thanksgiving is 22nd. 18:16:40 <nirik> if we slip a week... 18:16:41 <smooge> skvidal, well if you wouldn't do those lines of cocaine off the backs of servers at fudcon it probably would not happen 18:16:45 <pingou> I can't convince you to make it for the FUDCon emea ? :D 18:17:02 <skvidal> pingou: I suspect spot/robyn would set us on fire 18:17:09 <smooge> very very much so 18:17:11 <nirik> pingou: I don't think we have budget for that. ;( 18:17:15 <pingou> skvidal: s/set us/set me/ ? :) 18:17:20 <skvidal> pingou: no, us 18:17:22 <skvidal> all of us 18:17:25 <pingou> ^^ 18:17:32 * pingou burns 18:17:32 <skvidal> they have assassins everywhere 18:17:38 <smooge> kerosene is cheaper than tickets I think this is the usual answer 18:17:46 <spot> pingou: you get me and robyn. anyone else, you're welcome to cover out of your pocket. ;) 18:18:08 <pingou> spot: can we make exchange ? 18:18:14 <pingou> spot: ;-) 18:18:16 <skvidal> oooo - is pingou independently wealthy? I didn't know that 18:18:19 <spot> pingou: no exchanges, no refunds. ;) 18:18:21 <skvidal> excellent 18:18:30 <skvidal> pingou: can I have a grant? 18:18:37 <pingou> skvidal: if I was I would come to the security FAD ;-) 18:18:43 <skvidal> pingou: good point 18:18:45 <pingou> skvidal: of course, I'll pay you the beer when you arrive ;) 18:18:56 <skvidal> none for me 18:18:57 <nirik> right, so we could tenatively say: 2012-11-11 (arrive late afternoon/evening, get settled), 12, 13, 14 work on things, leave 2012-11-14? 18:19:05 <pingou> skvidal: then a very large coke 18:19:08 <nirik> but then we are having to do release from there if we slip. ;) 18:19:10 <halfline> ginger beer 18:19:38 <skvidal> halfline: only yankees drink such things ;) 18:20:02 <smooge> I thought aussies drank it 18:20:04 * nirik suggests ginger mead, but thats likely just me. ;) 18:20:15 <abadger1999> nirik: If I could find it, I would drink it ;-) 18:20:19 <skvidal> so... trailing off into the weeds 18:21:03 <abadger1999> 2012-11-11 works for me ... but I don't trust release not to slip. 18:21:09 <nirik> There's also the week after thanksgiving... 2012-11-25 to 28... 18:21:10 <smooge> I will put it that us hitting our deadline for F18 is not likely.. so I think the 16th is more likely the release date 18:21:35 <smooge> how about the 22-24 in Raleigh NC 18:21:47 <skvidal> what? why would any of us need to be around for the release. That's just silly. :) 18:21:59 <nirik> smooge: do they have turkey dinners there? :) 18:22:27 <skvidal> smooge: I will die a painful death if I am not around family over thanksgiving 18:22:30 <smooge> I don't know. I know that Westford has real turkeys in their parking lots then 18:23:03 <skvidal> nirik: moving on? 18:23:29 <nirik> problem with after thanksgiving is that many people are traveling then... 18:23:56 <skvidal> and flying on black friday? 18:24:01 <skvidal> that's a special kind of suffering 18:24:01 <smooge> oh sorry I was looking at the wrong month. I wasn't making it sit ontop of Thanksgiving 18:24:03 <abadger1999> Maybe we could plan on release sliping two+ weeks ;-) 18:24:13 <nirik> 26th-29th? come in 26th, leave 29th. 2 solid days of work 18:24:24 <pingou> 2012-11-11,12,13,14 might actually be hard for me, but we'll see 18:24:27 <skvidal> abadger1999: december in nc - how...... something... cold? uncomfortable but lacking any snow? 18:24:49 <smooge> bah. its not cold. its chilly but not cold 18:24:54 <abadger1999> As long as I'm not driving on glare ice I'm happy with cold weather. 18:25:11 * nirik doesn't care about cold as long as it doesn't cause flight delays. ;) 18:25:26 <abadger1999> true. 18:25:57 <nirik> or, for better budget spending: 26th-28th... come in 26th, leave night of 28th. Hack on things 27th and 28th days. 18:26:45 <smooge> hmm that works for me. 18:26:51 <nirik> votes? counter proposals? 18:27:01 <skvidal> 26th of november? 18:27:19 <skvidal> cool 18:27:55 <nirik> ok, if no objections I will take that to the list as a tenative? 18:28:03 <abadger1999> eh... 29th sounds better (just htinks about all the times late flights get rescheduled earlier in the day). 18:28:16 <abadger1999> but it still works for me. 18:28:22 <nirik> abadger1999: ok... we can see if we can swing that... 18:28:42 <abadger1999> I don't particularly care... just thinking of how much can get done. 18:28:44 <nirik> question: will RH be moved/moving to new tower then? I guess I can find out 18:28:50 <relrod> either way I'm going to miss a few days of school, so doesn't matter to me...just have to ask for the assignments beforehand and do them on flights. ;) 18:29:21 <nirik> so, if we are ok with that, I'll ask interested folks to go back and actually look up costs for those dates? 18:29:25 <nirik> and update the wiki page. 18:29:38 <nirik> so we can get a idea travel wise 18:30:23 <nirik> #agreed 2012-11-26 to 2012-11-29 for security fad in RDU 18:30:36 <nirik> #action everyone who would like to attend, please update wiki page. 18:30:49 <nirik> #info remote contributions also very welcome 18:31:03 <nirik> ok, anything else on this? or shall we move on? 18:31:22 <nirik> is there any people I didn't list that would be good to have to get the primary objective done? 18:31:39 * abadger1999 gets link to travel page for the meeting notes 18:31:48 <nirik> https://fedoraproject.org/wiki/FAD_Infrastructure_Security_2012/attendees 18:31:56 <nirik> we could just fold that into the fad page, no? 18:32:04 <abadger1999> https://fedoraproject.org/wiki/FAD_Infrastructure_Security_2012/attendees 18:32:06 <ctria> should remote register too? 18:32:24 <nirik> ctria: yeah, let us know if you are interested in helping, for sure... just note that you will be remote... 18:33:01 <nirik> ok then, moving along... 18:33:08 <nirik> #topic status.fedoraproject.org 18:33:20 * abadger1999 will fold attendees into the main page 18:33:48 <nirik> so, as many know, puiterwijk had an idea for a end user status application... so we could point people to it for outages instead of them having to check an irc /topic, etc 18:34:09 <nirik> we have such an application now thanks to puiterwijk and relrod. ;) 18:34:23 <nirik> #info status.fedoraproject.org runs in openshift currently. 18:34:37 <nirik> #info completely disconnected from our infrastructure 18:34:45 <ctria> great work 18:34:46 <nirik> #info updates are manually done by admins. 18:34:58 <puiterwijk> #info static webpage generated on push 18:35:05 <nirik> so, should we start using this? or do we have any questions or concerns about it? 18:35:21 <nirik> If we wanted to push this to some other site, how hard would that be? 18:35:27 <pingou> puiterwijk: stupid idea, rss ? 18:35:43 * nirik was also thinking rss might be nice if it was easy 18:35:58 <puiterwijk> pingou: maybe a good idea 18:36:09 <puiterwijk> #action puiterwijk will look into RSS with status.fp.o 18:36:10 <abadger1999> #info http://status.fedoraproject.org/ 18:36:48 * abadger1999 needs to get better at reading URLs that aren't highlighted as links :-/ 18:36:49 <nirik> also, there's one issue left: syncing ssh keys to grant access to folks who might need to update it... 18:37:25 * nirik has manually added a bunch of people for now, but we might setup a sync script on lockbox01 for it. 18:37:32 <ctria> could this get nagios alarms and automate results? 18:37:33 <relrod> lmacken also had an idea for auto refresh, and using something like https://github.com/lipka/piecon to show percentage of services up in the favicon, so people can see it at a glance 18:37:34 <nirik> or we could just leave it manual I suppose 18:37:34 * ctria runs 18:37:54 <nirik> ctria: we want it to NOT depend on any of our stuff being up/down/whatever. 18:37:56 <puiterwijk> I'm working on a python script to automate that from FAS 18:38:03 <puiterwijk> (the ssh key updating) 18:38:08 <nirik> puiterwijk: ok, cool. 18:38:20 <nirik> ctria: otherwise it could not update right if nagios was down, etc. 18:38:26 <abadger1999> Is it just checking by hitting a single url and seeing if that returns? 18:38:33 <nirik> also, nagios has a poor idea of what application is down. 18:38:44 <puiterwijk> abadger1999: no, it's manually updated by an admin 18:38:44 <nirik> abadger1999: what checking? 18:38:50 <nirik> it's a static page 18:38:51 <abadger1999> k 18:39:11 <abadger1999> ah -- I thought it was a periodically generated static page. Misunderstood. 18:39:13 <nirik> so it could be misleading if there's an outage and no one updates it 18:39:19 <abadger1999> sorry. 18:39:36 <nirik> or if someone updates it and forgets to clear the outage (but we have a nagios check for that that should yell at us) 18:39:41 <ctria> another way is that it would get a heartbeat from services via fedmsg if it doesn't then service is off 18:40:15 <puiterwijk> ctria: and what if fedmsg is off? it won't know if that's a fedmsg failure or a connection failure 18:40:40 <ctria> common� you have to trust something� what if admins are off :) 18:40:58 <skvidal> ctria: fedmsg is a nice service - I do not think we have used it enough to get the point of trusting it 18:41:19 <skvidal> for notification of outages 18:41:25 <pingou> ctria: admins are off ??? 18:41:37 <nirik> well, and also since this is not in phx2, we would need some way to get messages to it... 18:41:48 <nirik> pingou: some of us have always been a bit off. ;) 18:41:53 <skvidal> nirik: I'm sure mmcgrath would be happy to let us vpn in 18:41:55 <skvidal> ;) 18:42:04 <nirik> heh 18:42:32 <nirik> so, aside from enhancements, is everyone ok with us starting to use this and point people to it? 18:42:40 <pingou> +1 18:42:42 <ctria> +! 18:42:44 <nirik> or are their concerns/issues outstanding? 18:43:43 * nirik listens to crickets chirp 18:44:06 <puiterwijk> just one currently: that the ordering still needs to be fixed 18:44:21 <puiterwijk> (haven't had much time for that yet, but that's not crucial) 18:44:29 * skvidal will bite 18:44:32 <skvidal> what's wrong with the ordering? 18:44:36 <nirik> puiterwijk: oh, so how hard would it be to push this to another site if we wanted... 18:44:46 <puiterwijk> mizmo had advised to put major and minor outages on top (above the Ok services) 18:45:21 <puiterwijk> nirik: not too hard, only the management script would have to be adjusted to push somewhere else instead of git, and maybe the management script should regenerate the page 18:45:59 <nirik> puiterwijk: ok. I am just thinking of a case where openshift was down and all our stuff was up. We could change dns and point to one of our servers and push the page there so users wouldn't see problems. 18:46:21 <nirik> (as unlikely as that may be) 18:46:23 <ctria> nirik, if admins are up :) 18:46:40 <puiterwijk> nirik: that could be set up :) 18:47:09 <nirik> ok, so don't see any objections... I'd like to let it go for a few weeks before we officially announce it... so please do look for any issues before then. 18:47:33 <nirik> #agreed will start using status page, fix any issues, then announce in a few weeks.... 18:47:54 <nirik> thanks for all the work on this puiterwijk and relrod. (and others too) 18:48:11 <puiterwijk> maybe we should pick a date, and if no showstoppers are still on that date, we will announce it? 18:48:22 <puiterwijk> to know when everything should be fixed? 18:48:32 <nirik> sure. how about the 22nd? 18:48:50 <puiterwijk> works for me 18:48:56 <relrod> nirik: 23rd? so we can ask at the meeting again? 18:48:59 <puiterwijk> or 23rd, at the meeting? 18:49:10 <nirik> sure. Thats fine too 18:49:13 * nirik makes a note 18:49:28 <nirik> #info status page will be announced on the 23rd if no stoppers are found 18:49:41 <nirik> #topic Upcoming Tasks/Items 18:49:44 <puiterwijk> oh, just so you know: from 19:00 UTC, nagios alerts will be sent when the services are triggered ;) 18:50:10 <nirik> hum? 18:50:19 <puiterwijk> still about status.fp.o 18:50:36 <nirik> oh, right... 18:50:43 <puiterwijk> if any services are in minor or major status (instead of "good"), nagios will complain about it so we won't forget to put it back when done 18:50:45 <nirik> so if you do any testing of the page, please put it in downtime 18:50:55 <nirik> so, upcoming items: 18:50:59 <nirik> #info 2012-08-14 to 2012-08-28 F18 Alpha Freeze 18:51:00 <nirik> #info 2012-08-15 hosted03-> hosted01/02 migration (tenative) 18:51:00 <nirik> #info 2012-08-28 F18 Alpha release. 18:51:00 <nirik> #info 2012-08-31 end of 2nd quarter 18:51:00 <nirik> #info 2012-09-01 nag fi-apprentices. 18:51:00 <nirik> #info 2012-09-08 drop inactive fi-apprentices. 18:51:02 <nirik> #info 2012-09-10 to 2012-09-24 Skvidal is out 18:51:04 <nirik> #info 2012-09-18 to 2012-10-02 F18 Beta Freeze 18:51:10 <nirik> #info 2012-09-27 to 2012-09-28 Smooge at Puppetconf 18:51:10 <nirik> #info 2012-10-02 F18 Beta release 18:51:12 <nirik> I'm waffling about the hosted move and when to do it... 18:51:21 <nirik> anything else folks want to note? 18:51:27 <nirik> or schedule, or plan? 18:51:41 <skvidal> do we have an ETA on cloud network? 18:51:57 <nirik> not that I have heard. ;( smooge: you heard anything? 18:52:09 <nirik> if we get internal nets we can start installing/setting things up 18:52:11 <smooge> no 18:52:18 <smooge> I am going to ping management 18:52:54 <nirik> ok. Might be good to drop a ticket on it, in case other folks can work on it for us? 18:53:21 <nirik> dunno. 18:54:06 <nirik> ok, anything more for upcoming? if not, moving on... 18:54:16 <nirik> #topic Open Floor 18:54:21 <nirik> anything for open floor? 18:54:22 <skvidal> I have something 18:54:29 <nirik> take it away... 18:54:35 <skvidal> there was a meeting last week in #fedora-buildsys 18:54:42 <skvidal> to discuss non-koji buildsys goals 18:54:50 <skvidal> we're not discussing getting rid of koji 18:54:54 <skvidal> nor replacing it 18:55:02 * relrod has to head out to bring the dog to the vet :( - will read the tail end of the logs when I return. 18:55:04 <skvidal> this is for builds of pkgs that we cannot do in koji for one reason or another 18:55:11 <nirik> ah yeah. 18:55:14 <skvidal> I wrote up the requirements here 18:55:15 <skvidal> https://fedoraproject.org/wiki/User:Skvidal/BuildSystem 18:55:23 <nirik> cool. 18:55:24 <skvidal> and we're working on ui/backend requirements here 18:55:35 <skvidal> https://fedoraproject.org/wiki/User:Skvidal/BuildSystem/Specifications 18:55:40 <nirik> this is something we hope to offer backed by the new private cloud, right? 18:55:45 <skvidal> feel free to comment/edit, whatever 18:55:50 <skvidal> nirik: it's definitely related :) 18:55:59 <nirik> yeah. 18:56:02 <skvidal> if there are any pieces folks want to look at - please let me know 18:56:09 <skvidal> or come by #fedora-buildsys 18:56:14 <skvidal> or post to buildsys@lists.fedoraproject.org 18:56:48 <skvidal> anyway that's all 18:56:56 * skvidal loves the sound of crickets 18:56:56 <nirik> cool. 18:57:16 <nirik> I think there's tons of nice stuff there... I look forward to breaking^Wusing it. 18:58:00 * jsmith can't wait to use it too... 18:58:35 <nirik> ok, anything else? or shall we close out the meeting now? 18:59:09 <nirik> ok, thanks for coming everyone. ;) 18:59:12 <nirik> #endmeeting