18:00:52 #startmeeting Infrastructure (2012-07-05) 18:00:52 Meeting started Thu Jul 5 18:00:52 2012 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:52 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:52 #meetingname infrastructure 18:00:52 The meeting name has been set to 'infrastructure' 18:00:52 #topic Good morning Fedora 18:00:52 #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean 18:00:52 Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean 18:00:57 * skvidal is here 18:00:59 * abadger1999 here 18:01:08 * relrod is here 18:01:11 sdrfed17 is here 18:01:36 * ingm4r is 18:01:48 hey 18:02:07 * fchiulli is in the rafters. 18:02:47 ok, cool... lets go ahead and dive on in then. 18:02:52 #topic New folks introductions and Apprentice tasks. 18:03:02 If any new folks want to give a quick one line bio or any apprentices would like to ask general questions, they can do so now. Anyone? 18:03:33 any apprentice tasks to undertake 18:03:35 ? 18:04:28 sdrfed17: we have a list of easyfix tickets... http://fedoraproject.org/easyfix/ 18:04:34 look down to fedora-infrastructure. 18:04:48 I've been gone the last week or so, but will try and file some more. 18:05:07 If you have any questions on any of them, do ask in #fedora-admin and someone will take a look. 18:05:26 ok thanks nirik 18:05:31 i will take a look 18:05:53 cool. Are you more interested in sysadmin type tasks? or application development/programming? or both? 18:06:11 sysadmin 18:06:15 * threebean shows up late 18:06:20 I would like some new sysadmin-tasks,too 18:06:39 ok cool. 18:07:05 If either of you would like to be added to the apprentice group (if you aren't already) we can do so after the meeting over in #fedora-admin. :) 18:07:32 no i am already in the apprentice group and able to access bastion host 18:07:53 great. (Thought so, but my memory might be hazy after vacation. ;) 18:08:25 ok, moving along then... 18:08:25 #topic Applications status / discussion 18:08:42 * ianweller is here, kinda 18:08:46 any new applications news this last week? threebean / abadger1999 / lmacken / CodeBlock / pingou / ianweller 18:09:09 ianweller has been rewriting elections as a flask application 18:09:13 i sent my elections status to the list 18:09:29 ianweller: I think it's got good potential 18:09:32 yeah. I need to read again and reply to abadger1999's post on applications and ianweller's election post. 18:09:38 18:09:55 My post on applications and web frameworks has gone unanswered but RH Summit was last week. 18:09:58 i am going to back out on the election admins support for this release 18:10:09 and revert to the elections fas group 18:10:12 lmacken, threebean: Would be good if you weighed in on that post. 18:10:16 Still need to test the fedorahosted web app, but it's ready for testing. Still planning on testing it on hosted01 before we switch over to 01/02. Since it's a semi-real (if a bit out of date because it probably hasn't been rsynced in a while) hosted environment. 18:10:28 because the implementation really needs some discussion 18:10:30 abadger1999: so the short summary was: we have a lot of frameworks, can we consolidate some? 18:10:36 nirik: and 18:10:41 "let's pick a 'big app framework 18:10:42 ' 18:10:47 and a 'small app framework' 18:10:49 " 18:10:54 yeah. 18:10:55 18:11:05 ie: moksha/tg/$flavor_of_the_week for $bigapp 18:11:13 and flask for $smallapp 18:11:17 sounds like people are leaning toward: flask -> small app, pyrimid -> big app? 18:11:17 or something along those lines 18:11:23 yeah. 18:11:42 nirik: I think so... but we still have a lot of TG2 apps that seem to be in the pipeline. 18:11:47 abadger1999: will do 18:11:47 So I'm not 100% sure of that. 18:11:50 abadger1999: nod 18:11:59 threebean and lmacken would know that story better. 18:11:59 abadger1999: and that is, in fact, the crux of the issue 18:12:00 relrod: I am going to resync again too... so it should be pretty up to date. 18:12:07 abadger1999: the new-app-platform moving target 18:12:13 yep. 18:12:24 Especially as I look at porting the TG1 apps. 18:13:05 we also have lmacken's openhw2012 app deployed to a temporary box that skvidal set up. The repo for the app code is: https://github.com/lmacken/fedora-openhw2012 - we just need to add the proxy stuff but if you want to test it, -L via lockbox and go to /openhw2012 18:13:15 that's all the app news I have I think 18:13:28 That box has a deadline, too 18:13:37 after the end of july 18:13:40 we should be able to kill it 18:13:48 I've set a calendar reminder for myself for august 5th 18:13:49 ok. 18:13:56 I've got two open infra tickets for fedmsg stuff -- one for websockets proxying, another for a fedmsg CA. No rush, just remembering. 18:13:59 if the box ain't gone by then - I go bug mo and spot and lmacken until it is 18:14:07 skvidal: set a cron job to make it explode :) 18:14:17 threebean: I've been noodling on the CA thing 18:14:26 threebean: yeah, I hope to get to those sometime too. ;) 18:14:28 hi all - sorry i'm late 18:14:44 :) 18:15:00 threebean: ah yeah -- the websockets thing seems like it could be explained some. 18:15:08 threebean: how is the bus looking in staging? I assume those two tickets need fixing before we look at when we can move to prod? 18:15:13 since it's a relatively new thing 18:15:41 that we've never deployed before. 18:15:55 nirik: Here's a table of the stg status - https://github.com/ralphbean/fedmsg/blob/develop/doc/status.rst 18:16:50 abadger1999: agreed. so websockets is a natural fit for doing push-based notifications to a web client. we used to dabble with Orbited as a TCP proxy, but websockets seems to have out-moded it. 18:16:53 threebean: ok, so is the thought to get it all working in stg and push to prod at once? or just get a good bunch done, enable those in prod and work on the others as time permits? 18:17:20 nirik: I'd originally thought: "get it all working in stg first" but I'm leaning towards a move to production sooner. 18:17:21 threebean: so - websockets, unless I misunderstand is going to need some intriguing access to get to the server side from outside of phx2 18:17:31 threebean: is it possible we can use websockets tunneld via vpn? 18:17:41 nirik: reason being that my modules-staging dir is getting unwieldly and I'd like to slim it down before growing it too much more. 18:17:44 threebean: i guess - I am unclear what is going to be connecting to the websocket server 18:18:16 threebean: sorry for the pile-on of questions 18:18:25 threebean: yeah, I think it makes sense to get the ones done setup and working in prod and then do another stg batch. 18:18:52 skvidal: (np) -- the client will be a browser, most of which support native websockets-on-the-client these days. 18:19:18 threebean: so - an END user needs to have a full-duplex connection to this service? 18:19:53 well, for the service I have in mind, they only *need* the server->client stream. 18:19:58 threebean: how well does that cope with things like loadbalancers? 18:20:22 *should* be fine, although I've no experience with it in practice. 18:21:07 we can work on getting that working and sorted out of band? 18:21:14 sure 18:21:20 yeah :) 18:21:22 any other apps news? 18:21:59 ok, moving along 18:22:04 #topic https://fedoraproject.org/wiki/FAD_Infrastructure_Security_2012 18:22:22 I need to gather more info and then ping rbergeron when she's back and see what kind of budget we can get. 18:22:53 have we settled on a _when_? 18:23:07 if anyone interested in attending could add themselves to: https://fedoraproject.org/wiki/FAD_Infrastructure_Security_2012/attendees that would be great. 18:23:11 skvidal: nope... 18:23:28 i think I mentioned this last meeting when you weren't here -- does it make sense to decide who needs to attend, who we want to attend, and then decide where it makes sense to have it? 18:23:30 * dgilmore would prefer september or november 18:23:41 * dgilmore really wants to be there 18:23:50 abadger1999: sure 18:23:55 but im going to australia for 6 week in just over 2 weeks 18:24:04 and ill be travelling most of october 18:24:13 abadger1999: yeah, it could. 18:24:14 * relrod would prefer before end of august because of the semester starting :/ 18:25:02 relrod: I think thats going to be too soon to have logistics sorted... also we will be in alpha freeze at the beginning of aug... 18:25:08 skvidal: I think you have the best overview of who's been working on what... do you want to make up the short list of must-have people? 18:25:23 umm - sure? 18:25:34 heh, not to put you on the spot :-) 18:25:45 nirik: :( 18:25:56 we need captain fas, captain infra, capt various-kinds-of-auth 18:26:01 I could come up with a list too... but I'm not sure if people like mricon are must-haves or would-like-to-haves 18:26:03 relrod: do you have a fall break or the like? 18:26:18 abadger1999: I think mricon would be happy to come by NC for a while ;) 18:26:29 but I doubt he is a MUST have 18:26:36 * relrod pulls up the college calendar 18:27:06 also - if anyone is thinking of doing this in the first week of october - you have another thing coming :) 18:27:12 not sure im a must have but would like to work on ca bits 18:27:18 as well as a few others 18:27:30 auth stuff etc 18:28:03 nirik: yeah that could work actually I think 18:28:26 skvidal: im pretty much out all of october 18:28:44 I think primary we need: fas developers, sysadmin folks who can work on the setup part, possibly packaging if we still need packages for pam modules/etc. 18:29:08 I can make this easy.. While I would love to attend.. I think my usefulness would be small... so you can skip me in the scheduling unless you find a reason I have to be there :) 18:29:31 smooge: I thought you were captain various-kinds-of-auth 18:29:38 if dgilmore could make it we could sort out sign* security/management processes, which would be a good secondary goal. 18:29:39 oh well 18:29:47 well I can be :) 18:30:09 skvidal, I just wanted to make your job of many body program as easy as possible 18:30:26 * skvidal wonders how he got saddled with this 18:30:30 * skvidal glowers at abadger1999 18:30:42 :-) 18:31:12 nirik: id like to talk with smart people about our ca setup 18:31:44 skvidal: You could put on the wiki [needed roles] Captain FAS, Captain pam, [...] Captain Reynolds, and then make sure we have someone from each. 18:31:45 ok, so lets work on: a short list of must have folks, and nice to have folks... and gather more info? 18:31:58 abadger1999: guess who is captain fas? :) 18:32:04 abadger1999: captain tightpants is more like it 18:32:20 dgilmore: for koji? 18:32:29 dgilmore: I'd like to discuss the ca stuff with you, actually. 18:32:32 nirik: yeah 18:32:52 nirik: yeah, we can probably come up with our list of people concurrent with asking rbergeron about budget/template of what an organizer needs to know/turn in to make it work. 18:33:40 yep 18:34:00 #action nirik to continue to work out plan and scheduling and budget for a fad. 18:34:13 skvidal: anytime 18:34:17 dgilmore: ok 18:35:06 ok, moving on then? or anything else we should discuss on fad? 18:35:33 #topic Upcoming Tasks/Items 18:35:33 #info 2012-06-26 Fedora 15 end of life. 18:35:33 #info 2012-07-05 remove people with pkgdb bugzilla issues. 18:35:33 #info 2012-07-05 nag fi-apprentices 18:35:33 #info 2012-07-12 drop inactive apprentices. 18:35:36 #info 2012-08-07 to 2012-08-21 F18 Alpha Freeze 18:35:38 #info 2012-08-21 F18 Alpha release. 18:35:50 I had f15 eol on there still because I didn't see the announcement. 18:35:56 I guess it's waiting on rbergeron being back 18:36:08 anything else folks would like to schedule or note? 18:36:14 I have some things: 18:36:26 a) 6.3 updates. Next week? 18:36:43 sure - or we can apply them all tomorrow 18:36:46 b) hosted03 -> hosted-lists01 migration? where were we on that? 18:36:46 what could go wrong? 18:36:58 nirik: have we done enough speed tests? 18:36:59 heh. 18:37:18 c) hosted03 -> hosted01/02 migration redux 18:37:42 smooge: were you working on the hosted lists thing? 18:37:44 I am doing mailing list transitions next week. 11 will be EPEL -> fedoraprojects. 12 I hope to have hosted -> lists.hosted 18:37:54 cool 18:38:27 smooge: cool. You need me to file outage tickets/announce those? or you got it? any particular times? 18:38:54 I am waiting on some people in RHIT to tell me when they want to go to PHX2. If I don't have it by Monday.. Kevin and I will come up with a date and I will go 18:39:01 skvidal: on c) it looked like it was reasonably ok performance wise for me with nfs... but If you have some more heavy hitting you could do against it, that would be great! 18:39:14 nirik, I will be putting in the tickets tomorrow. 18:39:15 I will see if I can get to it 18:39:21 nirik: what's your timeline for making that switch, again 18:39:53 if we could do it at least a week or so before alpha freeze that would be nice... 18:40:07 but I guess hosted isn't frozen anyhow in alpha/beta... 18:40:22 smooge: sounds good. 18:41:26 ok, anything else upcoming? 18:42:23 #topic Open Floor 18:42:28 anything for open floor? 18:42:40 * relrod will set up the fedorahosted app on hosted01 this weekend for people to play with next week 18:42:55 I got pulled out into a meeting earlier, but I'm interested in becoming an infra apprentice 18:43:46 relrod: how weil the hosted app deal with hosted01/hosted02 split? 18:44:06 i was late getting here, but smooge helped fisnish getting me set up as an apprentice last meeting - not sure where to start helping out. 18:44:16 relrod: also - for maybe sometime in the future - if we end up sharding out projects to various servers - any idea on amount of modification necessary to make that fly? 18:44:29 skvidal: shouldn't matter at all, as long as the repo dirs (/git, /hg, etc, etc) remain in sync. 18:44:41 relrod: and if they don't? 18:44:43 samkottler / jaysonr: we can help point you the right way in #fedora-admin after the meeting. ;) 18:44:53 nirik: great, thank you :) 18:44:54 relrod: ie: if we split them so that we only have the bits necessary on a set of hosts? 18:45:12 nirik: thanks 18:45:14 skvidal: Well it'd be the same as if you followed the (current) SOP. 18:45:28 ok 18:46:58 related to that... 18:47:00 skvidal: as for sharding out to different servers, the CLI doesn't care where it's run. Run it on whatever server you want, and it'll set up the project on that server. We could put in sanity checks ("this project should really be processed on hosted123 not hosted456, are you sure you know what you're doing? Y/n") 18:47:21 relrod: well the goal I'd have is patching it into dns 18:47:25 so when you make project foo 18:47:44 it creates a dns entry foo.fedorahosted.org and points that to a specific server (or a ocuple of servers, depending) 18:47:48 we may be setting up a hosted-agilo01 or something... I can't get the agilo trac plugin to play nice with others. we have 2 groups that really want to use it. 18:48:09 nirik: so - that's how we can start it 18:48:12 so, this would be one way to test that... ie, if you want a agilo instance, it has to setup on that hose. 18:48:14 host. 18:48:17 we can move to foo.fedorahosted.org 18:48:26 for those folks who want to use agilo 18:48:29 skvidal: well that might be interesting (because it'll have to commit to the dns repo to do that, right?) 18:48:30 and just point them over to there 18:48:37 nirik: same - backend gluster FS, of course 18:48:43 relrod: indeed 18:49:01 we have a, currently unused, user setup to do things like that 18:49:27 brb 18:49:45 * nirik nods. 18:49:47 * lmacken rolls in super late 18:49:55 ok, we can look at testing this out with the agilo stuff... 18:50:06 hey lmacken. Any app news to note this week? 18:50:25 skvidal: so perhaps we write a wrapper script for lockbox that: 1) makes the dns commit, 2) uses ansible/func/whatever to `fedorahosted -p 123` (where 123 is the request ID) on a given hostedXX box. 18:50:28 nirik: openhw2012 work, thanks to skvidal, relrod, and mizmo for their help. 18:50:56 relrod: maybe - more thoughts, i suspect 18:51:00 nothing else this week other than that. I'll have that app done this week. 18:51:02 relrod: but out of band from this meeting 18:51:07 lmacken: cool. 18:51:14 skvidal: sounds good 18:51:58 lmacken: did we want to announce packager/tagger? or was there still some polishing up happening there? 18:52:38 nirik: I think we're pretty much ready... I was worried about a few bugs last week, but none are really blockers as the app just mostly works. 18:52:40 and yes, it recognizes agilo needing to be processed on a box with 'agilo' in the hostname already. https://github.com/CodeBlock/fedorahosted/blob/master/bin/fedorahosted#L184 18:53:06 did we solve/fix the running out of space issue ? 18:53:08 nirik: I've got a CLI tool for the packages app in review now. It'll be a bit before it makes it into stable though. 18:53:09 Did we get the disk full problem sorted out? 18:53:11 nirik: we also need to get our inode monitoring setup on those boxes, as we had a backed up cronjob that caused another explosion recently 18:53:17 nirik: shouldn't block an announcement, though. 18:53:39 huh. I thought I had setup nagios to monitor that... 18:53:39 I can doublecheck 18:53:44 for some reason the 'yum makecache' cron job for fedora packages started flipping out for some unknown reason, and prevented the tagger beaker cleanup script from running. 18:54:04 are they the same job or separate jobs? 18:54:09 different... 18:54:11 lmacken: the yum makecahce script that I saw was segfaulting 18:54:22 lmacken: which means something is VERY wrong further down than yum 18:54:26 can we tweak cron to prevent one from blocking another? 18:54:32 skvidal: yeah, it needs a good hard look 18:55:00 skvidal: that's J5 code that needs some review... it definitely worked fine on community01.dev 18:55:30 I do not know how you make that code segfault w/o maybe replacing various libraries that python calls 18:55:39 I've seen some sporadic timeouts on the bugs pages too... but not enough pattern to really nail down. 18:55:48 the makecache code is..... not very complicated 18:55:57 nirik: yup, the bugs page is hurting after the new bugzilla upgrade... since we can't do multicalls anymore 18:56:09 nirik: Oh -about bugzilla -- do we need to coordinate with red hat in order to test the python-bugzilla update? 18:56:11 I'll give the bugs page another look next week, as I think I can improve some of those queries 18:56:38 lmacken: you might ask pingou and/or tibbs to look at your queries... they have poked at the python bugzilla stuff a lot recently.... 18:57:07 nirik: cool 18:57:09 abadger1999: good point. I think we can test it... we just need to see that it doesn't cause a ton of queries next to the old one. 18:57:27 18:57:46 lmacken: if packages could also have a feature to show all bugs in one page that would be nice for me at least too. ;) Thats what I used the pkgdb one to list... 18:57:52 here is the bugzilla 18:57:59 multicall issue we were hitting: https://bugzilla.redhat.com/show_bug.cgi?id=824241 18:58:00 #info need to test out new python-bugzilla with all our apps. 18:58:36 and new Python 18:58:38 lmacken: ok 18:59:11 ok, anyone have anything more? or shall we call it a meeting? 18:59:21 we updated python recently, it caused explosions, and we reverted it. have we made any more progress since? 19:00:08 * lmacken seems to recall us hitting https://bugzilla.redhat.com/show_bug.cgi?id=750555#c76 19:00:24 I didn't recall reverting it... I thought we just restarted things and they started working. 19:00:28 skvidal: you recall? 19:00:32 * skvidal looks up 19:00:44 the python update 19:00:49 I don't think we reverted it, did we? 19:00:55 I thought we restarted the things it broke 19:01:07 oh, excellent. in that case, I'll look into setting up hash randomization for our apps in staging next week. 19:01:07 yah we did 19:01:13 the issue was os vs posix 19:01:24 for which call was it. 19:01:30 cool 19:01:34 the module moved 19:01:36 I remember this 19:02:17 lmacken: great. 19:02:27 one sec - finding the bug to confirm 19:02:33 #action lmacken to work on hash randomization for apps next week in staging. 19:02:33 skvidal: I just posed it earlier 19:02:41 **posted 19:02:59 lmacken: that was for f16 19:03:27 and RHEL 19:03:27 I was looking at the item for el6 that we ran into 19:03:45 ok 19:04:52 lmacken: should we schedule a bodhi 2.0 meetup? figure out status and such? 19:05:03 nod - moved os.random to posix 19:05:49 nirik: yeah, sounds good to me. let's touch base next week and schedule something. 19:05:50 and the conversation from then in #fedora-noc said 19:05:58 I restarted a bunch of apps to cope with the change 19:06:04 but I left the python update in place 19:06:12 * skvidal is just making sure it is what I thought it was :) 19:06:28 lmacken: ok. I think if we actually schedule something we will make progress. ;) It's easy to keep putting off... 19:07:20 nirik: yeah, true :) the porting has been going well, and I'm pretty much at the point now where I'm going through years worth of feature requests and figuring out how to implement/integrate. But yeah, we should meet and discuss them soon. 19:08:05 * lmacken will sync up the bodhi2 status on the wiki & trac next week and plan a meeting to review & prioritize 19:08:07 lmacken: yeah. Would be a great time to make sure people are on the same page/features make sense to implement, etc. 19:08:43 I did see till was attaching a bunch of patches... might also be time to do a 1.x roll up if those patches are worth pushing out before 2.0 is done. 19:09:10 nirik: yup, definitely. 19:09:25 ok, cool... 19:09:32 if nothing else, will close out in a minute. 19:09:52 nothing from me 19:10:18 thanks for coming everyone! 19:10:21 indeed 19:10:21 #endmeeting