18:00:00 #startmeeting Infrastructure (2012-07-26) 18:00:00 Meeting started Thu Jul 26 18:00:00 2012 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:00 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:01 #meetingname infrastructure 18:00:01 #topic Ola! 18:00:01 #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean 18:00:01 The meeting name has been set to 'infrastructure' 18:00:01 Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean 18:00:10 yo 18:00:12 * lmacken 18:00:18 yo 18:00:19 * CodeBlock here 18:00:30 sdrfed17 here 18:00:46 kinda here 18:00:54 sdrfed17 here 18:01:40 * nirik will wait a min for a few more folks to wander in 18:01:42 here 18:03:00 ok, lets go ahead and get started I guess. 18:03:09 #topic New folks introductions and Apprentice tasks. 18:03:14 If any new folks want to give a quick one line bio or any apprentices 18:03:14 would like to ask general questions, they can do so here. 18:03:24 anyone new or with apprentice questions? ;) 18:03:59 * nirik will move on then. 18:04:01 * skvidal is here 18:04:02 sorryt 18:04:17 no worries. 18:04:19 #topic Applications status / discussion 18:04:31 ok, new application(s) news this week or upcoming ? 18:04:38 We deployed https://fedoraproject.org/openhw2012 yesterday :) 18:04:44 * nirik cheers. 18:04:45 yay 18:04:59 and yesterday I also enabled Python hash seed randomization across all of our mod_wsgi apps. 18:05:20 lmacken: was that just mod_wsgi stuff? I thought it could affect other python cgis too? Or was I misremembering? 18:05:35 nirik: I think we covered the upload.cgi as well 18:05:40 ok, cool 18:05:46 that was on pkgs01, right? 18:05:57 yep. 18:06:20 hmm, the fix didn't make it there... I'll fix, and give the list another review :) 18:06:53 sounds good. 18:06:58 Application related -- but I'd save for discussing in open floor: We should discuss the web application frameworks RFC. 18:07:09 no major news here. working on fedmsg certs and trying to get all the fedmsg stg stuff safely into production before the freeze. 18:07:10 the next mod_wsgi release will have the ability to randomize the hash seed in the wsgi config, so we'll keep an eye out on that and plan to roll that out instead of this hotfix at some point 18:07:11 abadger1999: good idea. 18:07:53 * mdomsch_ 18:08:13 ok, any other upcoming apps news? anything folks want to try and land before the alpha freeze? (2012-08-07) 18:08:44 mediawiki 18:09:03 I think we can swing trying it in stg 18:09:05 smooge: yeah, would be good to get that moved along. In staging at least... 18:09:06 fedorahosted app, but that's not going to happen. 18:09:07 but I am not sure 18:09:16 oh, and is there a request in with RHIT to take on the haproxy port for websockets? 18:09:36 threebean: not yet. I can file that. Did we decide what port we wanted there? 18:09:42 * smooge wants to put haproxy on the cloud systems 18:09:51 s/haproxy/websockets 18:10:09 oh, now that the hosted list migration is done, skvidal we should talk OOB about what to do w/ ansible keys for the fedorahosted app CLI to create lists. 18:10:13 smooge: it needs to talk to the bus tho, so I don't htink that will work. 18:10:21 CodeBlock: ok 18:10:41 nirik, ugh. ok 18:10:49 smooge: you are out at site next week... would you like me to work on mediawiki to get it moving while you are otherwise busy? 18:10:52 nirik: No decision. I'll get back to you with one. 18:11:17 threebean, we will need to know what systems, where, how and what traffic will look like so that the firewalls and sensors know about it 18:11:24 #info need to finalize our websockets port and plan. 18:11:39 nirik, I think that would be a good idea. Would also help me figure out what I have broken in the rpmn that I don't see 18:11:52 smooge: ok. 18:12:03 #action nirik to get mediawiki updated in stg so we can test. 18:12:40 #info Python hash seed randomization deployed to all wsgi app servers. 18:12:55 #action lmacken to check any hosts that might also need the hash seed rand fix. 18:12:56 yay 18:13:00 * athmane notes that fedora pastebin is almost ready (sticky-notes in epel6 and fedora theme/skin is ready), just waiting for a dev box 18:13:15 athmane: we did make one. ;) paste01.dev 18:13:39 athmane: did you get added to the sysadmin-paste group? if not, lets sort it out after the meeting in #fedora-admin? 18:13:53 not yet 18:13:54 * skvidal makes a sysadmin-eats-paste group, adds everyone to it 18:14:11 heh. 18:14:21 ok, anything else application wise? or shall we move on? 18:14:49 I want paste 18:14:50 let's move on :) 18:15:05 but not the horse feet type this time 18:15:06 abadger1999: perhaps during the next freeze if you're interested I can look at trying to do a pkgdb release. All those hotfixes are bugging me. ;) 18:15:33 nirik: Sure. It'll be very very messy though. 18:15:41 nirik: But if you have time, you and I can do it. 18:15:47 oh? just too long since last one? or ? 18:15:50 * abadger1999 pencils in free time. 18:16:03 * skvidal adds 3 hours to the day 18:16:10 there - 27 hours for you, abadger1999 18:16:10 nirik: yeah.. long time. Lots of changes. developer of the code that I wanted moved into packages is awol. 18:16:11 abadger1999, let me know if you need some help for this 18:16:21 nirik: things like that. 18:16:30 yeah. ;( 18:16:37 skvidal: Thanks! You get me the best presents :-) 18:16:45 :) 18:16:45 well, I would like to get it done... so, happy to help. 18:16:55 #topic Sysadmin status / discussion 18:17:03 so, sysadmin news this week/upcoming? 18:17:15 #info smooge to be on site at phx2 next week. 18:17:23 I will be in PHX2 next week 18:17:26 or I could lag a lot 18:17:41 I have a full week planned out. will email it to people in a short while 18:17:53 I also set a 4 hour maint window next tuesday. 18:17:58 So, if there are things we need to reboot, unplug, or upgrade, we can do them then. 18:18:31 Some other notable things from this week: 18:18:49 #info got delegated reverse dns working at two of our sites (osuosl and coloamerica) 18:18:55 yay 18:19:12 now to see if we can get it elsewhere? 18:19:12 #info mass rebuild finished up and we found at least one thing to tweak on the builders 18:19:55 skvidal: how's the repave of builders going? pretty easily? 18:20:06 yah 18:20:07 #info got a cgit theme setup 18:20:08 * pingou gtg 18:20:12 go look at the reinstall notices 18:20:22 I've got buildvmhost-01->04 done now 18:20:26 and 08 and 09 18:20:29 if people didn't see it: http://fedorapeople.org/cgit/ (all the cgit sites) now have a new theme. 18:20:31 and I'm reinstalling them now 18:20:35 the buildvm's 18:20:38 cool, yeah, I see them flowing in 18:20:48 I wrote a script to do reinstalls of buildvmhost's easier 18:20:53 there's a lot of sitting here and watching it 18:21:08 question about names.. so I am clear 18:21:21 buildvmhost is the DOM0ish thing and not the builders 18:21:35 correct? 18:21:37 buildvm == virtual machine 18:21:43 #info cgit migration next week (the 1st). 18:21:44 buildvmhost == the host on which the vm run 18:22:35 thanks. I thought so but I would prefer to know versus assume 18:22:49 #info fedorahosted.org lists migration to lists.fedoraproject.org is done. 18:22:55 #undo 18:22:55 Removing item from minutes: 18:23:02 #info fedorahosted.org lists migration to lists.fedorahosted.org is done. 18:23:54 I'd like to do the hosted03->hosted01/02 pair soon. I was thinking perhaps next friday... then there would be lower load and we could test stuff a bit easier over the weekend. 18:24:38 ok, any other upcoming or recent sysadmin news? 18:24:53 #topic Upcoming Tasks/Items 18:25:01 #info 2012-07-30 to 2012-08-03 PHX2 trip for smooge 18:25:01 #info 2012-07-31 21UTC to 01UTC outage window 18:25:02 #info 2012-08-01 nag fi-apprentices 18:25:02 #info 2012-08-01 gitweb to cgit migration 18:25:02 #info 2012-08-03 hosted03-> hosted01/02 migration (tenative) 18:25:02 #info 2012-08-07 to 2012-08-21 F18 Alpha Freeze 18:25:04 #info 2012-08-08 drop inactive apprentices. 18:25:06 #info 2012-08-21 F18 Alpha release. 18:25:08 #info 2012-08-31 end of 2nd quarter 18:25:10 #info 2012-09-11 to 2012-09-25 F18 Beta Freeze 18:25:14 #info 2012-09-25 F18 Beta release 18:25:16 here's what I have on the list... 18:25:25 if anyone has anything else they would like to schedule or note, let me know. 18:26:10 ok. 18:26:18 #topic RFC on application frameworks 18:26:31 https://lists.fedoraproject.org/pipermail/infrastructure/2012-June/011922.html 18:26:48 So I evaluated where we're at in that message 18:27:26 and came up with the tentative idea to choose one high level, feature rich framework and one low entry-barrier, low dep framework for us to standardise on. 18:27:56 That would reduce how much maintainance and knowledge of other upstream frameworks we have to keep lugging around with us. 18:28:09 Question 1) Do we have agreement that that's a good idea? 18:28:29 Question 2) What two frameworks do we want (Strawman: flask and pyramid) 18:28:44 I think it sounds reasonable... but there may also be a lot of knowledge overlap between things like TG2 and pyramid? or is that not the case? 18:28:52 Todo: Start porting TG1 apps to one of the chosen frameworks. 18:29:02 lmacken, threebean ^ 18:29:13 *some* overlap 18:29:16 whats TG1? mm, voting, ? 18:29:20 bodhi 18:29:28 pkgdb, fas 18:29:33 mm, bodhi, pkgdb, voting fas. 18:29:33 whee. 18:29:42 if you name it at random, it probably is. 18:29:54 We're getting quite a few tg2 apps now too. 18:30:09 but not to the same level or criticality yet. 18:30:22 packages/tagger/raffle? 18:30:27 busmon 18:30:28 yep, those are tg2 18:31:14 so we obviously have a vested interest in TG* :) 18:31:25 Based on what we have, I can see us moving to tg2 instead of pyramid. But I'd hate to do that and then have either tg3 come out or tg2 die off. 18:31:32 question.. how was the openhw app written in.. and how did that og 18:31:41 smooge: pyramid 18:31:53 it went great... I'm writing all of my new apps in it, for the most prat 18:32:01 **part 18:32:06 are there cases where TG2 is "better" for an app than pyramid? or is there a lot of overlap there? 18:32:44 nirik: yes, TG2 still offers a "full stack" out of the box, where as pyramid does very little by default 18:33:09 Yeah, there's more lock-in with TG2. 18:33:18 as a result, if you wrote one TG2 app, you'll understand another one 18:33:28 that translation of knowledge doesn't apply as much with pyramid 18:33:31 so, perhaps: 'simple: flask, more complex: TG2 or pyramid" (yeah, thats another framework tho...) 18:33:38 lmacken could write one one way and I could write another in a different way. 18:33:45 I'd rather not have both tg2 and pyramid. 18:33:45 i'd say simple: pyramid :\ 18:34:06 abadger1999: that's not going to happen 18:34:14 Cool. 18:34:56 so, perhaps we approach this another way... 18:35:06 tg2 + SO + kid :-) 18:35:21 do we all agree that we want to move the tg1 apps to something else thats longer term supported? 18:35:26 mdomsch: TG1, you mean :) bodhi is still rockin' that stack :) 18:35:33 mdomsch: :-) 18:35:43 portin all of our TG1 apps to somethin else will be an epic endeavour that will take years 18:35:48 lmacken: no, I meant tg2. TG2 - SA + SO +kid :-) 18:35:56 mdomsch: oh, a hybrid stack :P 18:36:24 yeah, but in years, where will TG1 be? 18:36:29 But porting is really necessary.... I just fixed a bug in cherrypy2 this week which reminded me that cherrypy2 (the basis of tg1) is dead upstream. 18:37:29 so it's cost of porting to newer framework with less maint cost vs cost of maintaining older framework with less porting costs. ;) 18:37:34 I think we need to port too, but we have some time to do it. The code will keep running in the meantime. 18:37:41 18:37:46 I believe a future version of TG1 will support CP3 18:38:15 I think it's good to think about this and plan for it rather than be surprised when rhel7|8|orwhatever doesn't run the framework we have. 18:38:17 We can't leave it until a major bug is found in part of TG1's stack as we won't be able to port quickly. 18:38:32 the key is usin a framework that is well supported in Fedora, and one that we have people working with upstream (tg, and pyramid, for example) 18:38:36 But we can work through things one app at a time and get there when we get there. 18:38:47 lmacken: I believe tg-1.5 supports CP3 now. 18:38:57 abadger1999: yeah, I think so. So, TG1 is still evolvin. 18:39:10 when is anticipated EOL for TG1? 18:39:14 But TG1.anything doesn't have a lot of support upstream... there's what -- three devs that care about it? 18:39:17 mdomsch: there is no EOL 18:39:38 lmacken: that's 2.9 more devs than MM has today :-) 18:39:54 threebean, abadger1999 I would like to move away from the no EOL or "we have time". We have been saying that for 3+ years now. 18:40:12 smooge: (didn't know that) 18:40:17 and TG-1.5 has incompatibilities with TG1.0/1.1.... so jumping from tg-1.1.x to tg-1.5.x doesn't seem like a good cost::benefit ratio 18:40:27 smooge: +1 18:40:29 when stuff like that happens the EOL somehow always shows up on a Tuesday and we end up having to fix it on Monday somehow 18:41:06 +1000 18:41:23 I'm surprised that Murphy hasn't struck us down yet. 18:41:32 right. 18:41:33 I am paying him off left and right 18:41:48 ha :) 18:42:20 no really. 18:42:29 I haven't seen a bonus in 3 years now 18:42:41 anyhow, I think we need to think about this and come up with a plan... 18:42:48 I send them Western Union via mmcgrath 18:43:06 I agree 18:43:17 smooge: thanks for keeping him off our backs. ;) 18:43:19 so, I don't think the bucketing of frameworks into "this is for simple apps, these are for complex apps" works well. I'm pretty sure you can write nice complex apps in flask. 18:43:40 18:44:09 so, another approach: is there a way we can say "you can use whatever framework you like, but it must have this level of support and be stable for N years" ? or is that too difficult/wide open. 18:45:07 I think even as we are, we are much better off than other shops that have many more frameworks... ruby on rails, jboss, tomcat, etc. 18:45:48 I know I'm getting old when I say "don't make me learn ruby..." 18:45:53 I think the idea of supported for N years is not going to happen unless we went with jboss + Enterprise Ruby 18:45:59 * nirik shudders. 18:46:06 s/Ruby/Java/ 18:46:25 django & drupal seem to not suffer this as much, or maybe i'm just not looking close enough 18:47:00 mdomsch: they do too... perhaps not as much porting work, but they do have the issue. 18:47:13 mdomsch: drupal is actually the worst when it comes to support for N years/compat issues. 18:47:35 yeah, so django... From a maintainance point of view, I think it's interesting.. but from a coding and sysadmin/deployment side... it seems to be a big shift in style. 18:48:20 django seems to adopt the application server style of deployment similar to how I hear JBoss is structured. 18:48:39 So it really wants you to write apps that are then run inside of one django application server. 18:48:40 yeah, so it's 1 app per server... 18:48:41 (web2py works that way too) 18:48:42 yep 18:49:17 I'd really like to see us dogfood more JBoss web stuff 18:49:23 * lmacken *shifty eyes* 18:49:59 * skvidal can't tell if trolling or seriou 18:50:41 I don't think thats going to be too likely unless we had a big influx of jboss application developers/sysadmins who really wanted to help and could convince us that it was a viable platform for what we want to do. 18:50:49 * abadger1999 asks spot to send lmacken to JBoss Summit instead of PyCon this year ;-) 18:51:05 abadger1999: hahaha, nooooooooo 18:51:55 so, how about this at least: can we all agree that we should NOT add any new frameworks, and ask tg1 using apps to please move to one of the other frameworks we already have deployed? 18:52:02 Alright.. so I think we're leaving JBoss and drupal off the table :-) 18:52:28 nirik: you can try asking the apps to move, but I doubt they'll listen ;) 18:52:35 nirik: 2 things: (1) Do we wnt to include pyramid? -- openhw has a sunset so we don't truly have pyramid deployed. 18:52:52 bodhi will be the first real pyramid deployment 18:52:56 nirik: (2) That doesn't satisfactorily answer the question: what should we port the tg1 apps to. 18:52:58 abadger1999: I assumed so, since we have bodhi2.0 in the pipeline 18:53:18 lmacken: hey you! app! listen to me! computer! 18:53:20 Oh... or flask for that matter. 18:53:40 yeah, in that case it would be up to the people doing the work (whoever they were) 18:54:00 we have bodhi2.0 in pipeline, and some flask apps in the pipeline. But neither of those deployed. 18:54:01 seems like there is some decent flask momentum in fedora, in terms of packaging. Not sure if we have any upstream presence or not. 18:54:13 oh, and fedbadges is pyramid as well 18:54:21 * tflink has been lurking but if Flask is going to be discouraged, it would be nice to know sooner than later 18:54:23 so I guess... I'd rather just say "New apps in flask or pyramid" right now. 18:54:38 there are a couple of QA things that are young but written in Flask 18:54:43 and then if we decide to expand that later, do so later. 18:55:03 fwiw, openstack dashboard (formerly horizon) uses django 18:55:15 abadger1999: well, thats fine with me... lmacken / threebean ? does that cause problems for you with new tg2 apps? 18:55:15 neither of those have proper FAS identity layers, so I'd be cool with adding TG2 to that list 18:55:22 tflink: Interesting datapoint.... do you guys have other frameworks too or just flask right now? 18:55:33 abadger1999: tg2 and flask for now 18:55:38 k 18:55:56 lmacken: I think that ianweller wrote a fas-flask already. 18:55:59 ianweller: ^ ? 18:56:12 unless you count autotest - that's django but we're pretty much consumers there 18:56:24 http://pypi.python.org/pypi/Flask-FAS 18:56:24 nirik: I just have busmon (tg2) in stg. 18:56:27 abadger1999: that's good to know. I also got a ping from a Red Hatter asking me how to hook up the FASWho middleware to a TG2 app... so I'm not sure we want to turn our back on that stack just yet 18:56:45 http://git.fedorahosted.org/cgit/flask-fas.git/ 18:56:49 even though that middleware isn't the greatest 18:56:52 * mdomsch needs more weeks per day 18:56:58 lmacken: -- What I'd like is to say new apps only in X or Y. 18:57:08 (meaning pick 2) 18:57:43 flask and pyramid 18:57:44 and we'll support the tg1 and tg2 stuff in python-fedora (auth layers, templating, utility functions) until we have no more use for them in Infra. 18:58:01 port all tg1 to one of our chosen frameworks 18:58:13 that takes quite a while. 18:58:37 I'm sure it will. 18:58:40 Then we look at porting other apps (tg2 if pyramid is out or pyramid if tg2 is out) 18:59:13 s/porting/re-writing/, essentially 18:59:26 So... if we pick pyramid... it's likely to be years before the tg2 stuff in python-fedora is removed. 18:59:32 none will be a standard "port" 18:59:33 heck, maybe even decades ;-) 18:59:53 I'll use port loosely there ;-) 18:59:54 yeah, port is not the right word there... 18:59:59 * threebean will do it all. 19:00:13 sounds good, threebean. I guess we're done here? :) 19:00:17 yup 19:00:17 -) 19:00:26 anyhow we are at time here... is there any decision we can make? or ? 19:00:26 :D 19:00:42 agreed threebean to re-write all the apps! :) 19:00:48 #action do it 19:00:50 am I understanding correctly that new development in tg2 is being slightly discouraged? 19:01:25 * lmacken not sold on discouraging tg2 yet 19:01:38 why? 19:01:50 it feels like bet hedging. 19:02:05 all we want to do is tell people to focus on one thing -and hopefuly on a stable version of that 19:02:11 so that it is not a moving target 19:02:12 skvidal: because we have tons of tools built around it, upstream commit rights, tons of experience & real world deployments 19:02:24 but he's talking about NEW devel 19:02:27 not maintenance devel 19:02:59 right. 19:03:46 tflink: For me personally, I want to discourage new developemnt in either tg2 or pyramid. they both have lots of deps to maintain and aren't altogether easy for a new person to get started with. they have a similar niche in those ways. 19:04:41 I disagree... pyramid doesn't have very many deps, and for a new dev getting a tg2 app hooked into fas is extremely easy. 19:04:47 abadger1999: don't most frameworks have a decent number of deps? Flask by itself doesn't but I usually use plugins which bring in deps 19:04:48 from my side I don't care much which ones we use (since I am not writing the apps), but fewer would be nice in that it's less to deploy and maintain from a sysadmin side. 19:05:05 okay... soo do we want tg2 and either pyramid or flask? 19:05:48 we're over time. 19:05:55 Do we do this with a vote? 19:06:10 proposal: We're going to pick two web frameworks to standardize on 19:06:25 yeah, lets continue on list/next week/out of band 19:06:33 If that is ok'd now, I'll send out a mail to the list to discuss which two. 19:06:45 and threebean can start porting :-) 19:06:49 two is too arbitrary :( 19:06:52 I'll raise a counter-proposal: We pick three web frameworks to standardize on. nix tg1 and keep tg2, pyramid, and flask. 19:06:59 ok... one? 19:07:09 threebean: +1 19:07:13 lets go with 0... less bugs. 19:07:16 ha 19:07:19 nirik: Sold! 19:07:28 Ohh... I think we're too the point of absurdity :-) 19:07:34 anyhow. 19:07:48 lets continue over in #fedora-apps and try and come up with a consensus... 19:07:56 #topic Open Floor 19:08:05 anyone have anything for open floor real quick? 19:08:21 yes, I want to notice one quick thing 19:08:38 the time on the main infrastructure wiki page for meeting start time seems to be incorrect (it says 19:00 UTC) 19:08:55 oops. 19:09:06 puiterwijk: can you correct it? or tell me which exact page and I will fix it. 19:09:21 I wanted to introduce myself as hopeful new contributor, but I will wait for next weeks welcome round 19:09:25 nirik: I'll fix it 19:09:37 thanks much. ;) Sorry for the confusion... 19:09:53 ok, thanks for coming everyone! 19:09:55 #endmeeting