19:00:01 #startmeeting Infrastructure (2012-01-19) 19:00:01 Meeting started Thu Jan 19 19:00:01 2012 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:01 Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:01 #meetingname infrastructure 19:00:01 #topic Robot Roll Call 19:00:01 #chair smooge skvidal Codeblock ricky nirik abadger1999 lmacken dgilmore mdomsch 19:00:01 The meeting name has been set to 'infrastructure' 19:00:01 Current chairs: Codeblock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge 19:00:07 * skvidal is here 19:01:06 * nirik waits for more folks to wander in. 19:01:12 will start in a few. 19:01:39 * CodeBlock here 19:03:16 well, at least a few of us here... 19:03:30 * jac1bat0 is here 19:03:43 #topic New folks introductions and apprentice tasks/feedback 19:04:04 any new folks want to introduce themselves? or any apprentice folks want to ask about tickets or process or tasks? 19:04:59 #topic Post fudcon tasks and plans 19:05:11 ok, I had several items post fudcon to discuss. 19:05:19 #topic 2 factor auth 19:05:25 herlo was working on this... 19:05:29 so where are we at? 19:05:43 dangling from that preposition 19:05:44 :) 19:05:45 zing 19:05:56 okay - so it looks like we have a pam module thanks to npmcallum and herlo 19:05:59 which will do sudo, definitely 19:06:09 cool. 19:06:15 and will PROBABLY do logins - provided we force it in a few ways 19:06:18 do we have the cgi part too? or just the pam module so far? 19:06:26 hi 19:06:37 we don't have a cgi hooked into fas, no 19:06:46 I have an rpm built for pam_otp, testing proves promising 19:07:24 cool. So, more testing and poking needed, revisit next week? 19:07:41 yes, skvidal pointed out that pam_otp expects a second form of auth, but it works well for sudo with a dummy cgi 19:07:47 nirik: sounds like a plan 19:08:06 One thing we didn't really discuss at fudcon that I'm wondering about: what about our web apps? we would need to make the cgi able or callable somehow to allow them to do 2 factor via it... or they would have to add their own code 19:08:25 so thats something to consider as we move forward. 19:08:31 nod - not sure about that actually. 19:08:40 I've thought about it 19:08:41 it would be nice to do things in a way that would allow them to use the same path for 2fauth 19:08:46 but... let's just say I'm full of waffles on that 19:08:50 I do think that's a separate component, but could use the same cgi maybe... 19:08:52 yeah. 19:08:53 I concur a single path would be nice 19:09:00 yes 19:09:03 if only b/c you have to hit the same location 19:09:06 perhaps we can see what abadger1999 thinks on it... 19:09:10 to store the 'used OTP' 19:09:21 so one way or the other... 19:09:56 yeah. 19:10:00 just something I wanted to note. 19:10:07 anything else on 2fauth? 19:10:33 * herlo thinks that's enough :) 19:10:56 thanks for working on it herlo and skvidal. :) 19:11:03 #topic staging rework 19:11:06 and thanks to npmccallum 19:11:08 and npmcallum 19:11:11 right 19:11:13 :P 19:11:16 oh yeah, def. Sorry. 19:11:43 so, on the staging rework. I am thinking we look at doing that next week once folks are back and recovered from fudcon. 19:11:51 nirik: I'm excited to help with this stuff. 19:12:03 there's some good fun in that I think 19:12:17 we save our current repo off. We save off a staging checkout. We switch staging to use prod repo. we fix what breaks. 19:12:19 and some interesting problems to solve 19:12:28 aye 19:12:30 yeah, it will be fun for sure. 19:13:03 I will also want to rebuild a few stg machines... that are rhel5 still. 19:13:06 do you expect it to be that bad ? 19:13:21 * abadger1999 here now 19:13:31 pingou: hard to say. It shouldn't affect production, but it could break the staging hosts pretty bad 19:13:43 we may need to merge in things that never got merged 19:13:45 hey abadger1999 19:13:59 nirik: I like the idea you suggested of switching the machines in 'staging' to the production environment 19:14:02 and fixing them as we go 19:14:12 that way we'll know we can nuke staging when no boxes live in there naymore 19:14:17 yeah, then we can examine differences on each vs their prod counterpark 19:14:20 counterpart 19:14:33 nod 19:14:44 very nice 19:15:00 one other random thought on this: would this be a time to make a new puppet repo (ie, drop history) and look at possibly auditing it so we could make it public? 19:15:30 nirik: would it be okay if we did the staging drop first 19:15:34 got ourselves on a single branch 19:15:39 and then did the refresh? 19:15:41 yes, I think that would be a pre-req 19:15:44 oh okay 19:15:50 also, we have a bunch of cruft still. 19:15:52 I thought you were saying do both of them at once 19:15:54 things we don't use at all anymore. 19:16:06 well, around this time since we are messing with the repo. 19:16:11 +1 to that plan. 19:16:49 cool. Anything else on staging rework? 19:16:54 nirik: +10 to nuking stuff we don't use 19:17:02 +1 here 19:17:12 abadger1999: did you see the discussion above about 2factor and applications? any ideas there? or should we revisit later? 19:17:32 could it be integrated into fas itself ? 19:17:45 pingou: which? 19:17:54 the 2factor bits 19:17:55 nirik: We shouldn't let work on the web apps stop deploying this for sudo/login 19:18:02 the cgi could become a call to fas no ? 19:18:09 abadger1999: agreed. Just wanted it on the radar. ;) 19:18:44 pingou: not sure. Perhaps someday. ;) 19:18:53 nirik: I think once we get the cgi going, we can write something to do that. 19:19:00 ok. 19:19:24 #topic Any other fudcon todos 19:19:39 Anyone have other items from fudcon that we should put on our radar/discuss? 19:19:39 torrents! 19:19:43 one thing 19:19:53 oh yeah, torrents. ;) 19:19:53 does anyone in here have any strong feelings about the torrents? 19:20:09 is anyone going to raise a ruckus if we were to propose discontinuing that service? 19:20:49 at release time, I'm sure some people will 19:20:50 * nirik doesn't feel strongly for them, but since we do have them working on rhel6 now, my desire to kill it has diminished some. 19:21:53 skvidal: I have an email for the board written up, but was waiting for a new stats page... 19:21:57 skvidal: are we thinking after F17? 19:22:02 or sooner? 19:22:07 I'd be happy to propose it, and see what the board thinks. 19:22:24 nirik: nod - I have the new stats thing on my very short list 19:22:40 nirik: trying to decide if it is worth converting the stats to json or not 19:23:07 right now, I think nothing at all uses the stats other than humans looking at it. 19:23:20 torrent stats on fcomm does 19:23:23 yep 19:23:26 I know b/c that's why the json stats exist 19:23:27 :) 19:23:28 * abadger1999 would go with infra's recomendation on the Board.. stats would be very helpful for swaying the other members I'm sure. 19:23:34 oh, whoops. right. 19:23:40 and spins 19:24:02 spins rank by torrent downloads 19:24:11 that exists now 19:24:14 in ot 19:24:18 ok 19:24:20 top10 19:24:38 * nirik can't recall how it polls that 19:24:49 the json outputs 19:24:55 they get pulled hourly and daily 19:24:58 it uses the hourly for updates 19:25:59 ok 19:26:11 so, lets finish that, then propose to the board and see what happens? 19:26:24 sure 19:26:30 sunds like a plan 19:27:26 heya all 19:27:34 if we don't end up retiring them, hopefully we will at least get a idea what level of usage they would have to be down to to do that. 19:27:42 nod 19:27:42 morning dgilmore 19:27:44 hi dgilmore 19:27:58 nirik: I will see if I can get the json stats regenerating today 19:28:02 shouldn't take too long 19:28:11 if anyone wants to look at the stats themselves 19:28:16 you can login to torrent02 locally and run 19:28:37 wget -O - http://torrent02.fedoraproject.org:6969/stats?mode=everything 19:28:50 cool. 19:29:04 this is the important part 19:29:05 19:29:06 3498 19:29:06 19:29:06 19:29:06 3255 19:29:07 19:29:14 take total peers and subtract total seeds 19:29:18 thats /howmanytorrents? 19:29:19 and you get the number of 'clients' 19:29:24 right 19:29:43 so in this case right now it is 243 19:30:10 and then you can also fetch 19:30:11 http://torrent02.fedoraproject.org:6969/stats?mode=top10 19:30:29 anyway 19:30:41 ok, anything else from fudcon and/or on torrents? 19:30:45 nothing from me 19:30:57 well not on torrents 19:31:03 one thing from fudcon - not yet gotten to but want to 19:31:29 does anyone want to try out the latest glusterfs to see if setfacl/getfacl/chgrp etc 19:31:30 all work like we want them to work? 19:31:47 talking to jdarcy suggests this is all fixed now 19:32:10 I could look at that... 19:32:21 is that latest in fedora? or epel? 19:32:26 he said epel 19:32:32 cool. 19:32:32 so... maybe? 19:32:49 nirik: you could take junk05 - setup some vm's on it and try it maybe? 19:32:55 so, if that was working, what does that give us? 19:32:59 yeah, it's got the space 19:33:09 mirrored fs 19:33:13 that can be mounted from other systems 19:33:31 and written two either of them 19:33:46 * nirik needs to re-look at it. it's been a while and I've seen so many fs/cluster things I can't remember which could do what 19:34:06 * abadger1999 looks at his notes from the fudcon wrap up. 19:34:18 * CodeBlock writes two skvidal :P 19:34:21 nirik: but potentially it could give us a way to replicate fedorapeople, for example, other than drbd 19:34:32 CodeBlock: :) 19:34:39 I was thinking 'two' servers 19:34:41 that would sure help us in both distributing things more and avoiding spof of one site. 19:35:06 nirik: the question is how capable it is over a WAN 19:35:10 which is not obvious to me 19:35:17 yeah. 19:35:30 #info will test some glusterfs out and see how it looks. 19:35:37 the only other thing I can think of being able to do is to have a gluster backend that a series of webservers access for data 19:35:55 but right now we don't have so much web data that we can't just do the replication we do now 19:36:15 yeah. 19:36:23 I was thinking something like 2 VMs in a single gluster share 19:36:25 might speed up the web build tho, which is... slow right now 19:36:27 that then all the proxy servers mount 19:36:42 but read only 19:36:52 yeah, could work. 19:36:56 and then that space could be (maybe) mounted rw from fedorapeople or bastion 19:37:18 oh, one other fudcon thing: how's smolt? can we do anything to help there? or just wait and see... 19:38:07 openshift 19:38:13 latest I heard on that was it was being rewritten to use mongodb and openshift 19:38:13 abadger1999: it's running there now? 19:38:23 skvidal: not yet but that sounded like the plan 19:38:29 yeah, but wanted to know if thats going ok or anything we could do to help. 19:38:34 I guess we need to ask directly. ;) 19:38:50 We'll need to talk to npmccallum 19:38:55 yep. 19:38:58 ok. 19:38:58 abadger1999: excellent 19:39:04 shall we move on? 19:39:08 sure 19:39:17 See if he still has time to work on it now that fudcon is over. 19:39:17 #topic applications 19:39:39 So, I thought I would try and add in a applications section to the meeting... talk about current stuff going on with the apps we maintain. 19:39:53 and see if abadger1999 and lmacken could provide us updates on things. ;) 19:39:59 lmacken: you around? 19:40:00 Cool. 19:40:02 yeah 19:40:05 nirik: great 19:40:17 so, whats the status on the new community/tagger? 19:40:26 as far as fedora packages & tagger goes -- we need to get them properly deployed so we can start breaking things in dev again. 19:40:30 are we going to want to look at replacing the current one soon in prod? or more work needed first? 19:40:35 That leads into the FUDCon decision to try to integrate the web dev and sysadmin sides of infra better :-) 19:40:44 abadger1999: yep. 19:40:49 I would like us to port the statistics widgets before we remove the old community for good 19:40:59 but, it's not a blocker for killing it, imo 19:41:20 lmacken: ok. also, should we see if abadger1999 wants to remove pkgdb stuff in favor of community before we deploy in prod? 19:41:45 we could potentially do some merging beforehand.. but I don't think it's a blocker 19:41:49 note that the current community is rhel5 running only, and I'd love to start moving app servers to rhel6. ;) 19:42:19 ok, so I think I can probably fix the rhel5-only thing pretty quickly, now that fudcon is over 19:42:26 I'd like to figure out what we are going to remove and how we're going to provide/how long until we provide it in the new community. 19:42:38 But we don't necessarily have to remove the duplication beforehand. 19:43:06 yeah, the merging will be a little more long term. Right now, getting a prod instance of tagger/packages is crucial for us to keep iterating quickly on it. 19:43:15 coupld of minor questions 19:43:17 lmacken: in the old community? that might be good if the new one is going to be a while before landing I guess. 19:43:32 we've been holding up just fine with 1 beefy box for packages & tagger. Although, in prod it may be nice to have a seperate machine for the rpm indexer 19:43:32 I don't want us to get stuck maintaing both interfaces because tagger is just missing a few things that pkgdb has, though. 19:43:51 lmacken: right, you want to push out what you have to prod so you can work on the next version(s)? 19:43:53 lmacken: We'd want at least two boxes for failover 19:43:59 nirik: right 19:44:02 if we're going to be making it more central to what we do. 19:44:02 abadger1999: yeah 19:44:24 * nirik thinks 2 boxes for it, and 1 db box (this would tie into our making things more silo like) 19:44:35 nirik: sounds good to me 19:44:37 then we can load balance over those two. 19:45:02 1. which db backend is the new community using? (and tagger)? 19:45:15 2. do we want to maintain consistent access for old community urls? 19:45:30 skvidal: community & packages both do not use a db. tagger is currently using sqlalchemy & postgres on db02.stg 19:45:34 3. is there any merit in trying to have a slave db? 19:45:48 lmacken: thanks 19:46:04 also, we need to determine a url... admin.fedoraproject.org/community, or community.fedoraproject.org or packages.fedoraproject.org or something. 19:46:30 For #2, I'd say no -- I don't think many people were using the old community and the urls IIRC, weren't sufficient to pull up the same page in a new tab... (lmacken Is that right?) 19:46:33 * lmacken likes packages.fp.o 19:46:57 the new one has static-y urls, which is awesome. 19:47:06 +1² 19:47:13 lmacken: ehh... I don't 19:47:13 abadger1999: a bunch of people do use the old community, more than I expected actually 19:47:19 b/c we already have pkgs.fp.o 19:47:27 Which is kind of a bummer 19:47:45 abadger1999: also the bugz.fp.o 19:47:52 that will need some redirects, right? 19:48:01 pkgs.fp.o should really be git.fp.o 19:48:07 since git.fp.o has no web interface 19:48:09 if we want it to go to community instead of pkgdb, it would 19:48:11 lmacken: too confusing with git.fedorahosted.org 19:48:32 lmacken: ah. Were the urls staticy enough that they're valuable to keep working? 19:48:37 nirik: I think we're trying to phase out the 'fedora community' name 19:48:41 since it's already overloaded as it is 19:48:46 yeah, true. 19:48:55 abadger1999: for the old /community urls? nah, I don't think so 19:48:57 skvidal: Yeah -- although we might just point all of bugz.fp.o to the relevant new-community page. 19:49:08 what is the proper name here then? packages and tagger (two related, but seperate apps) 19:49:22 abadger1999: sounds fine to me 19:49:36 lmacken: git.fp.o does have a web interface. What it doesn't have is an index page. 19:49:40 I think we were going to start breaking them out into seperate pages... like "Fedora Packages", "Fedora Statistics", etc 19:49:46 lmacken: sorry 19:49:58 lmacken: d/git.fp.o/pkgs.fp.o/ 19:50:05 abadger1999: yeah, true. of pkgs.fp.o was the index of git.fp.o that would free up that url 19:50:10 s/of/if/ 19:51:04 so, I fear most choices could be confusing. ;) 19:51:11 so it seems like some more thought needs to be put into the naming & url structure of our new apps :) 19:51:23 yeah. we need to come up with something. 19:51:25 * skvidal uses pwgen for a new hostname 19:51:32 Gaemohd5.fedoraproject.org 19:51:37 haha 19:51:38 and we'll change it every week 19:52:01 * skvidal goes to register Gaemohd5.org now 19:52:03 hunt the wumpus. ;) 19:52:15 so our concerns are 19:52:31 1. existing urls 19:52:37 2. simplicity of the new url 19:52:39 the thing that would make the most sense to me: rename pkgs to git, then use packages for this app/interface. But thats likely to be kinda painful. 19:52:41 3. futureproofing 19:52:50 yes 19:52:52 so 19:52:59 since pkgs is 'fedora packages' 19:53:01 nirik: yeah, that's my initial thought as well 19:53:14 and since the interface pkgers use to do their work is fedpkg 19:53:18 nirik: I think mmcgrath had some reasons for not wanting git.fedoraproject.org.. not sure of all of them though. 19:53:20 would fedpkg.fedoraproject.org 19:53:30 be a better idea than 'git.fp.o' 19:53:31 ? 19:54:02 perhaps, confusion with git.fedorahosted; futureproofing for if we change scms again.... not really sure though. 19:54:03 how about this: lets take this discussion to the list and try and come up with some plan by next week? 19:54:14 +1 or sooner 19:54:22 sure, if we can come up with something. 19:54:23 sounds good. 19:54:24 yeah .... +1 19:54:44 lmacken: any other news from packages/tagger and/or bodhi ? 19:55:05 nirik: no bodhi news, I have a bunch of bugfixes & enhancements that I'm working on now 19:55:09 and lmacken's idea that we have the new-community web interface serve as the index to the pkgs.fp.o gitweb has merit. 19:55:16 you're gonna start hammering on bodhi 2.0 right? when should we look at making a 'bodhi01.dev' for early testing? ;) 19:55:46 nirik: yeah, I need to talk with spot & ralph about it, but I think bodhi 2.0 will be getting some cycles in the near future 19:55:53 where is ralph? 19:55:56 i thought he was starting 19:55:59 next week 19:56:03 ah 19:56:03 officially 19:56:09 would someone like to post to the list on this url naming stuff? I could, but I think others might have more of a horse in the race that I? 19:56:14 nirik: yeah, I don't think we need a bodhi dev box just yet 19:56:25 nirik: I can work on it 19:56:32 lmacken: When is the webdev meeting where we'll start talking together for the first time? Wed next week? 19:56:33 the summary email 19:56:36 ok, just wondering. I think it would be good to do that early, so we get feedback and don't surprise people. 19:56:37 and we can argue from there 19:56:47 skvidal: thanks. 19:56:58 #info skvidal to send email about url 's for web apps 19:56:59 abadger1999: it's usually thursdays @ 11, but we're up for changing that to make sure it's at a time where all interested parties can attend 19:57:13 abadger1999: you have any fas or pkgdb news to share? 19:57:20 lmacken: If you can adjust that later by one hour that would work for me. 19:57:28 abadger1999: ok, I'll talk to spot about it 19:57:44 lmacken: cool. I'll try to make it at 8:00 this coming week. 19:58:08 (Thursdays, this is my only meeting but I've been filling my morning increasingly with family obligations) 19:58:15 5/back 19:58:18 sry 19:58:18 * nirik would live to be there especially if it's here. ;) 19:58:36 fas, I'm slipping the release schedule 1 week -- I'll cut a beta on Monday. 19:58:44 We just didn't get around to it during fudcon. 19:59:02 python-fedora will be the next thing I cut a new release of after that 19:59:29 ok, so final looking at 2012-02-07 for fas? 19:59:31 lmacken: If you or threebean can put some cycles into python-fedora, the issues that I'm resolving are in the tg2 identity stuff. 19:59:36 nirik: yeah. 19:59:59 * pingou wouldn't mind to be around 20:00:01 lmacken: So it's kinda up your eventual alley (I'm assuming that the new f-comm and tagger are tg2 based?_ 20:00:15 abadger1999: yeah, we use the faswho middleware in tagger, w/o problems 20:00:19 cool. Any new pkgdb on the horizon? or waiting for things to shake out with packages/tagger before doing anything? 20:00:23 lmacken: ahh 20:00:28 lmacken: but not with BaseClient :-) 20:00:40 lmacken: that's where the issues I'm resolving are. 20:00:54 abadger1999: ah, interesting. point me at some tickets and I'll have a look. 20:01:17 lmacken: Cool. I'll catch you guys up on the remaining issues today or tomorrow. 20:01:31 nirik: pkgdb will be next after python-fedora. 20:01:40 ok, cool. 20:01:47 nirik: No release dates yet but the plan is: 20:02:24 One 0.5.x release which catches us up on hotfixes and some of the API stable changes from fchiulli and others. 20:02:43 Then I'll be merging with the devel branch and we'll make an API changing release. 20:02:54 sounds good. 20:03:05 In the future, I think we're going to be moving away from strict API stability 20:03:23 (or increase the releases ? :)) 20:03:24 Instead, we'll document when API changes and porting tips. 20:03:29 pingou: :-) 20:03:31 sounds reasonable 20:03:39 I guess we could discuss https://fedorahosted.org/fedora-infrastructure/ticket/3094 quickly now? 20:03:42 it's tagged meeting. 20:03:47 pingou: I'm hoping that not having to keep API stability will increase the number of releases we can make. 20:04:08 abadger1999: finger crossed ;-) 20:04:34 dgilmore: you have any thoughts on the above ticket? 20:05:30 nirik: not really 20:05:34 ok. 20:05:57 nirik: id be ok with just removing acls from eol release branches 20:06:36 * nirik is fine with that too. 20:07:00 abadger1999: you want me to take it to fesco? 20:07:07 Sounds good to me. So whatever gitolite allows (open or closed) would be our default. 20:07:12 nirik: Sure. 20:07:14 I think default is closed 20:07:31 wfm 20:07:48 * nirik sees we are running long. 20:07:56 #topic Upcoming tasks 20:08:04 #info 2012-01-19 - infrasturcture meeting. 20:08:05 #info 2012-01-24 - reinstall ibiblio01 (tenative) 20:08:05 #info 2012-01-25 - send meeting agenda 20:08:05 #info 2012-01-26 - infrasturcture meeting. 20:08:05 #info 2012-01-31 - fas 0.8.11 final release. 20:08:07 #info 2012-02-01 - nag fi-apprentices. 20:08:09 #info 2012-02-01 - 2012-02-03 dgillmore is at phx2 20:08:11 #info 2012-02-10 - drop inactive fi-apprentices 20:08:15 #info 2012-02-14 to 2012-02-28 - F17 Alpha Freeze 20:08:33 anything else upcoming folks want to schedule or talk about? 20:08:45 I'd like to note that we are down to 20 rhel5 instances. 20:08:54 I'm going to keep working to make that 0 20:08:54 * skvidal was summarizing the apps/urls discussion and I wanted to ask something when we have a chance 20:09:06 #topic Open Floor 20:09:09 skvidal: go for it 20:09:18 so I was re-reading it 20:09:29 lmacken: you said you wanted to get away from one overall 'app' 20:09:32 that is everything 20:09:51 would it make any sense for us to have apps.fedoraproject.org/substuff AND have a top level name for it? 20:10:05 or does that just overly complicate our llayout? 20:10:35 * skvidal listens to the crickets 20:10:36 I think users might find that confusing to find... 20:10:37 skvidal: that could work 20:10:46 but then again I always use history... 20:10:51 and I bet many others do too 20:10:58 I'll just send the summary email and see what replies we get 20:11:10 well, as far as finding apps, ralph and I had the idea of a peice of WSGI middleware that added a google-like bar at the top of the app that made it trivial to navigate around 20:12:06 that might be nice. 20:13:03 anything else for open floor? or shall we call it a meeting? 20:13:32 I'm sure there's other stuff, but there's always next week :-) 20:13:37 if anyone is looking for things to work on 20:13:39 come see me 20:13:48 and I'll see if I can dish stuff their way 20:14:09 skvidal: if any can be easyfix we could use some more for apprentices. 20:14:21 none of them are EASYFIX 20:14:30 ok. 20:14:30 but some of them might be INTERESTINGFIX 20:15:07 ok, thanks for coming everyone. ;) 20:15:14 #endmeeting