20:00:00 #startmeeting Infrastructure (2012-03-08) 20:00:00 Meeting started Thu Mar 8 20:00:00 2012 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:00:00 Useful Commands: #action #agreed #halp #info #idea #link #topic. 20:00:01 #meetingname infrastructure 20:00:01 #topic Robot Roll Call 20:00:01 #chair smooge skvidal Codeblock ricky nirik abadger1999 lmacken dgilmore mdomsch 20:00:01 The meeting name has been set to 'infrastructure' 20:00:01 Current chairs: Codeblock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge 20:00:28 * CodeBlock is here 20:00:32 * skvidal is here 20:00:45 * pingou is here 20:00:46 is here 20:00:53 who all is around for a lovely, exciting, thrilling, action packed, adventure infrastructure meeting? 20:01:09 Rainy, tired-ey, last-day-of-school-before-spring-break kind of day. 20:01:46 the last one sounds cool :) 20:02:10 :) 20:02:11 :) ok, lets go ahead and get started then... 20:02:14 #topic New folks introductions and Apprentice tasks. 20:02:20 If any new folks want to give a quick one line bio or any apprentices 20:02:20 would like to ask general questions, they can do so here. 20:03:01 anyone? ;) 20:03:14 we did have a number of new folks on the list... but the meeting time may not be great for them. 20:03:24 * wsterling here 20:03:54 #info new folks, hang out in #fedora-admin and say hi to get started. 20:03:58 #topic two factor auth status 20:04:05 skvidal: any news on this in the last bit? 20:04:19 yah 20:04:28 so the cgi is posted at github 20:04:36 and I've sent pam_url off to some folks to audit 20:04:42 ricky got back to me and was not happy about pam_url 20:04:48 for a couple of good reasons 20:04:55 one of which looks pretty easy to fix 20:05:00 the others are going to be annoying 20:05:06 bummer. ;( 20:05:15 so I'm waiting to hear back from bressers 20:05:16 would one of the other implementations be better to go with then? 20:05:25 they are all going to require some hacking now 20:05:40 yeah, I wonder if we couldn't ask someone who does security code to pick one and hack on it for us. ;) 20:05:41 buenos 20:05:42 to get them to work like we want 20:05:51 nirik: I doubt it... 20:06:14 possibly not, but we could ask around if we need to. 20:06:20 nirik: but I would love it. C is not my native language (so to speak) and it would be easier to fix pam_url if I was more capable at C 20:06:25 nirik: got any ideas? 20:06:28 on whom to ask? 20:06:40 what was that think about pam_python someone mentioned earlier... 20:06:44 not sure, I could ask bressers who to ask. ;) 20:07:01 but let me ask around, see if I can find anyone. 20:07:02 smooge: 1. it's not in any distribution of anything we have 20:07:16 oh got it 20:07:16 2. pam_python makes a big point on its site of saying how much slower it is to exec python 20:07:29 smooge: believe me, I looked 20:07:36 pam_pypy? 20:07:39 #action nirik to look around for C coders to work on pam_url for us. 20:07:53 I think that would have made a great GSOC 20:07:58 so, once we have pam_url working we can setup a test instance and then hopefully deploy? 20:08:01 smooge: umm I doubt it 20:08:13 nirik: I think so - we can definitely get it running and tested 20:08:17 if we had people who understand C etc etc. 20:08:45 nirik: I'm even happy with someone who commonly codes in C who can answer some questions for me 20:08:47 w/o mocking me for it 20:08:55 * skvidal is not joking about that last part 20:09:20 ok, I'll see what/who I can find. 20:09:25 anything else on this? 20:09:47 nothing so far 20:09:51 #topic Staging re-work status 20:10:05 ok, I meant to work on this some this week... but kept getting sidetracked. ;) 20:10:15 I'll try again soon... 20:10:27 #topic Applications status / discussion 20:10:43 abadger1999 and threebean and lmacken are all out at fudcon... 20:10:44 I think everyone working on apps is at pycon 20:10:47 pycon 20:10:50 not fudcon 20:10:54 yeah, sorry, pycon. 20:11:05 anyhow, anyone else have any news on any apps? 20:11:10 * pingou updates fedora-gather-easyfix's spec file 20:11:14 CodeBlock: how's the search crawler going? 20:11:16 * abadger1999 wishes those were synonymous... 200+ people at fudcon would be awesome 20:11:21 2000+ 20:11:48 well I made some tweaks to the crawler config... some helped, some didn't... still need to tweak more...also need to try to make the web UI faster 20:11:55 pingou: is the thought to deploy that to bapp01 and rsync to proxies like we do for websites? or run it on apps? or ? 20:11:55 it gets slow with a lot of results sometimes 20:12:28 CodeBlock: daMaestro said he could help out, he's run dpsearch before, so ping him if you like. 20:12:41 oh cool, will do 20:12:45 nirik: the version packaged at the moment is a simple html, rsync is easy, but with the integration with the bz, there is css and js file in addition 20:13:02 (but it would still run as a cron) 20:13:28 we should see what's the best way 20:13:43 ok. 20:14:04 any other applications news? 20:14:05 pingou: would you like to ask for package review on the packages you have that need to get into epel for us to deploy? (me is busy at pycon) could be an easyfix if people are already packagers 20:14:16 oh, I have one note: 20:14:25 abadger1999: done and done :) 20:14:33 Cool. 20:14:39 we now have a way to send httpd error_logs to log02... which could help us when looking for application problems. 20:14:49 abadger1999: I meant the two reviews , fedora-gather-easyfix and python-tgcaptcha2 20:14:53 however. Some apps are so noisy that it's useless. we need to fix that first 20:15:47 nirik: you're talking about the noise from fas that looked like info-logs? 20:15:58 I tried looking at how fas logs for example, but got confused. ;) So, we may want to look at having a VFAD on application logging someday... 20:16:03 yes. 20:16:03 ah, yues 20:16:03 :) 20:16:05 sorry 20:16:34 if we can get apps to log just tracebacks to error_log (or actual, you know, errors), we can send them to log02 and it will be much easier to see whats happening. 20:16:55 skvidal, I can ask around with my former group.. they did various pam stuff for Science! 20:17:00 nirik: I have a dumb question on some of the error logs 20:17:04 I can fix this 20:17:05 File does not exist: /srv/web/favicon.ico 20:17:08 obviously 20:17:21 does anyone care if I do? 20:17:29 +1 20:17:36 yes, please fix. 20:17:42 ok 20:17:44 thats one thing I think logs are good helping us do. 20:18:07 nirik: Fixing logging (to the point you're talking) may be impossible.... I know we've tried for years. 20:18:16 nirik: But we might be able to cut it down some. 20:18:18 abadger1999: ;( thats sad. 20:18:19 done 20:18:32 well should we just put one in there? 20:18:34 abadger1999: what is going on where it is outputting so much to the logs? 20:18:35 I know skvidal was successful on cutting some down for other apps 20:18:46 abadger1999: that's b/c I cut them down with a butcher knife :) 20:18:49 if we can't fix it at that level, we may be able to filter them in rsyslog I guess, but man that seems a kludge. ;) 20:18:59 it's not as graceful as a solution as I'd expect you to come up with ;) 20:19:18 abadger1999: what's creating the logs? 20:19:34 We don't know what's going on -- something about logging is mising tracebacks when we mess with logging in certain ways. 20:19:38 skvidal: pyhton's logging module 20:19:57 it's not just fas right? other things have the same issue? 20:19:59 oh dear god 20:20:05 * skvidal goes to hide his head 20:20:05 I'm happy for you to cut through the logs with a butcher knife. 20:20:18 logging makes me want to cry and hurt myself. 20:20:22 yeah. 20:20:33 #info we should really try and fix logging of applications so we can send them easily to a log host. 20:20:44 I was hoping that python-3 was going to get a different logging module but that's another story. 20:21:04 yeah, I doubt it would be at all easy, but perhaps if we got everyone together and hacked on it we could figure something out. 20:21:31 anyhow, just wanted to mention it. 20:21:39 any other application news? 20:21:57 do app *servers* count? 20:22:01 Most are built as el6 now 20:22:11 not this week. Hopefully we'll have some tangential news next week. 20:22:14 sure! thanks very much for working on that CodeBlock. 20:22:19 np 20:22:25 abadger1999: is there any way to cut down the volume? 20:22:29 abadger1999: or do we lose everything? 20:22:42 you know what... nevermind 20:22:48 I think nirik is right 20:22:48 we have just 3 rhel5 boxes left I think now... app07 (which is only serving old community), bapp01 and xen04 (the xen host bapp01 is on) 20:22:50 for post pycon 20:22:55 let's etup a VFAD 20:23:02 so we can spend a day or two just unfucking this 20:23:03 skvidal: we can cut down the volume... but we tend to lose some tracebacks -- and we aren't sure how to configure logging so that doesn't happen. 20:23:07 either via rsyslog fitlers 20:23:15 20:23:15 or by fixing the app 20:23:20 whatever is more possible 20:23:24 Cool. 20:23:29 * abadger1999 heads out of room to lunch 20:23:35 have fun abadger1999 20:23:41 #topic Upcoming Tasks/Items 20:23:46 #info 2012-03-07 to 2012-03-14 - Pycon 20:23:47 #info 2012-03-10 - drop inactive fi-apprentices 20:23:47 #info 2012-03-20 to 2012-04-03 - F17 Beta Freeze 20:23:47 #info 2012-03-27 - drop inactive maintainers from packages. 20:23:47 #info 2012-04-01 - nag fi-apprentices. 20:23:47 #info 2012-04-03 - F17Beta release day 20:23:49 #info 2011-04-03 - gitweb-cache removal day. 20:23:51 #info 2012-04-10 - drop inactive fi-apprentices 20:23:53 #info 2012-04-24 to 2012-05-08 - F17 Final Freeze. 20:23:55 #info 2012-05-01 - nag fi-apprentices. 20:23:57 #info 2012-05-08 - F17 release 20:23:59 That's what I have upcoming until f17 release day. ;) 20:24:14 I'd like to get bapp01 replaced before beta freeze if we can. 20:24:29 anyone have any other items they want to schedule or note? 20:24:58 mailman password cleanup 20:24:58 * nirik listens to the crickets. 20:25:11 smooge: ah yeah, are you gonna get that finished today? tomorrow? 20:25:27 tomorrow. 20:25:33 it should be a short set of things: 20:25:44 smooge: if oyu need help with spamming users yell at me 20:25:47 1) Email a list of people using seths spam-o-matic script 20:25:50 I have that spammer script which makes hatemail easy 20:25:52 oh okay :) 20:25:54 you know :) 20:26:09 2) Force mailman to do my will. 20:26:52 cool. Sounds good. 20:27:13 ok, anything else for upcoming? 20:27:22 not from me. 20:27:44 ok, on to this fun topic: 20:27:48 #topic Ssh keypair changes still needed. 20:27:49 nirik: what special magic needs to happen to rebuild bapp? 20:27:54 bah 20:27:55 #undo 20:27:55 Removing item from minutes: 20:28:10 CodeBlock: well, bapp01 has a bunch of one-off scripts that only run there. 20:28:27 so, we could do a migration instead of a replacement, but I think we can probibly replace it ok. 20:28:47 I'd like to wait until toshio is back from pycon to help putting out fires on it. 20:28:59 then we just need to schedule a day and replace it and fix anything broken 20:29:07 sounds good 20:29:10 * mdomsch votes for "not next week" 20:29:45 mdomsch: ok, when are you gonna be around? would be good to have you around too in case. ;) 20:29:53 the beta freeze is the 20th. ;( 20:30:27 I guess we could do the 19th... 20:30:37 and back it out if it became a pain 20:30:54 could we do a bapp02 built with el6 and see about putting it in replacement? 20:31:05 nirik: I'm offline tomorrow 3pm, back Mon 3/19 20:31:34 smooge: well we could, but it's going to be a lot more work... having to change what things point to and sync and such. I guess it might be worth it. 20:31:41 not that it matters im taking tomorrow off 20:31:53 nothing of MM points at bapp01 20:32:03 mdomsch: ok 20:32:08 dgilmore: cool. :) 20:32:21 bapp01 is the source, it pushes to app* from there 20:32:27 well I would say it would be a good way to see "what files are on bappX and what ones showed up on bappY and why are they different." 20:32:27 dgilmore: enjoy 20:32:38 so MM would be fine with a bapp02, get it working, then flip over 20:32:43 mdomsch: cool. 20:32:46 dgilmore: before you go - I was meaning to ask - do you have any thoughts on the buildvm boxes? 20:32:55 dgilmore: are they good enough for us to press forward with more of them? 20:33:09 ok, I can look again at how much tweaking a bapp02 would be. 20:33:15 #action nirik to look into possibly doing a bapp02 (rhel6) for migration. 20:33:27 nirik: more than happy to help with that too 20:34:00 one other issue is that we would need to make sure to shut off things on bapp01 as we move them... 2 mirrormanagers both pushing to apps might be very bad. 20:34:22 nirik: right 20:34:36 skvidal: quick glance they seem ok. but i want to lookat setting up a heavybuilder channel first, and see how that plays into things, especially with the kernel teasm request yesterday 20:34:57 dgilmore: what do they want? 20:35:41 skvidal: they are signing all modules at build time. need to look at running rngd or see how else we can make sure there is plenty of entropy for them to use 20:36:05 skvidal: part of that is making a new gpg key at build time 20:36:17 rngd is very easy to add. ;) I wrote up the puppet stuff for it, just didn't commit it. 20:37:15 ok, we can continue investigating the bapp01 thing... 20:37:40 skvidal: what was the next steps on those? we may want to hold off until we know about what hardware we have in the bladecenter. 20:37:58 not really 20:38:12 the x86-## are not going anywhere 20:38:17 so what I wanted to do was 20:38:18 true enough 20:38:25 lemme find the plan 20:38:26 one sec 20:39:21 take 2 more of the blades 20:39:26 turn them into 4 builders per blade 20:39:37 ok, the bc02 ones? 20:39:40 yes 20:39:42 on bc02 20:39:47 and then 2 more blades 20:39:50 1 of them one big VM 20:39:53 and the other 2 VM 20:40:00 cool. well, whenever dgilmore is ok with you moving ahead... 20:40:12 ideally 20:40:19 I've seen no issues with the current buildvms... 20:40:21 if we can move from the builders we have now over to VMs 20:40:31 then we can dismantle the x86## builders 20:40:46 and grow from there... depending on what we want to do. 20:40:53 yep. 20:41:03 lets revisit next week then? dgilmore ? 20:41:09 sounds round 20:41:32 #info will revisit next week on more migrations of builders to vm's 20:41:36 #topic Ssh keypair changes still needed. 20:41:41 so, some background: 20:42:01 we asked in october of last year for everyone to change their password and generate a new ssh keypair. 20:42:15 we marked inactive those people who didn't do that in january. 20:42:30 then we checked and a number of them had re-uploaded the same ssh key. 20:42:40 so, we marked them inactive again and asked them to please change. 20:43:14 as of sunday we had several who reactivated their account and didn't still change their key. 20:43:25 I don't want to keep playing wack-a-mole here. 20:43:52 I'd like to propose we mark these people admin_locked and ask them to come to us to unlock their account... 20:43:53 nirik: sure 20:44:11 can anyone think of a better way to deal with them? or counterproposals? 20:44:34 nirik: +1 lock them 20:44:47 ^ 20:44:56 nirik: so dumb question 20:44:56 Agreed 20:44:58 who is it left? 20:45:17 there were 9 as of sunday. 20:45:23 nirik: might be worth getting their contact info out of fas and chasing them down on the phone, if possible 20:45:33 people who re-activated since the last time we inactvated them, but didn't bother to change their key 20:46:20 well lets just clear their key 20:46:23 I suppose, but I'm not sure I want to have a long phone conversation about 'you must do this'... but... 20:46:39 smooge: then they just re-upload it again. ;( 20:46:40 nirik: I enjoy being intimidating to folks on the phone :) 20:46:42 nirik: kidding 20:46:47 ok we put in my key 20:46:52 skvidal: if you want to do it, thats just fine with me. ;) 20:47:02 nirik: gimme the list 20:47:02 we should look for time-zone, making sure there is someone they can talk to if they choose irc 20:47:22 skvidal: should be in email to sysadmin-main from sunday. 20:47:44 or we can run it again... it's in cron on lockbox01 20:47:48 nod 20:47:51 I'll just run it 20:47:55 cool. 20:48:14 #action skvidal to bug people in this state. Any we can't reach will get another email, then be admin_locked. 20:48:28 ok, anything else on this? 20:48:44 #topic Tickets from Ages past 20:49:03 ok, I thought I would have some fun and try and clean up old tickets... 20:49:21 and just drop 1 or 2 a meeting so we can see if we can close them or do something with them 20:49:39 'fun' 20:49:40 we currently have 171 tickets, some of which are...very old. 20:49:49 got a link? 20:50:16 https://fedorahosted.org/fedora-infrastructure/report/1?sort=created&asc=1 20:50:28 this is the list sorted in created order. 20:50:38 so, oldest ticket is: 20:50:44 .ticket 116 20:50:46 nirik: #116 (Fedora Poll) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/116 20:51:10 This was orig going to be a stand along poll for the front page, then a wordpress plugin, then... I don't know what. 20:51:49 There are several wp poll plugins 20:51:52 we have a similar ticket for a survey app (althought thats slightly different) 20:52:00 my ancient ones get a little better with the next MM push, if/when i ever get around to it 20:52:06 adrianhannah: yeah. Sadly, we are no longer running wordpress. ;) 20:52:26 mirror push waits on the FI message bus 20:52:28 Aha, that makes sense :p 20:52:35 hmm 20:52:39 so, do we still want this? close it? rescope to something we could actually do? 20:52:50 this looks like threebean https://fedorahosted.org/fedora-infrastructure/ticket/151 20:53:06 https://fedorahosted.org/fedora-infrastructure/ticket/506 <- just close this 20:53:47 skvidal: I kinda thought 506 was nice to do... not that anyone has any time to work on that kind of thing. ;( 20:53:51 right 20:53:56 I have a calendering one... 20:53:58 and it will almost immediately be out of date 20:54:31 nirik: so you'll be writing up a doc that will instantly bitrot, yay 20:54:39 s/you/someone/ 20:54:51 what about a wiki page ? 20:54:51 skvidal: perhaps it could be autogenerated somehow. 20:55:01 nirik: from what? the pkgs we have installed? 20:55:18 nirik: I can dump out a list of pkg names on every system b/c.... that will be unhelpful 20:55:53 yeah, not sure. ;( 20:57:09 * nirik reassignes 151 to threebean and updates it. 20:57:32 #605 should be duable 20:57:40 side note on the poll... I was pointed at http://www.primelife.eu/results/opensource/63-dudle 20:57:53 check if the user is in more than one group, if so show the options 20:58:05 pingou: cool. Would that be an easyfix type thing? or ? 20:58:31 nirik: slightly above imo, it includes a db scheme change 20:58:48 nirik: I had a web poll python based some time ago 20:58:52 pingou: could you update that ticket with info? 20:59:14 ok, we are running low on time... 20:59:47 feel free to frob the older tickets as you all see fit... if you can get something moving or it looks like an easyfix, update it. 20:59:50 #topic Open Floor 20:59:54 anything for open floor? 21:00:01 I am going to bring this up on list 21:00:05 but I thought I'd float it out here 21:00:22 I'd like input on what people use/need/want from our config mgmt system 21:00:33 I've been playing with and adding some patches to another tool recently 21:00:38 and it has been making me think 21:00:48 about what we DO use versus what we need to use 21:01:02 and how complicated those can be to disambiguate 21:01:16 so I'd love to hear back from folks about what it is they want from config mgmt 21:01:24 what features are necessary 21:01:30 this is not about any specific tool 21:01:44 cool. 21:01:54 I don't care about cfengine vs puppet vs bcfg2 vs salt - I care about what we're USING from those tools 21:02:25 it might be worth mining our git puppet repo and seeing how many of each kind of statement there are? not sure thats easily greppable tho... 21:02:50 nirik: well I think it is reasonable to generate a list of requirements of basic requirements 21:02:56 worth thinking about. yep. 21:03:01 my short list is this 21:03:06 pkg install/remove 21:03:11 file replacement 21:03:18 command execution 21:03:37 service 'chkconfig' 21:03:47 groups of things? (ie, do all this on 'appservers') ? 21:03:47 passwd setting/confirmation 21:03:48 process status checks 21:03:50 stg vs prod ? 21:04:05 pingou: I consider stg vs prod to be an ANTI-feature, personally. 21:04:25 nirik: so is grouping something that's in the config mgmt? 21:04:31 conditionals? if rhel5 {foo}, elseif AIX {doom} 21:04:39 nirik: or is grouping something that's in the tool that pushes the config mgmt? 21:04:57 nirik: so cases/conditions - what about templating? 21:05:25 not sure. We seem to not like templating all that much, but I guess it's a feature. ;) 21:05:56 nirik: conditionalized actions? 21:06:06 ie: if file is changed then run this command? 21:06:10 yeah, and related... variables? 21:06:23 if bacula5 is true, do this 21:06:38 variables inherited from the system AND applied from the mgmt side 21:07:07 does config mgmt implicitly need to include some sort of inventory system or does it need to simply have access to one? 21:07:23 for example - if we moved to a different tool for the config mgmt but kept using 'facter' - just as an example 21:07:33 I don't know that it matters... either way. 21:07:34 facter is the inventory system (to a limited extent) 21:07:37 ok 21:08:05 one approach to this might be to look at our more complex machines... and ignore puppet and say 'what all needs to be done here to make this from basemachine' 21:08:13 nirik: +1 21:08:18 I like that idea very much 21:08:30 this came up as part of a couple of things I was working on this week 21:08:32 then work back from that to see what would need to be known... 21:08:51 specifically trying to figure out all the places we would need to modify to make all of our http errorlogs go to log02 21:08:56 and to put it unkindly 21:09:00 it's a fracking mess 21:09:11 yeah 21:09:21 so I started looking at the yaml catalogs in puppet for each node 21:10:03 and it's kinda ugly - but they do make it easier to see what _should_ be happening 21:10:25 yeah. 21:10:30 the thing I've been playing with is called ansible (github.com/ansible/ 21:10:36 and let's ignore what it's doing 21:10:40 b/c that's immaterial 21:10:44 but there is one bit in there I like 21:10:47 the playbooks. 21:10:59 the nice thing about ansible (or something like it) is that you could write scripts in any lang right? whatever you like best? 21:11:00 essentially a description of what to do on the machine to make it 'right'. 21:11:12 nirik: yah - you write modules in whatever language 21:11:17 as long as they can execute on the remote node 21:11:20 that's all that matters 21:11:31 it pushes them all over to the remote system (using ssh/sftp) and runs them 21:11:46 and reports back results to the person calling them 21:11:51 * nirik nods. 21:11:55 but that's all the 'how they do it' bits 21:12:02 I was comparing a puppet recipe 21:12:04 to a playbook 21:12:15 I have to head out to go to a school meeting. 21:12:20 puppet is describing the state 21:12:26 ! 21:12:27 whereas a playbook is describing what to do 21:12:34 mdomsch: ? 21:12:46 typo 21:12:49 heh 21:12:59 I guess there's kinda two parts here... 'is this machine the way it should be now?' if not 'how do I make it be that way' 21:13:13 unless you want to just blindly replace files every run or the like. 21:13:21