20:13:08 <smooge> #startmeeting Infrastructure
20:13:08 <zodbot> Meeting started Thu Dec  9 20:13:08 2010 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:13:08 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:13:15 <smooge> meetingname infrastructure
20:13:22 <smooge> #meetingname infrastructure
20:13:22 <zodbot> The meeting name has been set to 'infrastructure'
20:13:57 * nirik has an update on tickets
20:14:19 <smooge> #topic Robot Roll Call
20:14:22 <smooge> cambot is here
20:14:23 * skvidal is here
20:14:23 * goozbach here
20:14:29 * tremble here
20:14:31 * ninjazjb here
20:14:46 * mmcgrath here
20:14:55 * casep here
20:15:09 <smooge> #topic Meeting Agenda
20:15:11 * gholms waves at the new victims^Wvolunteers.  Thanks for joining up!
20:15:37 <smooge> hi guys. I am running a bit late [I thought it was Wednesday again until Goozbach reminded me.]
20:15:41 <goozbach> https://fedorahosted.org/fedora-infrastructure/report/10
20:15:52 <goozbach> someone made the report for meetings
20:15:55 * ke4qqq here
20:16:04 * sgallagh is here
20:16:04 <goozbach> sadly that was my action item, and I didn't do it
20:16:16 <goozbach> just got sponsored tues :)
20:16:26 <smooge> goozbach, no problem. I thought that was Monday
20:16:35 <smooge> man I need to quit working weekends :).
20:16:38 <goozbach> smooge: you're behind a day
20:16:43 <smooge> then I can remember when Monday starts
20:16:44 * nirik worked this last week on cleaning up bugs.
20:16:58 <smooge> I see 5 items on the agenda.
20:17:08 <smooge> #topic Trac tickets... nirik
20:17:18 <goozbach> we've got one item which isn't on the report
20:17:26 <goozbach> (our meeting running long)
20:17:50 <smooge> nirik you had an update on bugs and tickets?
20:17:56 <nirik> yeah...
20:18:06 <nirik> so, we had 323 bugs the other week.
20:18:13 <nirik> we are now down to 271
20:18:23 <nirik> lots of junk that could be closed/didn't matter anymore, etc.
20:18:30 <nirik> and some things got assigned to folks and done.
20:18:39 <smooge> cool
20:18:42 <nirik> I created a number of more components
20:18:42 <smooge> I like that
20:18:48 <smooge> even more cool
20:18:56 <nirik> The two things I see left are:
20:18:57 <smooge> I need to check where my list of stuff should go
20:19:18 <nirik> 1) the General component still has a number of things in it... and it's the default, and it gets no one cc'ed on it.
20:19:31 <nirik> so, when someone files a new general ticket, it could easily be lost in limbo
20:19:56 <nirik> we could cc those to the infrastructre list. or we could cc them to 'sysadmin' or just try and watch trac.
20:20:23 <goozbach> I'd suggest sysadmin at the least
20:20:25 <smooge> I do not mind them going to sysadmin or infrastrucutre.
20:20:48 <goozbach> maybe a bit too much traffic to go to infrastructure
20:20:53 * nirik doesn't much care either.
20:21:01 <smooge> ok lets cc them to sysadmin
20:21:06 <goozbach> if folks are just lurking on the list they might not like it
20:21:19 <nirik> there's a ton of people in sysadmin. ;)
20:21:30 <smooge> ah yes... another issue
20:21:33 <nirik> but they already get nagios, so...
20:21:48 <smooge> I thought only sysadmin-noc got those
20:21:52 <goozbach> that's my take
20:21:54 <ke4qqq> but wait - isn't there a problem with the default choice being general in the first place?
20:21:58 <goozbach> I'm getting nagios
20:22:08 <goozbach> and I doubt I'm in -noc
20:22:13 <smooge> ah ok
20:22:25 <ke4qqq> instead of one of the components
20:22:46 <nirik> ke4qqq: well, there's lots of things infrastructure does.
20:22:52 <nirik> what would be the default component?
20:23:03 <smooge> ke4qqq, I am not sure it is a problem. general is the lowest common denominator for components..
20:23:16 <smooge> everything else is more specialized requests.
20:23:17 <ke4qqq> perhaps none - force a selection
20:23:26 <nirik> nagios and puppet commits go to sysadmin-members.
20:23:45 <sgallagh> ke4qqq: You can't force a selection on that widget. It has a default and a dropdown.
20:23:49 <nirik> ke4qqq: I don't think trac will let you do that.
20:24:01 <smooge> oh I got a mistake .. I was thinking of Type versus Component
20:24:11 <sgallagh> ke4qqq: The best you can do is set a default of "--CHANGE ME--" and watch as every ticket comes in as "--CHANGE ME--"
20:24:18 <ke4qqq> ah - ok
20:24:30 <smooge> hehehehe the old Red Hat Support System
20:24:48 <smooge> our largest architecture and problem group: Change Me
20:24:56 <nirik> I can cc them to sysadmin and see who complains. ;)
20:25:16 * nirik tries to recall what his point 2) was. ;)
20:25:42 <phuzion> nagios notifications go to sysadmin-members
20:26:30 <smooge> ok in that case.. sysadmin it is
20:26:50 <smooge> I think something ke4qqq brought up in passing last time would come up now
20:26:58 <smooge> #topic sysadmin groups
20:27:17 <smooge> Currently the sysadmin group has 100+ members in it.
20:28:15 <phuzion> Time to purge some members, you think?
20:29:13 <smooge> There are many reasons people are in the group (used to be interested, thought that being a sysadmin of their machine needed to be in syadmin group, etc).
20:29:44 <smooge> I am not sure 'purge' but at least a cleaning
20:29:53 <ninjazjb> Can you apply the 90 policy to this?
20:30:06 <sgallagh> smooge: Perhaps just send an email to all members of the sysadmin group requesting an acknowledgement of whether they want to remain
20:30:25 <sgallagh> smooge: Give people two weeks or so to respond, then cleanse?
20:30:46 <sgallagh> People can always re-request admission
20:31:01 <goozbach> how do we fast track the re-admission?
20:31:17 <phuzion> I'd say the quickest way to do it would be find all members that haven't accessed FAS in 6+ months, email them, give them 2 weeks to respond, and then purge whoever hasn't responded.
20:31:31 <ninjazjb> sgallagh: Isn't there a 90 policy where if a sysadmin doesn't access FAS in 90 days gets removed?
20:31:49 <nirik> ninjazjb: no
20:31:54 <ke4qqq> ninjazjb: policy exists - don't think it's acted on though
20:32:10 * nirik has never heard of that policy.
20:32:11 <phuzion> sgallagh: I'm not a fan of the "Respond to this email or get cleansed from the group" idea, tbh.
20:32:25 <rfelsburg> I know I'm still new to this, however i'm not sure readmission would need fast tracked. 2 weeks seems more than enough time for a response. If you miss that window, maybe waiting for readmission is a nice reminder to respond more quickly.
20:33:02 <ke4qqq> nirik: http://fedoraproject.org/wiki/Infrastructure/GettingSponsored#Grounds_for_removal
20:33:19 <ke4qqq> but it's 6 months
20:33:21 <ke4qqq> not 90 days
20:33:26 <nirik> ok.
20:33:30 * ninjazjb close :)
20:33:32 <nirik> so, just enforce that? ;)
20:33:34 <ninjazjb> but not really
20:34:29 <phuzion> So, is there a way to check and see if someone used FAS-provided credentials in a certain amount of time?
20:35:05 <CodeBlock> hi there
20:35:10 * CodeBlock is here
20:36:00 * ke4qqq wonders if we could just script members of sysadmin and check last time they logged into bastion
20:36:06 <smooge> yes.. there are ways to make a list of FAS users. Not sure I can say 'they used them as a sysadmin versus a packager' or some such but it would give a list.
20:36:27 <ke4qqq> and produce a report that shows people who haven't logged in in xx months
20:36:32 <ke4qqq> that mgiht not work for -test though
20:37:22 <nirik> ke4qqq: yeah, perhaps a cull over logs.
20:37:35 <nirik> you have not logged in in 6 months to any fedora server.
20:38:21 <phuzion> I'd say use that list to email people, asking them whether they are wishing to remain in FI or not, rather than just saying "you're gone"
20:38:22 <sgallagh> ke4qqq: I almost never log into bastion, but I'm constantly on hosted1/2 and the publictest machines.
20:38:49 <ke4qqq> sgallagh: how do you get to hosted1 w/o going through bastion - sure you don't have bastion set up in your ssh_config?
20:38:49 <sgallagh> ke4qqq: I'm not sure I've logged into bastion in the last 6 months
20:39:38 <smooge> hosted01/2 are outside the PHX2 firewalls so allow for direct logisn
20:39:42 <ke4qqq> hmmmm nm, apparently you can do so - I've always gone through bastion.
20:40:04 <nirik> all machines should log ssh tho right?
20:40:05 <phuzion> ke4qqq: hosted1 can be accessed without bastion.
20:40:06 <ke4qqq> yeah, guess svn access demands that
20:40:16 <nirik> so, we should be able to grep secure logs for ssh connections?
20:40:31 <sgallagh> nirik: That would be one approach
20:41:09 <phuzion> FAS logs the last login, right?
20:41:34 <sgallagh> phuzion: That wouldn't address the case of people who have given up on infra but are still packagers or use fedorapeople space
20:41:49 <sgallagh> Or fedorahosted projects
20:41:58 <phuzion> sgallagh: It would grab a few people though.
20:42:31 <phuzion> Use that as a starting list, then go through with other methods to build the list up to a larger size.
20:42:34 <sgallagh> phuzion: I think scraping the ssh logs of the privileged systems makes more sense
20:42:44 <CodeBlock> I say just get the list and write a script to do some log01 searching, and see who hasn't logged into FI servers (minus hosted, people) and go from there
20:43:20 <CodeBlock> well.... hm
20:43:31 <CodeBlock> does log01 have hosts' secure log? *checks*
20:44:21 <smooge> well I will start on making a list of people who are in the sysadmin groups. I will check which people have been active and who have not and I will present my findings by the 6th
20:44:25 <smooge> of January
20:44:45 <smooge> which brings up the next item
20:45:03 <smooge> #action smooge  I will start on making a list of people who are in the sysadmin groups. I will check which people have been active and who have not and I will present my findings by the 6th of january
20:45:09 <CodeBlock> yes, it does have secure logs, cool
20:45:13 <CodeBlock> that'll be easy to script then
20:45:31 <smooge> #topic US Holiday breaks
20:46:14 <smooge> Starting on the 25th of December until the 3rd of January, Red Hat will be on its winter closing and many members will be out and about
20:46:30 * nirik should be around almost all the time.
20:46:32 <smooge> I myself will be for part of it in Anaheim seeing a certain mouse
20:46:54 <smooge> I would like to put in another sloshy freeze for that week.
20:47:21 <smooge> basically no major changes without +1 on list or email.
20:47:24 * skvidal is going go be awol from the 26th until the 30th I suspect
20:47:33 <skvidal> okay so I'll be AWL not awOL
20:47:35 <skvidal> but whatever
20:47:53 <phuzion> I'd suggest an infra freeze starting the 24th of December through the 1st of January, since that's when most people that will be traveling will be gone
20:48:08 <Southern_Gentlem> phuzion,  23rd
20:48:17 <goozbach> +1 to the 23rd
20:48:25 <sgallagh> +1 to the 23rd as well
20:48:49 <smooge> Ok 23rd to the 2nd. I dont want ANYONE making changes after boozing it up on New Years Eve OR hangover day
20:48:59 <phuzion> smooge: valid point.
20:49:22 <Southern_Gentlem> smooge, 2 is sunday might move that the the 3rd
20:49:24 <phuzion> Many people will be severely hung over, or possibly still intoxicated on the 1st.
20:49:38 * gholms considers writing pam_beergoggles
20:49:40 <smooge> now on the other hand this is the time of year where lots of people are away from school and work
20:49:41 <goozbach> ten minute warning
20:49:55 <sgallagh> phuzion: I love New Year's Day. Best skiing day of the year, since no one else shows up :)
20:49:56 <CodeBlock> agreed. And I don't do any of that, and will be around if anything goes weird.... so if anyone needs a point of contact, who is known to not be drunk/etc, I have access to puppet and most stuff, so.... yeah
20:50:00 <CodeBlock> and with that, I'll be back in a few.
20:50:06 <smooge> while these people are mostly interested in Visa/Mastercard/etc they always like to look for things to try over break.
20:51:06 <skvidal> smooge: so you're saying that dec 23rd we should change the fp.o page to say 'we support wikileaks' so we'll be left alone?
20:51:14 <smooge> In the case that something pops up that needs MUST_GET_SOMEONE_NOW attention, we will make sure there is a listed number and procedure of things
20:51:26 <phuzion> smooge: You talking about a potential DDoS or something?
20:51:28 <ke4qqq> skvidal: a banner in rotation should suffice :)
20:51:37 <smooge> no because all the people who hate wikileaks will just point at us.
20:51:39 <skvidal> ke4qqq: should we rotate opposing ones?
20:51:44 <smooge> it is a no win situation.
20:51:49 <skvidal> ke4qqq: so 'we [don't] support wikileaks'
20:52:01 <skvidal> ke4qqq: juat have a random var as to which one you get? ;)
20:52:19 <ke4qqq> skvidal: worksforme
20:52:24 <skvidal> ke4qqq: :)
20:52:33 <smooge> anyway.. I am not sure about DDoS but we saw a lot more ssh attempts and such last break
20:52:38 <brianlamere> (keep in mind how this may affect the ability of people to use Fedora in DOD networks)
20:53:07 <rbergeron> smooge: you're going to disneyland without me?
20:53:10 <rbergeron> :(
20:53:15 <smooge> talk to the kid
20:53:20 <rfelsburg> smooge: do we employ any ssh attempt throttling?
20:53:26 <smooge> we do
20:53:37 <phuzion> Honestly, I'd say that the potential of getting DDoS'd during break is highly unlikely.  Visa and Mastercard and PayPal were DDoS'd because of the wikileaks stuff going on and their stance on it.  Fedora, as far as I know, has made no official statement about Wikileaks.
20:53:41 <smooge> I am more looking at the general picture than specific ones
20:54:27 <smooge> basically you have bored people and internet connections. you have a plethora of "hey someone on #phreak said run this command! you get dancing penguins" and you end up with sometimes the best defences having a hole in them
20:54:37 <phuzion> But I do agree, we should have a contact listed as the go-to-guy in case of emergency.
20:54:53 <smooge> so if something comes up and me or skvidal or dgilmore aren't available.. we will have a fall back plan
20:55:15 <skvidal> I should be not very far from workable network access
20:55:31 <smooge> ok insight meeting ran long and we have a bunch of tickets we aren't oging to get to this meeting.. and cloud starts up soon.
20:55:32 <skvidal> and I know I'll be available for about 5-6 hours of the 26th and 29th
20:56:08 <phuzion> I'll be available for about 18 hours nonstop on the 26th starting very early in the morning.
20:56:21 * nirik should be around most all the time.
20:56:42 <smooge> so I am going to close early and we will move discussion to #fedora-admin. Next week we will have a signup sheet and what to do if raptor attacks fedora servers
20:57:02 * rbergeron will feed the raptors hotdogs instead
20:57:12 <smooge> #endmeeting