19:00:17 <nirik> #startmeeting Infrastructure (2013-05-09)
19:00:17 <zodbot> Meeting started Thu May  9 19:00:17 2013 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:17 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:17 <nirik> #meetingname infrastructure
19:00:17 <nirik> #topic welcome y'all
19:00:17 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean
19:00:17 <zodbot> The meeting name has been set to 'infrastructure'
19:00:17 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean
19:00:24 * abadger1999 here
19:00:28 * lmacken 
19:00:31 <nirik> hey folks. Who's around for a infrastructure meeting?
19:00:37 * puiterwijk 
19:00:40 * threebean threebean
19:00:42 * holmed 
19:00:49 <nirik> 6 beans!
19:00:53 <threebean> 9!
19:01:10 * skvidal is here
19:01:21 * swa_work here
19:01:31 * nirik guesses it depends on what the default operator is.
19:02:02 <puiterwijk> nirik: nope, threebeans threebeans => "(Int -> Int) -> Int" :)
19:02:13 * threebean sees your point and raises a SyntaxError
19:02:14 <nirik> :)
19:02:25 <holmed> Its so nerdy, i love it :D
19:02:26 <nirik> anyhow, lets go ahead and get started.
19:02:27 * tflink is here
19:02:33 <nirik> #topic New folks introductions and Apprentice tasks
19:02:42 <nirik> Any new folks like to introduce themselves?
19:02:49 <nirik> or apprentices with questions or comments?
19:02:53 <holmed> Im Ed
19:02:59 <holmed> Hello :D
19:03:24 <nirik> welcome holmed.
19:03:29 <holmed> I haven't really had much experiance working on FOSS
19:03:51 <nirik> are you more interested in sysadmin type tasks? or application development? or both?
19:04:19 <holmed> What do they both do? Without sounding too brainless
19:04:30 <nirik> no worries. ;)
19:04:46 <nirik> So the sysadmin team does more day to day operations tasks... and some scripting work.
19:05:13 <nirik> the Apps/devel folks build and maintain (some of) the applications that the sysadmins run day to day
19:05:21 <nirik> see apps.fedoraproject.org for a full list.
19:05:29 <holmed> What kind of things do they script? and What operations tasks?
19:05:34 <holmed> The SysAdmins
19:05:46 <nirik> see http://fedoraproject.org/wiki/Infrastructure/GettingStarted for much more info and links to other stuff. ;)
19:06:20 <nirik> adding projects, responding to outages, rebooting servers, applying updates, changing configuration based on needs, etc.
19:06:33 <skvidal> cursing
19:06:36 <skvidal> you forgot cursing
19:06:39 <swa_work> yah skvidal
19:06:43 <holmed> Ah, yeah that's probably where i'd prefer to be ;)
19:07:07 <nirik> holmed: anyhow, ping me in #fedora-admin after the meeting and we can get you started. ;)
19:07:18 <nirik> and do ask questions as we go. :)
19:07:33 <holmed> Ok thanks :)
19:07:52 <nirik> #topic Applications status / discussion
19:07:57 <nirik> ok, any applications news?
19:08:03 <nirik> I have a few things from this week...
19:08:30 <nirik> #info fas security issue was discovered, analyzed, patched and announced.
19:08:44 <nirik> #info fedocal calendar was announced.
19:08:56 <nirik> https://apps.fedoraproject.org/calendar/
19:09:25 <nirik> Anyone have other items from the apps side?
19:09:44 <threebean> Fedora Hosted fedmsg messages coming soon.
19:09:51 <abadger1999> #info fas security announcement: http://lists.fedoraproject.org/pipermail/advisory-board/2013-May/011996.html
19:10:14 <threebean> They'll remain disabled by default, but there's a SOP to enable them if someone asks for them.
19:10:21 <nirik> we have a freeze coming up next tuesday
19:10:31 <nirik> are there any releases planned to get in before that?
19:10:54 <abadger1999> I would like to get a new python-fedora out but I don't know that I can get it out there.
19:10:58 <puiterwijk> abadger1999: is that a security announcement? it seems like a board meeting announcement?
19:11:11 <abadger1999> It's an API design question so I want to get it setup.
19:11:12 <holmed> Sorry, What's a SOP? And what is a freeze?
19:11:12 <abadger1999> oops
19:11:19 * abadger1999 looks for the correct link
19:11:24 <puiterwijk> #undo
19:11:31 <puiterwijk> oh, I can't do that :)
19:11:38 <nirik> holmed: SOP is "standard operating procedure"
19:11:39 <abadger1999> #undo
19:11:39 <zodbot> Removing item from minutes: <MeetBot.items.Info object at 0x263ae610>
19:11:43 <nirik> so, docs on how to do something.
19:11:45 <abadger1999> #info http://lists.fedoraproject.org/pipermail/announce/2013-May/003154.html
19:11:48 <abadger1999> #chair puiterwijk
19:11:48 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik puiterwijk ricky skvidal smooge threebean
19:11:58 <holmed> Ah thanks :)
19:12:01 * skvidal looks up confused again
19:12:02 <skvidal> oh
19:12:05 <threebean> #info SOP for fedorahosted fedmsg messages http://infrastructure.fedoraproject.org/infra/docs/fedorahosted-fedmsg.txt
19:12:23 <nirik> freeze is something we do several weeks before each fedora milestone... Alpha/Beta/Final release. It's where we make fewer changes and require review of those changes.
19:12:32 <nirik> So we don't cause problems with our infrastructure before a release.
19:12:51 <puiterwijk> abadger1999: thanks
19:12:54 <holmed> Ah ok
19:13:06 <nirik> and more on that topic in a few actually. ;)
19:13:09 <threebean> (before that freeze on Tuesday, I'd like to get out a release of fedmsg_meta_fedora_infrastructure and another of fedora-tagger.)
19:13:18 <nirik> threebean: cool.
19:13:49 <holmed> So what is the Alpha/Beta release for? Infrastructure apps?
19:13:57 <nirik> oh, I had a general app question...
19:14:05 <nirik> holmed: no, Fedora the linux distro. :)
19:14:26 <holmed> Ah! I'm starting to get it now ;)
19:14:43 <nirik> are httpd logs useful for things like busgateway/fedocal/etc... basically the split out per app server instances, does it make sense to save httpd logs for them?
19:15:17 <holmed> Also where do pre-release bug reports go?
19:15:42 <nirik> holmed: for fedora? or for fedora infrastructure?
19:15:59 <holmed> Just for Fedora
19:16:03 <abadger1999> You'd think I'd learn to re-read what I type before hitting return:   re python-fedora update: It's an API design question so I want to get it setup right the first time
19:16:06 <nirik> bugzilla.redhat.com
19:16:12 <holmed> Ok thanks
19:16:22 <threebean> nirik: hmm, I think don't think busgateway01 has any httpd running on it.
19:16:32 <nirik> threebean: ok. that answers it for that instance. ;)
19:16:43 <threebean> but saving/storing httpd logs for fedocal sounds like a good thing to me ;p
19:16:46 <nirik> I'll drop a note to the list about this.
19:17:14 <nirik> any other application news?
19:17:16 <skvidal> hmmmm
19:17:19 <skvidal> speaking of saving httpd logs
19:17:37 <nirik> yeah?
19:17:37 <skvidal> more automagic there might be good
19:17:43 <skvidal> from either apps themselves
19:17:46 <skvidal> or from apache
19:17:50 <skvidal> given FAS this week
19:18:03 <nirik> yeah, I just added the stg instances to log02's sync script.
19:18:08 <nirik> thats what made me think about this.
19:18:12 <skvidal> :)
19:18:18 <nirik> but yeah, might be better to do it in a better way
19:18:25 <nirik> #topic Sysadmin status / discussion
19:18:41 <nirik> So, on the sysadmin side, we did a full update/reboot cycle this last week on our machines and cloud.
19:19:07 <nirik> mirrorlist server melted down a few times. ;( we should look at mitigation/solutions there.
19:19:07 <skvidal> and almost everything went wrong
19:19:12 <holmed> Sorry, which machines are these? What do they do?
19:19:19 <skvidal> holmed: they do everything
19:19:29 <skvidal> and by which machines it is 'all of them' pretty much
19:19:41 <nirik> holmed: all our servers that build, manage and produce fedora.
19:19:57 <holmed> And you take them all down at the same time?
19:20:05 <nirik> nope. :)
19:20:07 <threebean> well, we didn't want to.  :P
19:20:26 <skvidal> well the cloud is "special"
19:20:31 <nirik> we have a bunch we can do anytime... either because they are not always doing something, or because we have several servers doing that same thing.
19:20:47 <nirik> we have a very small number that when we do them it causes outages... mostly database servers.
19:20:55 <nirik> we had a scheduled outage tuesday to do those.
19:20:57 <holmed> Oh
19:21:29 <nirik> lets see... what else did we have on the sysadmin side...
19:21:59 <skvidal> new builder migration to ansible
19:21:59 <nirik> Oh, we have some new storage that will hopefully come on line soon.
19:22:08 <nirik> skvidal: ah yes.
19:22:23 <nirik> #info builder setup has been moved to ansible.
19:22:30 <nirik> skvidal: can we remove the old builder repo now?
19:22:34 <skvidal> not quite yet
19:22:35 <nirik> or move it aside?
19:22:37 <nirik> ok
19:22:37 <skvidal> I need to port the scripts over
19:22:41 <skvidal> but we can stop using it
19:22:47 <holmed> So, to clarify, SysAdmins Administrate the Systems of Fedora e.g. Website host? And what else do these servers do?
19:23:09 <skvidal> I still need to portand test the bkernel boxes
19:23:14 <nirik> skvidal: oh, there's a problem with releng01. I'll ping you outside the meeting to track that down.
19:23:20 <skvidal> nirik: ah - thanks
19:23:53 <nirik> holmed: yeah. So, websites, builders, compose machines, signers, updates management, download servers, wiki, docs, you name it.
19:24:02 <nirik> holmed: if it's under 'fedoraproject.org' it's us. ;)
19:24:07 <threebean> holmed: lots and lots -- there's a pretty comprehensive list of the services we maintain at https://apps.fedoraproject.org/
19:24:36 <holmed> Wow. I didn't think it was that big.
19:24:45 <nirik> I think we are around 180ish instances now?
19:25:43 <jerzyr> this graph (fedora apps) is really super
19:25:56 <nirik> it's pretty cool. thank threebean for setting it up
19:26:29 <nirik> ok, I think thats it on sysadmin... unless we want to try and brainstorm for a few about mirrorlists.
19:27:29 <nirik> #topic Freeze
19:27:36 <nirik> so, we have a freeze coming up next tuesday.
19:27:48 <nirik> We discussed changing what machines were covered.
19:28:01 <nirik> shall we finalize that before this freeze and use the new method?
19:28:18 <skvidal> yah
19:28:19 <skvidal> sure
19:28:23 <nirik> it was: "assume every machine is frozen, except for list of machines that is not"
19:28:29 <nirik> (I think thats what we decided)
19:28:42 <nirik> if so, we need to for sure note the ones excepted.
19:29:10 <skvidal> nirik: right
19:29:13 * nirik looks back on the list.
19:29:49 <nirik> oh right. None of *stg* or *dev* are frozen ever
19:30:17 <nirik> I can send a proposed list of non frozen hosts to the lists for discussion?
19:30:21 <skvidal> I think so
19:30:30 <skvidal> and then we can add them to the ansible inventory files in host_vars
19:30:39 <skvidal> so we can dump them out with that inventory script pretty easily
19:30:53 <nirik> right. and we can note the script. Should we have that dump out to /var/log/ on lockbox or something?
19:32:01 <nirik> #info will finalize the list of non frozen hosts before tuesday.
19:32:31 <nirik> #topic geoip
19:32:43 <nirik> So, there's a thread on the list about a geoip service...
19:32:54 <holmed> What is GeoIP?
19:32:55 <nirik> the anaconda folks would like to be able to use something to set some defaults on installs.
19:33:05 <nirik> locating someone by their ip address.
19:33:12 <holmed> Also who're the anaconda folks?
19:33:16 <nirik> http://en.wikipedia.org/wiki/Geoip
19:33:31 <nirik> anaconda is the fedora installer. :)
19:33:47 <holmed> Oh, i thought i'd seen it somewhere
19:33:53 <threebean> and the anconda folks are the people who work on improving the fedora installer.
19:34:26 <holmed> And what needs to be done about geoip?
19:34:28 <nirik> so, not sure where this discussion will lead, but we might be running another apache mod to provide this information to them
19:34:44 <nirik> holmed: they have asked us if we can run a server to provide that info to them.
19:35:13 <nirik> I've suggested they also look at the geoclue library, which might do what they want without us having to run a server. ;)
19:35:46 <nirik> so, mostly I just wanted to bring this up...
19:35:48 <holmed> The location of the user?
19:36:04 <skvidal> nirik: I'm in favor of not running another service
19:36:08 <nirik> yes. So they can set things like timezone and language... or at least default those to the local area.
19:36:28 <holmed> Ah i get it
19:36:30 <nirik> if you install fedora in germany, it could see that and set your timezone and lang and possibly other stuff.
19:36:34 <nirik> skvidal: me too.
19:36:36 <holmed> Yeah
19:36:58 <nirik> ok, moving on, just wanted to note it. :)
19:37:01 <nirik> #topic Private Cloud status update / discussion
19:37:11 <nirik> we also updated and rebooted our private cloud this week.
19:37:15 <kushalk124_> Geoclue seems to be a good option
19:37:34 <nirik> https://fedoraproject.org/wiki/Infrastructure_private_cloud
19:37:38 <skvidal> and things broke
19:37:40 <skvidal> left and right
19:37:44 <skvidal> We fixed most of them
19:37:45 <nirik> and yeah, it didn't go great.
19:37:49 <nirik> but I think we are all back working right?
19:37:55 <skvidal> nirik: yes and no
19:38:02 <skvidal> yes - all the instances we know about are back up
19:38:15 <skvidal> no - there  are stil lsome old 'shut off' instances that I can't figure out who they belonged to
19:38:25 <skvidal> I'm going to go back through the logs and figure out who the yare
19:38:28 <skvidal> and either kill them
19:38:33 <skvidal> or try to restore them
19:38:49 <nirik> ok.
19:38:53 <nirik> sounds good.
19:38:54 <skvidal> otherwise it came up ok
19:39:06 <nirik> #info reboot didn't go well, we have provided feedback to openstack folks about it.
19:39:08 <skvidal> also nirik and I discovered we didn't have the same thought on what the passwords were for those clouds :)
19:39:12 <skvidal> that's been corrected :)
19:39:17 <nirik> #info back up and running now for the most part.
19:39:22 <skvidal> on the plus side
19:39:28 <skvidal> other than the actual time they were off
19:39:33 <skvidal> the instances came back pretty well
19:39:36 <skvidal> and the suspending of them worked
19:39:40 <skvidal> except for a couple of little details
19:39:44 <skvidal> like the time being way offf
19:40:09 <skvidal> anyway - that's all
19:40:45 <nirik> yeah, always learning on this stuff. ;)
19:40:52 <nirik> #topic flock
19:41:04 <nirik> http://flocktofedora.org/
19:41:09 <nirik> so, this is coming up in aug
19:41:35 <nirik> submissions are open for talks, hackfests, sprints, etc
19:41:51 <nirik> everyone should think about what they could submit.
19:42:04 <nirik> Or... what they would like to see other folks talk on. :)
19:42:19 * threebean isn't sure if oddshocks is in the meeting today..
19:42:20 <nirik> we have about a million things we could do as hackfests.
19:42:31 <threebean> but we should probably do a talk on the badges stuff we're working on this summer.
19:42:36 <nirik> +1
19:43:04 <nirik> I think we should do a ansible hackfest.
19:43:24 <skvidal> ooo
19:43:26 * nirik might try and find time to actually work on formulas again and do something around that.
19:43:32 <skvidal> or a postgres replication hackfest
19:43:51 <nirik> I could do any kind of intro to infra or whatever talks if people think there would be interest.
19:43:53 <pvod> skvidal: +1
19:43:56 <nirik> skvidal: yes!
19:44:10 <skvidal> nirik: I wonder if TGL is coming  - maybe get him to lead that one :)
19:44:29 <nirik> that would be great. At least figuring out for sure how we want to go.
19:44:43 <nirik> we could also do something around bugtrackers if we wanted.
19:45:06 <threebean> yeah
19:45:11 <puiterwijk> I have currently submitted a talk on OpenID
19:45:14 <nirik> (get qa/kernel/maintainers all involved to gather actual better requirements)
19:45:25 <nirik> puiterwijk: excellent. Like the one you gave on irc?
19:45:31 <puiterwijk> nirik: yeah
19:45:47 <nirik> great. I found that very informative.
19:45:54 <skvidal> so last year
19:46:04 <skvidal> I had a bike shed built in my yard
19:46:04 <puiterwijk> nirik: let's hope the flock organizers think the same
19:46:11 <skvidal> and I think I might talk about selecting paint colors for it
19:46:23 <skvidal> we ultimately went with 'off white/grayish'
19:46:27 <skvidal> but it took a lot of discussion
19:46:29 <skvidal> :)
19:46:43 <nirik> skvidal: I think that might be too controversal... the flock folks would never select that talk. ;)
19:46:49 <skvidal> heh
19:47:18 <puiterwijk> I had also been thinking about submitting an hackfest for the OAuth ideas we had (at least discuss it)
19:47:24 <nirik> anyhow, please propose talks/etc... and if you can think of ones other people could give, ask them. ;)
19:47:38 <Southern_Gentlem> *crys*
19:47:46 <nirik> puiterwijk: I think that might go better as a workshop ?
19:48:05 <puiterwijk> nirik: sure, could do as well. I just didn't know the major difference between those three ;)
19:48:34 <nirik> yeah, might be good to make sure what is what there. ;)
19:48:42 <nirik> ok, any other flock talk? (har har)
19:49:11 <nirik> #topic Upcoming Tasks/Items
19:49:20 <nirik> #info 2013-05-14 to 2013-05-28 BETA infrastructure freeze
19:49:20 <nirik> #info 2013-05-20 - bugzilla upgrade.
19:49:20 <nirik> #info 2013-05-28 F19 beta release
19:49:20 <nirik> #info 2013-05-31 end of 1st quarter
19:49:20 <nirik> #info 2013-06-01 nag fi-apprentices
19:49:21 <nirik> #info 2013-06-08 drop inactive apprentices
19:49:23 <nirik> #info 2013-06-18 to 2013-07-02 FINAL infrastructure freeze.
19:49:25 <nirik> #info 2013-07-01 nag fi-apprentices
19:49:27 <nirik> #info 2013-07-02 F19 FINAL release
19:49:31 <nirik> any other items folks would like to note or schedule?
19:49:41 <abadger1999> bugzilla upgrade -- where are we on testing our scripts?
19:49:42 * nirik should move this all to a infrastructure calendar.
19:49:43 <tflink> ooh, bugzilla upgrade in the middle of freeze ... again
19:49:48 <nirik> tflink: yeah. ;(
19:50:09 <nirik> abadger1999: good question. I think mostly we are good... not sure if bodhi has been checked yet.
19:50:19 <nirik> there might be some others.
19:50:28 <abadger1999> I know rview-stats and pkgdb-bugzilla-sync are working both in prod with py-bz-0.8 and partner w/ that + new bugzilla server
19:50:37 <abadger1999> not sure about anything else
19:50:40 <nirik> did partner get updates?
19:50:43 <nirik> updted?
19:50:46 <abadger1999> (and I know that pkgdb is broken)
19:50:55 <abadger1999> at least the bugz. link
19:50:59 <tflink> wouldn't the stg switch to partner-bugzilla and python-bugzilla-0.8 catch most of the possible breakage?
19:51:02 <threebean> hm, I haven't checked the /packages app either.
19:51:17 <nirik> tflink: mostly... we don't test bodhi there tho for example.
19:51:46 <abadger1999> nirik: not sure about partner being updated or not.
19:51:51 <nirik> yeah, partner is on 4.4
19:52:00 <threebean> (/packages looks okay in staging)
19:52:57 <nirik> so, I guess we should try and check the list thru again and see...
19:53:30 <nirik> most things are ok though now I think.
19:53:54 <nirik> we do have the new python-bugzilla in prod now.
19:54:01 <nirik> (but bugzilla.redhat.com isn't 4.4 yet)
19:54:34 <nirik> #info Open Floor
19:54:52 <nirik> #topic Open Floor
19:54:54 <nirik> whoops
19:54:57 <puiterwijk> heh :)
19:55:03 <abadger1999> nirik: is the new python-bugzilla everywherre in prod?  I know some hosts (fas servers, app01) weren't upgraded
19:55:05 <nirik> Any items for open floor? questions? comments?
19:55:18 * abadger1999 checks if those were fixed
19:55:18 <nirik> abadger1999: I think it is now, I didn't exclude it on tuesday.
19:55:24 <abadger1999> <nod>  Cool
19:55:52 <threebean> I just got fedora-tagger-2.0.3 pushed out and everything seems to be fine :)  https://apps.fedoraproject.org/tagger/nethack
19:56:09 <nirik> threebean: does this have that tag url that bodhi needed?
19:56:13 * threebean nods
19:56:20 <nirik> coolio
19:57:20 <skvidal> nirik: something for open floor
19:57:28 <skvidal> the fas thing this week
19:57:32 <nirik> yeah.
19:57:40 <skvidal> it was discovered by relrod looking at something in the code he was working on
19:57:44 <skvidal> and noticing it was 'odd'
19:57:55 <skvidal> if anyone here notices something 'odd'
19:58:02 <skvidal> and thinks 'no, surely they know about this'
19:58:08 <skvidal> don't call use 'shirley'
19:58:08 <skvidal> :)
19:58:12 <skvidal> please tell us
19:58:13 <nirik> +a lot
19:58:14 <abadger1999> +1
19:58:35 <threebean> :P
19:58:36 <skvidal> to be fair the obscurity of the issue helped us
19:58:49 <nirik> agreed. I'm amazed at how long it had been around.
19:58:51 <skvidal> but we're quite happy relrod mentioned it
19:59:11 <skvidal> we are reasonably confident that this was not used by anyone
19:59:15 <skvidal> but not COMPLETELY so
19:59:19 <nirik> yes indeed. Kudos to relrod for noticing it.
19:59:20 <skvidal> so - when in doubt - change your password
19:59:42 <skvidal> but I would not freak out about changing the password :)
19:59:59 <puiterwijk> skvidal: probably also the security question/answer if you're really scared?
20:00:11 <nirik> when in doubt, scream and shout? oh wait, thats not right.
20:00:33 <skvidal> puiterwijk: indeed - but to be fair my security question/answer is more 'nirik calls me and says - someone requested your account password be changed, that's hilarious, isn't it?'
20:00:33 <nirik> anyhow, if nothing else, will close out the meeting in a few.
20:00:42 <nirik> skvidal: :)
20:00:43 <skvidal> puiterwijk: :)
20:00:48 <puiterwijk> skvidal: heh, yeah :)
20:00:54 <Southern_Gentlem> require a yearly password change like a lot of companies and uni do
20:01:01 <nirik> lets continue over in #fedora-admin, #fedora-noc and #fedora-apps
20:01:10 <nirik> Southern_Gentlem: not worth the pain and horror I fear.
20:01:29 <skvidal> okie doke
20:01:34 <nirik> #endmeeting