19:01:16 <smooge> #startmeeting Infrastructure (2013-10-10)
19:01:16 <zodbot> Meeting started Thu Oct 10 19:01:16 2013 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:16 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:01:24 <smooge> #meetingname infrastructure
19:01:24 <zodbot> The meeting name has been set to 'infrastructure'
19:01:31 <smooge> #topic welcome to all and to all a welcome
19:01:41 <smooge> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
19:01:41 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
19:01:48 * relrod here
19:01:49 * anshprat here
19:01:52 <oddshocks> What is kickin infra team
19:02:06 * lmacken 
19:02:08 <smooge> hello
19:02:20 <smooge> hi lmacken how is the weather in Co?
19:02:24 * ianweller here
19:02:50 <smooge> We are having a front go through so I may get knocked off from a thunderstorm
19:02:57 <lmacken> smooge: pretty decent today, a little overcast
19:03:03 * robyduck here
19:03:15 * abompard here too
19:03:19 <smooge> #topic New folks introductions and Apprentice tasks
19:03:28 <smooge> Hello any new folks here?
19:03:48 <akshayb> new here
19:03:50 <dgilmore> hi im new
19:04:00 <akshayb> Hello
19:04:01 <dgilmore> sorry bad joke
19:04:06 <dgilmore> hi akshayb
19:04:18 <akshayb> Hi
19:04:21 <dgilmore> welcome
19:04:27 <smooge> hello akshayb
19:04:30 <smooge> hi frankieonuonga
19:04:36 <smooge> #chair frankieonuonga
19:04:37 <zodbot> Current chairs: abadger1999 dgilmore frankieonuonga lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
19:04:40 <akshayb> Hi
19:04:44 <frankieonuonga> hi smooge
19:04:46 <abadger1999> hola
19:04:59 <smooge> hello akshayb what are you looking at?
19:05:02 <frankieonuonga> hi guys
19:05:16 <dgilmore> abadger1999: ¿como estas?
19:05:38 <akshayb> I was looking at doing some sysadmin stuff
19:05:38 <frankieonuonga> dgilmore: ola amigo
19:05:47 <robyduck> is dgilmore still in cusco?
19:05:55 <robyduck> :)
19:06:03 <dgilmore> robyduck: no en Illinois
19:06:09 <akshayb> @smooge Have recently started with python
19:06:18 <abadger1999> dgilmore: un poco enfermo.
19:06:48 <dgilmore> abadger1999: :(
19:06:58 <smooge> akshayb, cool. I am too (sort of.. )
19:06:59 <dgilmore> frankieonuonga: óla
19:07:21 <dgilmore> akshayb: any bit in particular interest you?
19:08:19 <smooge> I am wanting to go over the Apprentice program and flesh it out in the next 2 weeks. It really needs some docs and work to make it better (like all things). People who have been or are apprentices should email me at smooge@fedoraproject.org with suggestions of what would help
19:08:25 <akshayb> dgilmore: No preferences as such
19:09:00 * croberts steps in late
19:09:03 <croberts> hi everyone
19:09:24 <smooge> #info Apprentices please email smooge@fedoraproject with ideas on what apprenticeship means to them and what they want to do.
19:09:34 <frankieonuonga> welcome croberts.we have just started.not gone too far
19:09:45 <anshprat> smooge: I ve a few, will send them to you..
19:09:58 <smooge> ok next item on the agenda will be
19:10:06 * frankieonuonga offering to help smooge to do the write up if it is after wed next week
19:10:28 <smooge> thanks frankieonuonga it will be a couple of days
19:10:40 <smooge> s/days/weeks/ to get the information so yes
19:10:45 <smooge> I would welcome the help
19:10:54 <smooge> #topic Applications status / discussion
19:11:06 <smooge> abadger1999, lmacken etc.. up to you
19:11:23 <smooge> any app work you want to cover?
19:11:32 <abadger1999> slow week I think -- many people are out this week.
19:11:50 <abadger1999> I've been sick and busy with fesco/fpc things so not much from me.
19:11:55 <lmacken> bodhi1 and 2 work underway, probably going to do a bodhi update on releng4 today
19:12:04 <abadger1999> the wallpaper election in nuancier-lite worked.
19:12:13 <abadger1999> there were a few things that I had to hotfix for it.
19:12:38 <abadger1999> but nothing that blocked valid votes.
19:13:00 <anshprat> abadger1999: ah, I missed that.. to which list was the info for the election sent?
19:13:05 <abadger1999> lmacken: bodhi1.x update?  Or all the way to bodhi2?
19:13:07 <smooge> sorry to hear you have been sick.
19:13:17 <lmacken> abadger1999: just a bodhi1x bugfix update. not there yet :)
19:13:17 <smooge> bodhi3 with python3 :)
19:13:20 <abadger1999> <nod>
19:13:34 <lmacken> I'm going to try and make some headway this week on our app logging setup in stg
19:13:37 <lmacken> .ticket 3627
19:13:39 <zodbot> lmacken: #3627 (Configure centralized logging for all of our webapps.) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/3627
19:13:44 <croberts> frankieonuonga: what is the status on your search app
19:13:56 <relrod> I did some Fedora Mobile and HRF week last weekend - HRF (Human Readable Fedmsg: https://github.com/fedora-infra/hrf/) can now query datagrepper directly, which saves a slow http request from Fedora Mobile, which makes it show the fedmsg newsfeed about twice as fast :)
19:14:03 <abadger1999> anshprat: I think the design list but I'm not sure -- pingou did hte heavy lifting.  I just took on the role of bug fixer since he's on vacation and the election was ongoing.
19:14:18 <frankieonuonga> abadger1999: get well soon
19:14:23 <abadger1999> thanks
19:15:00 <ianweller> the whole datanommer/datagrepper stack is up to date in staging, and working
19:15:03 <anshprat> abadger1999: alright , guess will sign up for those as well.. what are the other elections we are planning to use nuancier for?
19:15:09 <abadger1999> Ah -- one thing I noticed when working on nuncier hotfixes,  we need to work on our app deployments a bit.
19:15:13 <frankieonuonga> croberts: still working on something..nothing that I have pushed out.Java was not accepted so need to understand what was written and optimize those algorithms . will take at least 2 weeks before i can comment
19:15:17 <ianweller> before going into production with it i would like to have a plan of what to update on what machines in what order
19:15:38 <lmacken> ianweller: excellent.
19:15:48 <ianweller> i am fairly certain i can go like this:
19:16:13 <ianweller> busgateway01 update everything -> datagrepper01 update everything -> everything else update fedmsg
19:16:15 * abadger1999 looking for tickets
19:16:43 <ianweller> i just am not sure what order the update on busgateway01 should go in, in order to minimize downtime
19:16:44 <lmacken> ianweller: didn't we hit a bus -> irc issue this week or something?
19:16:59 <ianweller> lmacken: that was related to the fedmsg plugin not being loaded in supybot
19:17:16 <ianweller> i should probably get someone to give me admin access to supybot just in case that happens again
19:17:36 <ianweller> er, no, that was irc -> bus.
19:17:38 <abadger1999> In particular, this ticket is probably an easyfix if an apprentice wants to take it on: https://fedorahosted.org/fedora-infrastructure/ticket/4046   -- HTTPS services need haproxy change
19:17:39 <lmacken> ah okay, so nothing mysterious
19:17:47 <ianweller> lmacken: we hit a bus -> irc issue that was quickly fixed by threebean yes
19:18:41 <ianweller> i intend to come up with a plan for datanommer/datagrepper upgrade in production, email the list today, and make it happen tomorrow.
19:18:50 <ianweller> (EOF)
19:19:19 <abadger1999> ianweller: was that a packages-out-of-sync issue?
19:19:37 <ianweller> abadger1999: that was a legitimate bug that we found, that i'm looking for the diff for
19:19:38 <abadger1999> or "new package, needed to resync the configs from ansible?"
19:19:40 <lmacken> I'd like to figure out how to do zero-downtime bus reboots. I'll bet we could do it with this feature: https://lwn.net/Articles/542629/
19:19:41 <abadger1999> okay.
19:19:56 <ianweller> https://github.com/fedora-infra/fedmsg/commit/c857d15f75736528c870a74a05e07181164f9012 found it
19:20:51 <abadger1999> ah.
19:21:35 <abadger1999> k
19:22:49 <abadger1999> For ticket 4046 -- we just need our haproxy config modified so that it actualy checks all services to see if they're up or down.  Right now, many aren't being checked so if one node goes down we get transient errors.
19:23:17 <abadger1999> whenever haproxy sends a request to the down node.
19:24:01 <smooge> and thus we spent our evenings and morning earlier this week :)
19:24:29 <smooge> or something like that..
19:24:37 <smooge> anything else on applications?
19:24:53 <anshprat> I would need some help on askbot - rhbz 1010166
19:25:00 <frankieonuonga> i hope i gave croberts a good answer
19:25:06 <anshprat> stuck on celery versions..
19:25:29 <smooge> frankieonuonga, it does make sense to me
19:25:42 <smooge> anshprat, ok
19:25:46 <smooge> .bug 1010166
19:25:50 <zodbot> smooge: Bug 1010166 askbot-0.7.49 is available - https://bugzilla.redhat.com/show_bug.cgi?id=1010166
19:26:32 <anshprat> I think I just fixed it for el6 - now need to check for rawhide..
19:26:46 <smooge> anshprat, very cool.
19:28:34 <smooge> anshprat, kevin will be back on Monday I believe. He should be able to help on the items
19:28:50 <smooge> I am going to move to the next topic.
19:28:53 <anshprat> smooge: alright
19:29:30 <smooge> #topic Sysadmin status / discussion
19:29:41 <smooge> Ok since Kevin is out.. it is mainly me.
19:29:51 <smooge> We will be going into Beta freeze real soon
19:30:08 <smooge> [Actually I need to make sure I didn't miss it already.]
19:30:16 <lmacken> how soon is real soon? :)
19:30:16 <tflink> it's on tuesday
19:30:45 <smooge> So next tuesday. Please make sure any changes to applications etc are done before Monday afternoon.
19:31:19 <smooge> On tuesday 90% of changes will need to have two +1 from sysadmin-main people before being pushed to production.
19:31:57 <smooge> I need to put out the beta freeze tickets tomorrow
19:32:27 <smooge> lmacken, does that cause problems with your schedule?
19:32:41 <smooge> just want to get any plans dealt with.
19:33:36 <tflink> I may need some help getting a new blockerbugs version deployed (probably tomorrow) if the sudo issue I'm hitting is still around
19:33:52 <smooge> tflink, I believe it is :/
19:33:57 <lmacken> smooge: nope, should be fine
19:34:03 <smooge> glibc is the problem with the sudo items
19:34:17 <smooge> hmm need to report it agaisnt 6.5 and see if we can make it a blocker :)
19:34:18 <croberts> frankieonuonga :) you did
19:34:19 <tflink> ok, I thought that was just f19 nodes
19:34:32 <smooge> oh sorry.. maybe different bug then
19:34:44 <smooge> tflink, don't mind me. My coffee ran out 5 minutes ago
19:34:49 <tflink> probably not, I don't pretend to know much about it
19:35:04 <tflink> an offhand comment kevin made last week is about it :)
19:35:20 <smooge> Oh then that might be F19 only then.
19:35:29 <smooge> so I need to make it a different bug :)
19:35:33 <smooge> Ok my items:
19:35:46 <smooge> 1) Cisco UCS. We have a blade server with 4 blades.
19:36:06 <smooge> We should be able to make them buildvmhosts for dgilmore soon.
19:36:24 <smooge> then next quarter I can order 4 more blades and work on another chassis
19:36:48 <dgilmore> :)
19:36:50 <smooge> Let me just say that Hardware as a Service is a completely weird concept if you don't know that is what you are working on
19:37:13 <smooge> once I figured that out.. it became a lot clearer
19:37:17 <dgilmore> I need to set aside one or two of the old blades to separate build channels
19:37:42 <smooge> dgilmore, I don't know what is going to be the lifetime on the old blades.
19:37:51 <smooge> will work with nirik on that after he gets back
19:38:07 <dgilmore> smooge: well, we are going to need them for a  bit
19:39:08 <frankieonuonga> smooge: i feel you. i work for a cloud compute company building the platform and trust me..sometimes I dont even know what I am doing but it works
19:39:15 <smooge> dgilmore, I will make sure that the warranty on it is extended one more year.
19:39:37 <dgilmore> smooge: well if its not its not.
19:39:59 <smooge> dgilmore, no it will be. IBM hardware senses when it has no warranty
19:40:06 <smooge> down to the minute
19:40:18 <dgilmore> :)
19:40:59 <smooge> Third item for sysadmin items is it will be hardware inventory and maintenance time. We need to work out what hardware we own, what part of its lifetime it is in and what to replace it with and when
19:41:21 <smooge> #topic Upcoming Tasks/Items
19:41:26 <smooge> 1) Beta time on Tuesday
19:41:33 <smooge> 2) Inventory and budget
19:41:42 <frankieonuonga> just out of curiosity cant we spin of something where we are using commodity h/w
19:41:52 <anshprat> smooge: where is the inventory for h/w ?
19:42:09 <anshprat> (existing h/w)
19:42:14 <puiterwijk> anshprat: that is in the private repo
19:42:18 <smooge> 3) Re-enliving CSI documentation
19:42:22 <anshprat> puiterwijk: ok
19:43:09 <anshprat> smooge:  CSI ?
19:43:27 <ianweller> anshprat: http://infrastructure.fedoraproject.org/csi/free-software-policy/en-US/html-single/
19:43:36 <ianweller> "community services infrastructure standards"
19:43:37 <smooge> Community Something Infrastructure. A bunch of documents that were written a while ago on how to run our infrastructure
19:43:43 <ianweller> (i'm not sure if that's the most recent copy)
19:43:57 <smooge> ianweller, I expect it is
19:44:03 <anshprat> thanks! google gave me this.. https://fedorahosted.org/csi/
19:44:18 <smooge> yep that is upstream for them
19:44:23 <anshprat> ok..
19:44:55 <smooge> My plan is to have apprentices rewrite sections of them into asciidoc versus docbook and update to current practices.
19:45:50 <smooge> While not glamorous it is one of the most important skills a sysadmin needs to have. Being able to document and explain what is going on.
19:45:58 <smooge> Ok last itme
19:46:05 <smooge> #topic Open Floor
19:46:21 <smooge> any one have items for the open floor?
19:46:28 <anshprat> croberts shoule be able to guide other interns on csi I guess..
19:47:21 <smooge> that is a good idea. And something I need to write an email about.
19:47:23 <smooge> Thank you
19:47:36 <croberts> anshprat: csi
19:47:47 <smooge> Ok I am going to close in 3 minutes if there isn't anything.
19:48:28 <anshprat> one unrelated thing - python-lamson needs to be dropped I guess due to license change..
19:48:29 <frankieonuonga> yes , we really need to encourage people to  know more about other languages in genral. i realized when i tried to suggest something on java..it was evident most of the community was stuck to python.  not a bad thing..but we really need to open minds up
19:48:34 <anshprat> where do I follow that up?
19:48:59 <frankieonuonga> and I do not mean this in a negative way
19:49:01 <croberts> ansphrat? I should guide interns? I am an intern to Fi-apprentice myself
19:49:15 <smooge> anshprat, I would contact legal@fedoraproject.org
19:49:19 <smooge> and the maintainer
19:49:41 <anshprat> croberts: you have experience with twiki and docs guidelines I guess, that's why I said..
19:50:08 <anshprat> smooge: legal part is done, guess will check the maintainer, thanks
19:50:27 <smooge> frankieonuonga, the main reason for our reluctance on other languages is that they all good and fine when there are people committed to staying at the project for long term (which usually means employed to work on it).
19:50:49 <frankieonuonga> oh i see
19:50:57 <smooge> When that doesn't happen we end up with zombie software which the next "I know X" person says is the worst code they have ever seen and want to replace with something new
19:51:11 * anshprat agrees with smooge
19:51:19 <frankieonuonga> wow. ok then makes sense
19:51:29 <frankieonuonga> incase i ever leave you are not left stranded
19:51:34 <frankieonuonga> ok sounds good
19:51:48 <smooge> I don't mind java if we have someone who is tasked with be the java person so that they can guide others in it.
19:52:08 <frankieonuonga> i can take it off
19:52:12 <frankieonuonga> not a problem
19:52:26 <frankieonuonga> but as you said it might need support from someone inside
19:52:27 <smooge> frankieonuonga, yeah.. we had a bunch of different little projects with special languages and people wanting us to deploy scala or ruby but not on rails
19:53:04 <smooge> I don't see a problem in testing what we are working on if we can drive towards having some java people really help out.
19:53:37 <smooge> But in any software there is always a social side that has to be solved as much as the technical side :)
19:53:37 <mirek_> I have one question
19:53:38 <frankieonuonga> well i think this one will have to wait for kevin to get back cause he might think i am going behind his back
19:53:49 <smooge> frankieonuonga, no problem
19:53:50 <mirek_> who is member of cloud team?
19:54:37 <frankieonuonga> mirek_: dgilmore should be from what i recall . but i might be wrong . whats up ?
19:54:47 <frankieonuonga> maybe i can help
19:54:53 <dgilmore> mirek_: whats up?
19:55:01 <mirek_> I just have few question, some of them I sent to mailing list
19:55:17 <mirek_> dgilmore: is arm in fedora cloud?
19:55:20 <dgilmore> mirek_: okay
19:55:23 <dgilmore> mirek_: no
19:55:39 <dgilmore> mirek_: not sure its really possible
19:55:39 <mirek_> dgilmore: and is possible to get arm as emulated on x86?
19:55:51 <dgilmore> mirek_: need to look into it though
19:56:02 <dgilmore> mirek_: not easily
19:56:11 <mirek_> dgilmore: and second question - how can i edit existing AMI or upload new one?
19:56:22 <mirek_> dgilmore: ok
19:56:22 <dgilmore> mirek_: f20 is easier but still difficult
19:56:41 <dgilmore> mirek_: an admin can upload new images
19:57:06 <mirek_> because I need to edit AMI used for builder for copr
19:57:33 <smooge> ok I need to close out in 1 minute for the next meeting
19:57:43 <dgilmore> mirek_: okay
19:57:44 <smooge> mirek_, dgilmore could you work on #fedora-admin
19:57:49 <frankieonuonga> ok thanks guys
19:57:57 <dgilmore> yeah
19:57:57 <smooge> Thanks for coming everyone. Do continue in #fedora-admin, #fedora-apps and #fedora-noc
19:57:57 <frankieonuonga> thanks for coming by .
19:58:00 <mirek_> smooge: will do :)
19:58:01 <frankieonuonga> went smooth
19:58:01 <anshprat> thanks all
19:58:08 <smooge> #endmeeting