18:01:24 <smooge> #startmeeting Infrastructure (2015-10-22)
18:01:24 <zodbot> Meeting started Thu Oct 22 18:01:24 2015 UTC.  The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:01:24 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:01:24 <smooge> #meetingname infrastructure
18:01:24 <zodbot> The meeting name has been set to 'infrastructure'
18:01:24 <smooge> #topic aloha
18:01:24 <smooge> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson
18:01:24 <smooge> #topic New folks introductions / Apprentice feedback
18:01:24 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:01:30 <smooge> Hello All
18:01:31 * pingou here
18:01:37 <puiterwijk> I'm here, but also in another meeting
18:01:40 * danielbruno here
18:01:49 <smooge> sorry for my 1 minute delay.. was writing out the section for later in the meeting
18:02:10 <smooge> How are things for everyone? Do we have any new people or apprentice questions?
18:02:33 <danielbruno> Yeah, I have a question
18:02:42 <smooge> hi danielbruno
18:02:45 * sayan here
18:02:47 <smooge> what is the question?
18:03:00 <danielbruno> there is some priority in some easyfix ticket to work on?
18:03:38 <danielbruno> I've worked and closed one few days ago, and now I'm looking for another task to work :)
18:04:03 <smooge> hmmm
18:04:07 <jflory7> Hi all, my name is Justin - just wanted to introduce myself now. I am a student at the Rochester Institute of Technology majoring in Networking & Systems Administration. I am interested in looking into an apprenticeship in the future once I have a little more on my hands. For now, I am just an interested observer. :)
18:04:15 <jflory7> .hello jflory7
18:04:17 <zodbot> jflory7: jflory7 'Justin W. Flory' <me@justinwflory.com>
18:04:24 * lnxslck here
18:04:28 <linuxmodder> .fas corey84
18:04:28 <zodbot> linuxmodder: corey84 'Corey Sheldon' <sheldon.corey@gmail.com>
18:05:07 <smooge> danielbruno, I believe that there is some priority in them, but at the moment I can't think what it is.
18:05:51 <smooge> nirik, who usually knows all and sees all is on vacation and I need to make sure I have the right info before I answer
18:05:59 <linuxmodder> question:  any  folks from the server config side  able to spare some  time / knowledge over the next few weeks  ?   doing more  cross OS server stuff at a local np org and admittedly  ou tof  league
18:06:00 <smooge> is it ok if I email you after the meeting?
18:06:01 <pingou> welcome jflory7
18:06:28 <jflory7> Thanks!
18:06:30 <danielbruno> smooge: can we talk about later?
18:06:31 <linuxmodder> jflory7,  welcoem
18:06:43 <smooge> linuxmodder, possibly. I can talk more after the meeting
18:06:52 <smooge> yes we can danielbruno
18:07:13 <danielbruno> smooge: cool, I'll ping you on #fedora-admin, ok?
18:07:16 <linuxmodder> kk worst  case ping or email me  (may be  afk  most of  afternoon after mtg)
18:07:17 <smooge> hi jflory7 that is a well known school. I am sure we can work out something as time goes on
18:07:18 <linuxmodder> thanks
18:07:44 <smooge> danielbruno, sounds good
18:07:53 <linuxmodder> cydrobolt, ^^
18:08:00 <jflory7> Sounds good, I am looking forward to it!
18:08:06 <linuxmodder> jflory7,  cydrobolt  is in your locale too
18:08:08 <smooge> next on the meeting....
18:08:13 <smooge> #topic announcements and information
18:08:13 <smooge> #info Still in f23 final freeze - everyone
18:08:13 <smooge> #info buildvm-01.stg is now fedora 23, please test - kevin
18:08:13 <smooge> #info Mirrorlist servers now use threads and more performant - patrick
18:08:13 <smooge> #info Fixes for some openvpn failures now live - patrick
18:08:14 <smooge> #info new shiny app: mdapi: http://209.132.184.236/ exposes the repo metadata over a simple API - pingou
18:08:16 <smooge> #info new release of fedora-packages - threebean
18:08:18 <smooge> #info new release of fedora-tagger - threebean
18:08:21 * relrod here late, sorry
18:08:34 <smooge> So we are still in Final freeze and will be til at least November 3rd
18:08:46 <smooge> The product will slip a week due to some end of the run blockers
18:08:56 <smooge> So no Halloween release ... again ;(
18:09:14 <pingou> arf
18:09:43 <Southern_Gentlem> closer than it was
18:09:44 <smooge> patrick has done a lot of work on mirrorlist to get it to behave better under load
18:09:54 <smooge> agreed Southern_Gentlem
18:10:51 <smooge> we have been having problems with mirror system going all sideways at various times of the day (usually when a good many of people in Europe reach 0900 and another set when Americans reach about that time)
18:11:10 <smooge> patrick and adrian debugged the problem and came up with a good soltion.
18:11:31 <linuxmodder> it was all sorts of  sideways ~ 9am UTC today even
18:11:46 <smooge> while he was working on that he was getting hit with an openvpn problem where it decides that instead of doing a reconnect like it is supposed to so.. it goes into lala land
18:12:04 <puiterwijk> Today 9am UTC was something else, but I'm still analyzing all log files to find outthe exact cause
18:12:09 <smooge> that was the second problem I think
18:12:20 <smooge> but puiterwijk is still working on the analysis
18:12:26 <puiterwijk> And yes, there were also issues with openvpn that I've dianosed and am working on a fix
18:12:42 <linuxmodder> puiterwijk,  I am up and about at  odd hours feel free to ping me if you need any help
18:12:48 <smooge> pingou and threebean do you have time to talk about your items?
18:12:56 * pingou can
18:13:04 <puiterwijk> linuxmodder: me too, but will do. Unfortunately there's not much we can do until I finalized analysis
18:13:36 <linuxmodder> I use openvpn on several providers for  other things if that needs  specific testing
18:13:51 <pingou> http://209.132.184.236 aka mdapi is small api exposing the content of the metadata of our repositories
18:14:01 * threebean is here
18:14:03 <puiterwijk> Well, I found the exact cause in a race. I just need to develop the patch fully and send it upstream
18:14:07 <pingou> for examples: http://209.132.184.236/rawhide/pkg/guake http://209.132.184.236/rawhide/changelog/guake  http://209.132.184.236/rawhide/filelist/guake
18:14:43 <pingou> it's using asyncio (is py3), so holds the load quite well and so far it's been quite fast :)
18:15:15 <linuxmodder> oddly  one of my  ovpns is  209.x.y.z lol
18:15:21 <threebean> I mentioned it on the list, but the new fedora-packages release contained a lot of fixes that had accumulated over the last year.  we're actively developing it now and we wanted to get all the un-deployed features out the door so we could have a clean slate.
18:15:31 <threebean> in some of the upcoming changes, we're going to try using pingou's new mdapi service.
18:15:59 <smooge> pingou, that is useful.. I actually need that a lot
18:16:57 <smooge> ok for the general slate.. is fedora-packages a service, an rpm, a toolkit?
18:17:25 <pingou> I've been testing it with 1000 queries by batch of 100, the results are quite satisfying
18:19:36 <puiterwijk> smooge: fedora-packages is https://apps.fedoraproject.org/packages/
18:20:00 <puiterwijk> It's an aggregator for several pieces of information (pkgdb ownership, bugzilla bugs, bodhi versions, etc)
18:20:06 <smooge> cool
18:20:30 <puiterwijk> And currently is reasonably broken ;(
18:20:33 <smooge> there are similar named rpms toolkits etc so I like to be clear when I am trying to put stuff in my long term memory (eg the logs)
18:20:48 <smooge> pingou, cool
18:21:05 <puiterwijk> Well, I think threebean fixed the bodhi2 compat issues, but it's still not very good with its cache handling etc
18:22:15 <smooge> ok thanks threebean and pingou
18:22:32 <smooge> any other announcements? if not I will move to the next topic
18:23:14 <smooge> #topic Learn about: Budgets - smooge
18:23:51 <smooge> OK now for one of the most boring but neccessary parts of systems administration. More boring than backups but more important (as you need the money to have the backups)
18:24:40 <smooge> Red Hat is the primary sponsor for the hardware we use to bring the Fedora project to everyone. We have multiple other partners who give us proxies but all the build systems, and core download systems are run on systems paid for by Red Hat
18:25:33 <smooge> Red Hat runs on a yearly budget scheme which works from March to end off the next February, with the budget year called by the time it would end in. We are currently in FY2016 and I am now working on the budget for FY2017
18:26:48 <smooge> Because Red Hat is a reasonably large company it takes time to get budgets dealt with so I end up in October to try and get a budget request for January 1st for purchases I might not get until 2017.
18:27:19 <smooge> Which can be very tricky for various groups who are used to coming up with and idea and getting hardware for it the next day :).
18:28:19 <smooge> So currently I am asking for various groups to come up with an idea of any project needs they see in the next year. This can be "we are expanding into MIPS128 and will need build servers for that" to "we are going to host a github replacement and will need storage for that."
18:28:55 <smooge> I then try to get some ideas on hardware costs and put together the initial request with the idea that if approved it will show up late the next eyar.
18:29:27 <smooge> I also spend a lot of time doing inventory of existing hardware, donataed hardware, etc etc to see which hardware is no longer worth keeping under extended warranty and need replacement.
18:30:10 <smooge> When I first started this we tried to get systems out of the way in 4 years but Moore's law is pretty much slowed down so a 7 year old system can do most of the work of a new one. It just costs more in replacemnts.
18:31:14 <smooge> So if you are working on a project and will be looking to expand etc it would be good to let me know now so I can better have the resources when you need it :).
18:31:36 <smooge> This is also the type of lecture you get when no one offers to learn and teach about a service already in Fedora.
18:32:04 <smooge> So please please please find something and volunteer next meeting. I don't want to pull out the acturial tables on hardware lifetimes
18:32:14 <smooge> Any questions?
18:32:20 <lnxslck> not really
18:32:21 <puiterwijk> smooge: would it be an idea to ask the infra list for people who are thinking about big new projects?
18:32:53 <smooge> I thought I had but I will do so as I must have misrememebred
18:33:25 <pingou> smooge: I guess it also depends on how much is available
18:33:48 <pingou> like if we have 50To of storage un-used, I guess a big project or two should be fine :)
18:34:27 <smooge> well that is where the dance macabre comes in. I put in a request. It gets looked at, mixed with other projects and then paired down to what can be afforded that year. We then go back to see what that will affect and what needs to be redone.
18:34:44 <smooge> Wet, lather, rinse, repeat (until ship date)
18:34:47 <pingou> fun :/
18:35:07 <aikidouke> smooge - is this kind of a corporate ask for more than what you need so that you can at least get what you have to have?
18:35:37 <aikidouke> also, is there any kind of project template/budget request?
18:36:50 <smooge> aikidouke, they actually check against what was requested in past years and what the other projects need. So when I ask for too much it gets more cut out.
18:36:53 <aikidouke> sorry if that first q comes off wrong, I've worked for the same corp for the last 18 yrs and have no idea what RH culture is like
18:37:44 <aikidouke> ok, good to know. I just had an idea, but need some time to vet it...i will work on it and get with you sep
18:38:03 <smooge> so I find that working out a reasonable request I can expect 5% wiggle room. If I ask for the moon I can expect a 60% and then negotiate back up to what I needed
18:38:21 <smooge> I am always looking for good template/budget requests
18:38:32 <aikidouke> yeah, that's diff that what I'm used to.
18:38:35 <puiterwijk> aikidouke: note that for most projects you don't need extra hardware. Only for very big projects do we actually need extra hardware
18:39:37 <smooge> puiterwijk, well even small projects might need hardware overall. If we have 8->20 virtual servers each with 8 GB of ram each.. we may need 1 more server.
18:39:39 <aikidouke> puiterwijk: ok, I might just email you, smooge, and nirik offline once I get some idea together
18:39:52 <smooge> that would be great. I can be reached at
18:39:54 <smooge> .fas smooge
18:39:55 <zodbot> smooge: smooge 'Stephen J Smoogen' <smooge@gmail.com> - ssmoogen 'Steven Smoogen' <smooge@smoogespace.com>
18:40:04 <smooge> huh.
18:40:10 <puiterwijk> .hellomynameis smooge
18:40:11 <zodbot> puiterwijk: smooge 'Stephen J Smoogen' <smooge@gmail.com>
18:40:13 <smooge> I wonder who that other guy is  :)
18:40:16 <puiterwijk> smooge: ^ I think you meant that one
18:40:49 <smooge> ok so anyway that is just the surface layer of minutea(sp) that is my life. I wear a tie, so you don't have to.
18:41:13 <smooge> #topic Open Floor (or shut up smooge shut uppppp)
18:41:14 <puiterwijk> smooge++ thanks for handling that part!
18:41:35 <pcreech> thx smooge, informative!
18:41:35 <dgilmore> smooge: I wants the hardware in the racks
18:42:15 <pingou> smooge++ thanks for the tie!
18:42:15 <zodbot> pingou: Karma for smooge changed to 2 (for the f23 release cycle):  https://badges.fedoraproject.org/tags/cookie/any
18:42:19 * smooge goes to work out floor space planning as we have N servers and Y rackspace and power. Next week in infra
18:42:22 <lmacken> gotta wear a tie when doing budgets
18:42:26 <aikidouke> I haven't checked yet today - but if you haven't responded to my apprentice workday e mail yet, i would appreciate any feedback
18:42:53 <smooge> aikidouke, do you have a link in case it got eaten by mail grue?
18:43:10 <aikidouke> uno momento por favor
18:44:16 <aikidouke> https://lists.fedoraproject.org/archives/list/infrastructure%40lists.fedoraproject.org/thread/2VMCDOBS5K7PTEIGXOCKJXVZFMJSUCK7/
18:44:28 <aikidouke> gah sorry bout that weechat and irssi users
18:45:18 <smooge> cool thank you. PLease do read through this and respond
18:45:26 <smooge> ok anything else?
18:45:38 <smooge> I will close out the meeting in 1 minute otherwise.
18:47:02 <smooge> #endmeeting