18:00:33 <nirik> #startmeeting Infrastructure (2015-01-08)
18:00:33 <zodbot> Meeting started Thu Jan  8 18:00:33 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:33 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:33 <nirik> #meetingname infrastructure
18:00:33 <nirik> #topic aloha
18:00:34 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:34 <zodbot> The meeting name has been set to 'infrastructure'
18:00:34 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:38 * pingou 
18:00:42 * webpigeon waves
18:00:46 * danielbruno here
18:01:04 * puiterwijk is here
18:01:07 <smooge> here
18:01:13 * lmacken 
18:01:16 <nirik> morning everyone.
18:01:22 * nirik will wait a few min for folks to wander in
18:01:30 <kushalk124> hello :)
18:01:32 * sborza is here
18:02:00 <webpigeon> it's morning o.O
18:02:01 * threebean threebean
18:02:14 <webpigeon> threebean: that makes sixbean
18:02:23 <mpduty> hi
18:02:38 <threebean> webpigeon: or worse, nine! :|
18:03:14 <nirik> heh.
18:03:18 <nirik> #topic New folks introductions and Apprentice tasks
18:03:25 <nirik> any new folks like to introduce themselves?
18:03:34 <nirik> or apprentices with questions, suggestions, comments?
18:03:46 <mhurron> ya hi, I'm Sean, new here
18:04:05 <fale> I'm Fabio A Locati, an Italian freelance sysop, dba and I'm new here :)
18:04:14 <threebean> fale: hi :)
18:04:23 <nirik> welcome fale and mhurron. :)
18:04:24 <mhurron> I'm a sysadmin, know a little python, looking to help out
18:04:24 * threebean waves to mhurron
18:04:47 <nirik> question for both of you: are you more interested in sysadmin work? or application development? or both?
18:05:33 <mhurron> I know more sysadmin, I don't know if I can be useful for development, So more admin I suppose
18:05:35 <fale> nirik: a little bit more on sysadmin because I already have competence in the area so it would be easier to start from there for me
18:06:00 <nirik> ok, cool.
18:06:18 <nirik> If you are interested I can add you to our apprentice group after the meeting... see me in #fedora-admin and I can get you setup.
18:06:26 <fale> nirik: thankyou
18:06:34 <kushalk124> Hey, I am Kushal currently located in Germany, I plan to help in application development in python !
18:07:06 <nirik> kushalk124: welcome. Do see the kind folks in #fedora-apps after the meeting...
18:07:23 <kushal> kushalk124, also please say your full name or else people will be confused.
18:07:31 <kushalk124> nirik, Thanks :) I would definitely do so,
18:07:46 <kushalk124> kushal, Oh yes, Kushal Khandelwal  here
18:08:06 <sborza> nirik: not sure if I'm already a member of the apprentice group, but would like to be added
18:08:17 <nirik> kushal: yeah... :)
18:08:29 <sborza> I've been out for a few weeks (new baby), but am now back and ready to contribute
18:08:39 <nirik> sborza: can check. ;) see me after the meeting in #fedora-admin and I can see...
18:08:49 <sborza> nirik: will do, thanks!
18:09:07 <pingou> sborza: and congratulation
18:09:09 <pingou> s
18:09:15 <pingou> :)
18:09:25 <nirik> cool. Any other new folks or apprentices with questions?
18:09:31 <sborza> pingou: thank you!
18:09:55 <nirik> #topic Applications status / discussion
18:10:08 <nirik> any applications side news or discussion this week? ;)
18:10:17 * pingou has been working on unit-tests for mirrormanager2
18:10:30 <nirik> I'll note that I sent out a email asking for status on various stuff... please do read it and reply if you have info on those items.
18:10:37 * pingou should
18:10:40 <threebean> pingou: thanks for that :)
18:11:21 <lmacken> I pushed out a new bodhi release to prod this week that contains bochecha's patches to work around the weird \n behavior that started happening after our postgres upgrade
18:11:28 <threebean> I'm aiming to get an FMN release out in the next week with lots of little enhancements and bugfixes all over.  -- https://apps.fedoraproject.org/notifications
18:11:32 <pingou> threebean: got a couple of (simple) bugs so far :)
18:11:42 <lmacken> hopefully just a weird SQLObject issue that will be resolved with bodhi2 + SQLAlchemy
18:12:07 <nirik> lmacken: yeah. Oh, have you had a chance to try and setup bodhi2 on bodhi02.stg ? ;)
18:12:19 <nirik> #info lots of work on mirrormanager 2 unit tests.
18:12:31 <nirik> #info bodhi1 update out in production to fix db issue with newlines
18:12:42 <lmacken> threebean: will it support anitya filters?
18:12:44 <nirik> #info work on FMN to get it ready for prod
18:13:06 <lmacken> nirik: not yet, that's the goal for the next couple of weeks
18:13:08 <threebean> lmacken: sure will
18:13:14 <lmacken> threebean: excellent :)
18:14:14 <lmacken> having irc notifications for all upstream releases for all of my packages will be great
18:14:38 * nirik nods
18:14:45 * pingou likes
18:15:18 <nirik> we do have the f22 schedule from fesco now, so we will have to figure out when we want to land new apps/etc in production...
18:15:26 <nirik> and avoid freezes, etc
18:16:38 <nirik> anything else application wise?
18:16:41 <threebean> https://fedoraproject.org/wiki/Releases/22/Schedule
18:16:52 <lmacken> bodhi atomic composes have been going smoothly :)
18:16:52 <nirik> yep
18:17:01 <threebean> wow, alpha freeze starts at end of february?
18:17:13 <threebean> it's always seems like such a short window inbetween.
18:17:14 <nirik> lmacken: excellent. Wonder how many people are consuming them... I guess as always we never know. ;)
18:17:24 <nirik> threebean: yeah, it's a short ramp. ;(
18:17:47 <lmacken> nirik: heh, yep
18:18:40 <nirik> #topic Sysadmin status / discussion
18:18:48 <nirik> ok, lets see... on the sysadmin side...
18:19:10 <threebean> ooo
18:19:12 <nirik> we are down to 53 hosts in puppet.
18:19:20 <pingou> \ó/
18:19:22 <nirik> threebean: your proxy01.stg seems to be mostly working well. ;)
18:19:33 <threebean> the proxy layer redone in ansible has been put in place in stg now.  :)
18:19:38 <nirik> I wanted to get puiterwijk to look at the varnish config. I just hacked it up so it would work
18:19:38 <threebean> cool, cool.  :)
18:19:58 <nirik> it also needs selinux work... (the varnish setup, won't start in enforcing)
18:20:31 <threebean> eh, yeah.  we'll definitely want that for the front door.
18:20:57 <nirik> yeah, once thats sorted, I think we could look at making a proxy10 or something somewhere and adding it to the mix...
18:21:01 <threebean> haproxy needed some selinux help too, but that's nailed down now.
18:21:07 <nirik> cool.
18:21:50 <nirik> I think haproxy in stg is all green (aside gallery01.stg, which doesn't matter)
18:22:00 <nirik> https://admin.stg.fedoraproject.org/haproxy/proxy01
18:22:27 <threebean> could do a rube run too -- if anyone has some free time to mess with it.
18:22:31 <nirik> #info proxy01.stg needs a bit more work, but overall is looking great in ansible
18:22:37 <nirik> yeah, great idea
18:22:39 <threebean> maybe an apprentice?
18:22:49 <threebean> https://github.com/fedora-infra/rube
18:22:57 <nirik> sure... ^
18:23:11 <threebean> there's always something new broken with it each time I go back to mess with it.. but it's useful to figure out where we have fires burning we didn't know about.
18:23:18 <lmacken> threebean: yep, always.
18:23:27 <nirik> what happened with running it in jenkins?
18:23:31 <nirik> just too broken from run to run?
18:23:59 <threebean> nirik: yeah, and jenkins itself kept blipping out and I haven't made time to follow up on it.
18:24:05 <lmacken> looks like a virtualenv issue or something
18:24:09 <threebean> (if anyone wants to try running it.. use these instructions..)
18:24:11 <threebean> https://github.com/fedora-infra/rube/tree/develop/rube.fedora#running
18:24:26 <nirik> ok, fair enough
18:24:57 * lmacken got a stack trace when logging in to jenkins :(
18:25:02 <nirik> Also, I have been fighting a curl issue on f21... trying to move our builders to f21 but they sometimes hang in builds. ;(
18:25:05 <lmacken> javax.servlet.ServletException: java.lang.RuntimeException: Failed to serialize hudson.model.User#properties for class hudson.model.User
18:25:06 <webpigeon> oooh dear
18:25:13 <nirik> lmacken: ;(
18:25:16 * pingou ducks
18:25:31 <threebean> rock solid :p
18:25:43 <pingou> gotta love jenkins...
18:25:53 <threebean> I hear about teams doing really incredible stuff with jenkins, but it seems like we just don't have the person-power to keep it afloat.
18:26:06 <nirik> also this month we will be having some storage moving fun... no details or exact time yet, but hopefully this month... we will need an outage to move nfs server mounts.
18:26:26 <nirik> threebean: yeah. Keep wondering if buildbot would do better...
18:26:43 <nirik> #info storage moves likely later this month.
18:26:43 <lmacken> threebean: I think we'd have the same type of issue with elasticsearch
18:26:51 <lmacken> but I really want to use it :P
18:26:54 <nirik> #info smooge and nirik will be in phx2 datacenter in feb
18:27:09 * tflink might be able to help with buildbot
18:27:14 <smooge> hopefully :)
18:27:24 * smooge will be booking tickets later today
18:27:55 <pingou> the best point of Jenkins is its UI
18:28:07 <pingou> stability wise, we're not managing apparently :(
18:28:22 <nirik> there's also that mozilla one... tinderbox? dunno.
18:28:44 <lmacken> let's just run tests in taskotron :)
18:28:49 <pingou> \ó/
18:28:51 <tflink> the new UI for buildbot nine will be much better
18:29:06 <pingou> tempting :)
18:29:38 <nirik> tflink: oh, while you are around: any thoughts about moving taskotron clients and other stuff thats f20 now to f21? or rhel7? or you want to keep it with 20 for now and move at f22?
18:30:01 <tflink> http://nine.buildbot.net/
18:30:41 <tflink> I'd rather keep stuff as is for now - the bits we're currently working on will require some major client changes
18:31:01 <nirik> ok, fair enough. just wanted to mention it. '
18:31:01 <tflink> sorry for being late, still have the pre-DST meeting time in my head
18:32:24 <nirik> #topic nagios/alerts recap
18:32:44 <nirik> I've been trying to fix up things that alert, but hard to say if it's having too much effect. ;)
18:33:18 <nirik> just need to keep plugging away. ;)
18:33:20 <nirik> #topic Upcoming Tasks/Items
18:33:20 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:33:31 <nirik> anything anyone has upcoming that they would like to schedule or discuss?
18:33:56 <pingou> dev-conf early february
18:33:59 <nirik> Do we want to look at a FAD for 2015? or is it too soon to talk about?
18:34:15 <nirik> pingou: you and puiterwijk are going?
18:34:16 <pingou> puiterwijk bochecha dgilmore and pbrobinson will be there
18:34:19 <pingou> and I
18:34:21 <nirik> ah, nice
18:34:29 <pingou> nirik: +1 for FAD
18:34:35 * pingou was thinking rel-eng tooling FAD
18:34:47 <threebean> also, PyCon US in April.
18:34:57 <threebean> pingou: and yeah, rel-eng tooling FAD would be sweet.
18:35:03 * nirik nods.
18:36:04 <nirik> #topic Open Floor
18:36:16 <nirik> anyone have anything for open floor? ideas, comments, suggestions?
18:36:47 * tflink is also going to be at devconf
18:36:53 <smooge> cool
18:36:55 <nirik> nice
18:37:27 * pingou adds tflink to the list :)
18:38:25 <nirik> alright, if nothing else will close things out in a minute...
18:38:32 <smooge> anyone going to be at FOSDEM?
18:38:55 <pingou> smooge: I will
18:39:28 <fale> smooge: I will if I'll be able to fit it with my exam schedule
18:39:47 <smooge> cool
18:40:00 <smooge> ok close it out
18:40:19 <nirik> alright. Thanks for coming everyone.
18:40:22 <nirik> #endmeeting