18:00:05 <nirik> #startmeeting Infrastructure (2015-02-26)
18:00:05 <zodbot> Meeting started Thu Feb 26 18:00:05 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:05 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:06 <nirik> #meetingname infrastructure
18:00:06 <zodbot> The meeting name has been set to 'infrastructure'
18:00:06 <nirik> #topic aloha
18:00:06 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:06 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:22 <smooge> morning
18:00:35 <puiterwijk> hi. I'm here, but also in another meeting
18:00:36 * danielbruno is here
18:00:38 <janeznemanic> hi
18:00:38 * sborza waves hello
18:00:39 * lmacken 
18:00:52 * pingou here
18:00:57 <kushal> I am here
18:01:02 <kushal> though sleeping I guess
18:01:16 <nirik> :)
18:01:37 * relrod here
18:01:48 * tflink is here
18:02:16 <nirik> morning everyone. ;)
18:02:21 <threebean> hey hey
18:02:25 <nirik> #topic New folks introductions and Apprentice tasks
18:02:32 <nirik> any new folks like to introduce themselves?
18:02:38 <nirik> or apprentices with questions or comments?
18:03:36 <kushalk124> Hey, I have started with some packaging, and looking for a reviewer and sponsor for my first package.
18:03:47 <kushalk124> And I would like to be added to the apprentice group again :)
18:03:55 <nirik> kushalk124: cool. have a link? sure, we can add you...
18:04:04 <kushalk124> Also, I would like to contribute to some python apps.
18:04:24 <kushalk124> Hey nirik, https://bugzilla.redhat.com/show_bug.cgi?id=1194576
18:04:29 <nirik> excellent.
18:04:38 <kushalk124> :)
18:04:54 <danofsatx> good day, folks.
18:05:02 <nirik> kushalk124: see me in #fedora-admin after the meeting and I'll get you added...
18:05:09 <nirik> morning danofsatx
18:05:17 <kushalk124> nirik, sure
18:05:34 <dgilmore> hey
18:05:40 <nirik> morning dgilmore
18:06:10 <nirik> #topic Applications status / discussion
18:06:20 <nirik> any applications side news this week? or upcoming?
18:06:35 <pingou> #info new nuancier deployed
18:06:40 <pingou> #info new anitya deployed
18:06:56 <nirik> isn't there a nuancier election going on now?
18:06:57 <pingou> (with a nice bunch of fixes and new features :))
18:07:02 <pingou> nirik: yes it is :(:
18:07:04 <pingou> :(
18:07:42 <nirik> pingou: ok.
18:07:56 <pingou> I wasn't happy about it, but I learned about the new election when it was already started
18:08:21 <pingou> while there was 6 months work piled for the release :/ so we tested it carefully with gnokii before pushing to prod
18:08:26 <nirik> as long as the update didn't break them...
18:08:32 <pingou> (and so far, no reports so looks like it went ok :))
18:09:04 <nirik> ok, good.
18:09:43 <nirik> ok, anything else? I know since we are in freeze things are a bit quiet. ;)
18:09:56 <threebean> yeah, a new fmn release went out right before freeze
18:09:57 <intrk> did I miss the new comers' introduction?
18:10:09 <threebean> contained changes that made the format of some emails nicer (mostly).
18:10:11 <nirik> intrk: yeah... but feel free to jump in now with one. :)
18:10:19 <threebean> an update of the-new-hotness went out too with similar cosmetic changes.
18:11:46 <nirik> #info fedora notifications and the-new-hotness had releases just before freeze.
18:12:41 <intrk> okay, well... Hi! I'm a fairly young engineer, 25yo. Started at the Helpdesk, went through AV/Security support tech job and now I'm a Network Security technical support engineer. Got my RHCSA last year but since I work mostly on firewalls at the moment I want to have some more experience with Linux and thought this might be the place.
18:12:54 <nirik> intrk: cool. welcome. ;)
18:13:17 <nirik> I guess from that background you are more interested in sysadmin type things? or are you also interested in application devel stuff?
18:13:53 <intrk> Both to be honest, however, I haven't dabbled in programming too much
18:14:31 <intrk> I think I can read basic code and have passed an introductory course for Python and would love to get my feet with that also
18:14:32 <nirik> cool. :) We have both....
18:15:03 <nirik> see me in #fedora-admin after the meeting and I can get you added to our apprentice group... and see #fedora-apps for the application side of things.
18:15:09 <nirik> and again welcome. ;)
18:15:30 <nirik> #topic Sysadmin status / discussion
18:15:40 <intrk> :) superb, thank you
18:15:50 <nirik> lets see... I wish I could say it was a quiet week with the freeze and all, but it's not really been. ;)
18:16:13 <nirik> we have a possibly ongoing issue with out squid proxy. (I'll explain that more in a bit here)
18:16:53 <nirik> I did manage to clean up some more playbooks monday... our ansible check/diff output is getting lower.
18:17:06 <nirik> smooge reinstalled osuosl02 (one less machine in puppet)
18:17:13 <pingou> cool
18:17:19 <pingou> how many left?
18:17:27 <nirik> (after I live migrated guests off it.)
18:17:38 <nirik> For the record, virsh migrate is pretty awesome. ;)
18:17:59 <smooge> I reinstalled sb10 but it crashed out and we are in a email trail of dead letters on getting it back
18:18:07 <nirik> 25 hosts left in puppet.
18:18:24 <pingou> nirik: any big one remaining in there?
18:19:05 <nirik> pingou: most of the ones left will be tricky. I sent a list to the list.
18:19:16 <pingou> oh :(
18:20:22 <nirik> I also looked into torrent software. There's no good news there really. Opentracker (what we use now) has some new commits, but nothing that seems very useful.
18:21:17 <nirik> I did have the thought to see if we could switch from offering people torrent files to offering them magnet: uris... in that you can list hosts you can download from, so we could list our download.fedoraproject.org, etc.
18:22:05 <nirik> I'll look at proposing something around that sometime. ;)
18:22:08 <nirik> lets see...
18:22:27 <nirik> #topic nagios/alerts recap
18:22:27 <nirik> http://ur1.ca/jr7j4
18:22:28 <pingou> googling how to make magnet uri isn't really helpful :/
18:22:49 <nirik> pingou: http://en.wikipedia.org/wiki/Magnet_uri
18:23:06 <pingou> better than google :D
18:23:17 <nirik> so basically that would allow us to not seed. We would just track and get some people to download the images from https first and seed.
18:23:57 <nirik> oh, another thought I had was to make another torrent server so we could have both listed in the magnet link and can reboot one anytime we need to without outage...
18:24:55 <nirik> anyhow, nagios has been somewhat less noisy...
18:25:14 <nirik> I see proxy07 is high on the list, but I don't think any of those caused actual notifications
18:26:03 <nirik> #topic Upcoming Tasks/Items
18:26:03 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:26:12 <nirik> anything upcoming anyone would like to schedule or note?
18:26:56 <nirik> ok, I have 2 items now...
18:27:00 <nirik> #topic Meeting process
18:27:28 <nirik> so, we have had this same meeting process for a long time, I read something the other day (which I cannot for the life of me find now) that had a interesting process...
18:27:37 <nirik> basically the way it worked was:
18:27:51 <nirik> * Open a shared document (for us this would be on gobby or something)
18:28:25 <nirik> * Over the next week people add things to it. There's a 'status/info' section, there's a 'discuss' section and a 'todo/actions'
18:28:44 <nirik> * next meeting people read that and we don't talk about status/info stuff... unless it needs discussion.
18:29:07 <nirik> then we discuss the things on the list to discuss, and then make todos/actions which get added to the next shared doc
18:29:35 <nirik> So, just a thought... I might send this to the list first and get more feedback before we change anything.
18:29:54 <nirik> but I wanted to toss it out here too. ;)
18:30:09 <threebean> interesting
18:30:26 <threebean> yeah, the meetings take on a kind of pre-fab feel since the agenda is the same each week.
18:30:46 <intrk> We do something similar at my current job with a share OneNote document. It works out fairly well.
18:30:48 <nirik> yeah, and we all forget what we worked on really.
18:30:53 <threebean> yeah :P
18:31:03 <nirik> if we add stuff as we do it/think of it... it makes it easier.
18:31:10 <lmacken> I like it. The zodbot output is nice, but having a rw doc could be more collaborative if people have a week to poke at it
18:31:13 <tflink> interestingly enough, we've been talking about something similar for qadevel meetings
18:31:31 <nirik> tflink: cool. did you read something similar to this process recently?
18:31:42 <nirik> it's driving me crazy that I cannot find the blog post or whatever I read. ;(
18:31:52 <tflink> nirik: no, just trying to figure out how to make the meetings go faster
18:32:01 * tflink wasn't the person who suggested the shared document
18:32:27 <intrk> https://sprint.ly/blog/kanban/ ?
18:32:29 * nirik wonders if it could help blocker meetings too.
18:32:47 <tflink> we decided to try suggesting prepared #infos first because it's less complex
18:32:48 <nirik> nope, that wasn't it. :( oh well.
18:33:02 <tflink> nirik: we've wanted to support async blocker debate for a long time
18:33:04 * oddshocks pops in
18:33:12 <nirik> so, should we try this for next week? or discuss on list first and try week after?
18:33:58 * tflink is fine with either but suspects that there may not be much feedback from a list discussion
18:33:59 <nirik> I think having status/info stuff as #info in the doc would be nice... could cut and paste it into the meeting
18:34:25 <nirik> tflink: true. Most people on irc meetings are... in the irc meeting. ;)
18:34:50 <nirik> ok, so let me do this: after the meeting I will make a doc, and send info about it to the list...
18:34:57 <nirik> then we can see what we can do with it next week. ;)
18:35:00 <threebean> cool :)
18:35:11 <nirik> #info will try adding a shared doc and see if it makes our meetings more helpfull.
18:35:21 <nirik> #action nirik to make doc and mail list.
18:35:33 <nirik> So, the other thing I wanted to go over was:
18:35:45 <nirik> #topic Learn about an app: squid / kojipkgs
18:35:50 <threebean> oooo
18:36:01 <nirik> So, we have currently having some issues with squid. So, I thought I would describe how it's setup
18:36:13 <nirik> and how we might do it better or replace it or find out how to fix it.
18:36:42 <nirik> so, we have a machine that is 'kojipkgs01' internally, and kojipkgs.fedoraproject.org externally.
18:36:51 <nirik> It runs apache/httpd on port 8080 locally only.
18:36:59 <nirik> It has a nfs mount to the koji packages store.
18:37:13 <nirik> it runs squid listening on port 80 and 443.
18:37:36 <nirik> squid has it setup that the httpd on port 8080 is a 'parent' cache, so it queries it for the actual files
18:37:56 <nirik> then squid caches in memory or disk for requests.
18:38:07 <nirik> This worked without problem for a long time.
18:38:31 <nirik> Then recently, I nuked the old rhel6 version and replaced it with a shiny rhel7 one.
18:38:52 <nirik> I gave the new one tons and tons of memory with the idea it could memory cache most things.
18:39:08 <nirik> But we are running into an issue where after a few days/week it just gets really really slow.
18:39:18 <nirik> restarting it 'fixes' it
18:39:27 <pingou> takes to long to search in its cache?
18:39:33 <nirik> I adjusted the memory cache down today to see if that would help.
18:39:51 <nirik> it's not clear. There's 0 errors in squid logs or messages or dmesg on the host.
18:40:16 <nirik> squid is also running in it's default mode of 1 process.
18:40:29 <nirik> squid has a smp mode, but I cannot seem to get it to function at all.
18:41:14 <nirik> Some ideas: make a kojipgks02 setup the same way and make kojipkgs a round robin to both... that way at least when we restart builds don't die.
18:41:25 <nirik> Replace with varnish? but not sure how that would work.
18:41:33 <dgilmore> nirik: maybe we could use active/active
18:41:35 <nirik> Replace with memcached? but again not sure how that would work
18:41:49 <dgilmore> replace with mod_cache
18:42:08 <nirik> I think I looked at that and couldn't get it working, but I could try again.
18:42:15 <dgilmore> replace with something else
18:42:45 <smooge> 3) Run the RHEL-6 version on the RHEL-7 system and see if it is a kernel/architecture problem?
18:43:14 <dgilmore> run a Fedora 22 version
18:43:23 <dgilmore> or fedora 21
18:43:27 <nirik> I can't get smp mode working here on f21. ;)
18:43:31 <smooge> Or see if we built a RHEL-6 version with equivalent memory if it still happened
18:43:39 <dgilmore> maybe its a bug in rhel 7's squid that is fixed in fedora
18:43:52 <nirik> I still have the old rhel6 disk, but that is really not something I want to go back to.
18:44:06 <nirik> I can see if there's any differences.
18:44:35 <nirik> there's a big version difference.
18:45:16 <nirik> squid-3.3.8-12.el7_0.x86_64 vs squid-3.4.9-3.fc22.x86_64
18:45:48 <dgilmore> nirik: maybe a quick test is rebuild fedora's squid for rhel7
18:45:48 <nirik> ok, some things to try. ;)
18:46:06 <nirik> sure, although I hate to have to maintain it... for security updates forever
18:46:34 <dgilmore> indeed
18:46:47 <dgilmore> or we run Fedora server
18:47:16 <nirik> perhaps.
18:47:22 <nirik> #topic Open Floor
18:47:30 <nirik> anyone have any items for open floor?
18:47:36 <nirik> questions, comments, ideas?
18:47:51 <pingou> oh
18:48:06 <pingou> I was wondering if there would some more people interested in testing progit
18:48:27 <pingou> afaics it starts behaving nicely
18:48:50 <pingou> UI could be improved, but functionality is there
18:49:41 <nirik> cool. Perhaps a call to list?
18:49:44 <smooge> how would people test it?
18:49:51 <pingou> at one point I'll start looking at packaging it and writing some doc on how to deploy it
18:50:04 <pingou> smooge: for the moment the easiest is to use the dev instance: http://209.132.184.222
18:50:17 <pingou> make a project, fork one, check the pull-requests, issues
18:50:24 <pingou> see what needs improvement
18:50:49 <pingou> and flood me with requests and bug report :]
18:51:19 <smooge> okie dokie
18:52:18 <nirik> pingou: or perhaps devel list at this point?
18:52:24 <pingou> nirik: sure thing
18:52:29 <nirik> cool.
18:52:33 <pingou> thanks :)
18:52:40 <nirik> #info everyone go test progit. :) http://209.132.184.222
18:52:49 <nirik> ok, if nothing else will close out in a minute.
18:52:52 <pingou> I should blog about it as well :)
18:54:03 <nirik> yes. :)
18:54:09 <nirik> ok, thanks for coming everyone!
18:54:12 <nirik> #endmeeting