18:00:53 <nirik> #startmeeting Infrastructure (2015-01-22)
18:00:53 <zodbot> Meeting started Thu Jan 22 18:00:53 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:53 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:53 <nirik> #meetingname infrastructure
18:00:53 <zodbot> The meeting name has been set to 'infrastructure'
18:00:53 <nirik> #topic aloha
18:00:53 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
18:00:53 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean
18:00:57 * pingou here
18:00:58 * puiterwijk is around
18:01:04 * fale here
18:01:07 <janeznemanic> hi
18:01:08 * relrod here
18:01:22 * bochecha is also here (for once)
18:01:42 * sayan here
18:02:26 <nirik> alrighty. welcome everyone
18:02:28 * tflink is here
18:02:40 <nirik> #topic New folks introductions and Apprentice tasks
18:02:49 <nirik> Any new folks like to introduce themselves?
18:02:56 <nirik> or any apprentices with questions or comments?
18:03:00 <andreasch> Hello everyone, my name is Andreas Christoforou and I am working as a system administrator also I am a fedora ambassador and I would like to help and contribute to the Fedora infrastructure.
18:03:04 <dgilmore> hola amigos
18:03:15 <nirik> andreasch: welcome.
18:03:24 <pingou> welcome andreasch
18:03:30 <nirik> are you more interested in sysadmin (seems likely) or application development type tasks?
18:03:33 <pingou> hóla dgilmore
18:03:51 <andreasch> sysadmin;)
18:04:01 <Corey84> not new to group but willing to take on apprenticeship tasks
18:04:24 <Mohamed_Fawzy> hi
18:04:26 <Corey84> both for me sysadmin is strong suit tho
18:05:08 <nirik> welcome to the fun Corey84. ;)
18:05:16 <Mohamed_Fawzy> hello sorry for late. my name is mohammed  i work as linux sysadmin
18:05:23 <Corey84> nirik,  lol
18:05:31 <Mohamed_Fawzy> also I am a fedora ambassador and I would like to help and contribute to the Fedora infrastructure
18:05:34 <nirik> welcome Mohamed_Fawzy.
18:05:41 * sborza is here
18:05:51 <Mohamed_Fawzy> sorry for late
18:06:00 <nirik> Please all of you see me after the meeting in #fedora-admin and I can get you added to the apprentice group and point you the way of outstanding tasks. ;)
18:06:13 <pingou> welcome Mohamed_Fawzy
18:06:36 <Mohamed_Fawzy> Wow good
18:06:40 <Mohamed_Fawzy> i like that
18:07:05 <nirik> ok, any other new folks or apprentices with questions before we move on?
18:07:16 <Corey84> nirik,  seems adam3535  never was able to do o on ambassador side either last mtg possible we can hit both issues after this mtg?
18:07:48 <smooge> apologies.. late
18:07:53 <nirik> Corey84: not sure I understand the issue... but happy to look into it...
18:07:54 * danielbruno is late, but here
18:08:24 <Corey84> he was planning to look into why i had not got added to ambassadors group but was on ML
18:09:01 <nirik> ah, no idea. ambassadors control that...
18:09:12 <nirik> unless there's some technical issue with the account system.
18:09:45 <Corey84> worst case ill tag up with Southern_Gentlem  later
18:09:59 <nirik> sure.
18:10:01 <nirik> #topic Applications status / discussion
18:10:09 <nirik> ok, any applications side news this week? or upcoming?
18:11:12 <sborza> nirik: on the application front re:graphite-web, we're looking into building out the proper spec files + rpms
18:11:23 <nirik> cool.
18:11:30 <sborza> once they're built i'll ping the group on getting it added for testing
18:11:47 <pingou> new pkgdb2 released, major release coming up
18:11:50 <sborza> is there issue that it uses the Apache 2.0 license? don't think so but wanted to confirm
18:12:34 <nirik> sborza: should not be, thats a acceptable license
18:12:48 <pingou> pkgdb-cli release and a new one coming up (related to the future pkgdb2 one)
18:12:57 <pingou> I also updated fedocal
18:13:20 <pingou> all the changes have been announced on the infrastructure list
18:13:39 <nirik> yep. I really appreciate that... nice to see whats changed and when a release comes out.
18:14:02 <nirik> I'd love such for all our apps. ;)
18:14:17 <pingou> I thought the same, if something break everyone knows what changed in the most recent version
18:14:21 <nirik> #info new pkgdb release this week
18:14:30 <nirik> #info new fedocal with some bugfixes
18:14:42 <pingou> and on these words I will keep pushing new releases in the evenings and on fridays :-p
18:14:49 * pingou ducks
18:14:49 <nirik> I know threebean has been working on fixing FMN... it's suffering a backlog again. ;(
18:15:11 <pingou> yeah FMN has an interesting problem: its caching strategy is too good
18:15:39 <Corey84> FMN ?
18:15:52 <Mohamed_Fawzy> i dnt knw or get wht u all talk about
18:15:53 <pingou> https://apps.fedoraproject.org/notifications/
18:16:21 <nirik> It's our notification service.
18:16:23 * threebean here now
18:16:40 <nirik> it listens to messages and then according to a users preference notifies them... via email, irc, etc
18:17:00 <Mohamed_Fawzy> i got that but wht FMN
18:17:08 <pingou> FedMsg Notification
18:17:09 <bochecha> Mohamed_Fawzy, that's what FMN is
18:17:33 <tflink> how's bodhi2 coming along?
18:17:35 <pingou> FMN is the application running at https://apps.fedoraproject.org/notifications/
18:17:37 <Mohamed_Fawzy> good thanks
18:18:03 <pingou> lmacken: update on bodhi2?
18:18:29 <nirik> I made a bodhi02.stg a bit ago... we were going to install bodhi2 on it and get it up in stg so we could see it. ;)
18:18:35 <pingou> I rebuild mirrormanager2 on stg today, the playbook went through nicely and all seems to run, which scares the hell out of me :)
18:18:38 <nirik> but not sure where we are there.
18:18:47 <pingou> rebuilt*
18:19:39 <lmacken> pingou: I'm working on polishing up the masher side of things at the moment
18:19:49 <lmacken> haven't had time to work on deploying/packaging the frontend stuff yet
18:19:52 <nirik> #info mirrormananger2 in staging reinstalled and all looks ok.
18:20:09 <nirik> pingou: whats the next steps for mirrormanager2?
18:20:16 <pingou> nirik: good question
18:20:27 <nirik> lmacken: is that something someone could assist with? it's making fedora packages? or upstream releases?
18:20:41 <pingou> I have no idea really how to test if stg is returning the same thing as prod
18:21:36 <nirik> as far as the end mirrorlists?
18:21:51 <bochecha> pingou, can we all set our yum configs to point to the stg mirror manager ?
18:21:56 <pingou> yes the mirrorlists, the UI itself I expect works
18:22:13 <bochecha> pingou, then we'll see if it at least returns something usable :)
18:22:14 <lmacken> nirik: yeah, I think we need to RPMify some more of the pyramid stack. Will have to touch base with threebean about it since I think he packaged up bodhi2 itself.
18:22:19 <pingou> bochecha: I expect this will always work, as most mirrors are up to date
18:23:09 <nirik> well, it may be then we just need to forge ahead and move things into production and fix any reported issues...
18:23:10 <threebean> yeah, i did lots of the pyramid stack.  some pieces might be missing though (cornice?)
18:23:26 <bochecha> didn't I package cornice during the Denver FAD?
18:23:38 <lmacken> bochecha: I believe so
18:24:04 <pingou> nirik: when would be a good time
18:24:17 <bochecha> pingou, tomorrow evening, around 5:45pm ;)
18:24:35 <pingou> bochecha: oh cool, I was planning on leaving at 5:46 for 2 weeks off :)
18:24:42 <nirik> pingou: so, do we need to do it all at once? or can we do mirrorlists first, then frontend/backend? or is that mixing data from the current v1 one with v2 mirrorlists bad?
18:25:16 <pingou> nirik: I *think* we may be able to move mirrorlist first
18:25:23 <pingou> the pickle hasn't changed normally
18:25:28 <bochecha> threebean, cornice is in: https://admin.fedoraproject.org/pkgdb/package/python-cornice/
18:25:34 <nirik> FYI, for folks who are new... mirrormanager is the thing that keeps track of our mirrors and part of it is what people query with yum to find packages.
18:25:43 <pingou> the mirrorlist itself also barely changed (just some optimization from lmacken )
18:26:06 * lmacken crosses his fingers and hopes those don't cause any regressions
18:26:07 <pingou> (in other words, when mirrormanager breaks, we know it, fast! :D)
18:26:15 <nirik> pingou: but it's available in epel7/rhel7 right? (current ones are 6, which I want to upgrade from)
18:26:17 <pingou> lmacken: well, you have unit-tested it ;-)
18:26:25 <pingou> nirik: they are
18:26:47 <nirik> so I would say, lets try and do one of those, see how it goes, then the rest, then the backend/frontend.
18:26:53 <pingou> mirrorlists.stg is el7
18:27:26 <pingou> nirik: I'm ok with this, just that from next week friday timing will be short w/ FOSDEM and DevConf coming up
18:27:55 <pingou> so all joke aside, we may want to upgrade one mirrorlist next week Monday or so
18:27:57 <nirik> true... can try and do it monday and that gives a week to fix anything we see?
18:28:03 <nirik> right. +1
18:28:05 <pingou> leaving us 3 days before FOSDEM to fix things
18:28:24 <pingou> nirik: and let's announce it here and there :)
18:28:25 <nirik> I suspect we will see issues pretty quickly...
18:28:44 <pingou> not too many I hope
18:28:59 <nirik> yeah, I hope it will be smooth. ;)
18:29:08 <pingou> fingers crossed ;)
18:29:22 <nirik> #info will look at converting one mirrorlist server from mirrormanager1 to 2 on monday and add it to the rotation.
18:29:48 <nirik> ok, any other application news?
18:30:30 * pingou blogged a little about the new branch process coming up in pkgdb2
18:30:40 <nirik> oh yeah, I saw that. Much feedback?
18:30:46 <pingou> none :)
18:30:55 <nirik> figures. ;(
18:30:58 <nirik> #topic Sysadmin status / discussion
18:31:02 <pingou> so it's perfect! :)
18:31:06 <nirik> ok, on the sysadmin side of things....
18:31:41 <nirik> I reinstalled out tummy01 server and then reinstalled proxy03 on it with threebean's excellent proxy ansible playbooks.
18:31:49 <smooge> cool
18:31:54 <nirik> There were some minor tweaks but overall it seems to be running great now.
18:32:00 <smooge> I need to take that and work on next people
18:32:41 <pingou> what is on tummy01?
18:32:45 <nirik> I also wrote playbooks for blockerbugs this morning and migrated blockerbugs01.stg to ansible/rhel7
18:33:04 <nirik> pingou: proxy03, smtp-mm-tummy01, unbound-tummy01
18:33:24 <pingou> and tummy is a location or ?
18:33:25 <pingou> (name sounds weird to me)
18:33:32 <nirik> yes. It's a donated server from tummy.com. ;) (who I used to work for)
18:33:41 <pingou> ok :)
18:33:46 <nirik> it's here in denver (at a datacenter)
18:33:50 <Mohamed_Fawzy> Good
18:33:59 <pingou> nirik: is it angry?
18:34:10 <nirik> no?
18:34:29 <pingou> hungry**
18:34:41 <nirik> ha. ;) No, it's full and content
18:34:45 <pingou> and this, ladies and gentlemen, is how you miss a joke :)
18:34:47 <Mohamed_Fawzy> lol
18:35:18 <nirik> so, with the blockerbugs01.stg move this morning, we are down to 50 hosts left in puppet.
18:35:20 <pingou> nirik: how many app/hosts left in puppet?
18:35:27 <nirik> ^ :)
18:35:31 <pingou> big ones?
18:35:50 <nirik> well, now that we have proxy playbooks we can do the proxies... there's a bunch of them
18:36:05 <smooge> so the other 40 we can just turn off?
18:36:09 <nirik> a few are machines that are going to go away before too long
18:36:19 * pingou _ó/\o_ threebean for the proxy playbok
18:36:21 <smooge> no one needs lockbox
18:36:28 <nirik> oh, reminds me...
18:36:32 <pingou> smooge: +1
18:36:42 <nirik> I have koji01/02 now setup and ready to go to replace koji03/04
18:36:58 <nirik> I was thinking of doing that tomorrow. It actually should only be a short blip... no real outage
18:37:13 <nirik> and that will be 2 less in puppet then. ;)
18:37:22 <bochecha> another one in puppet is pkgs01 :x
18:37:32 * nirik nods. yep.
18:37:54 <tflink> any updates on the new openstack?
18:37:58 <nirik> we also have some more remote sites we need to reinstall... and we have run into issues with rhel7 anaconda not wanting to nuke old installs right. ;(
18:38:17 <pingou> we're almost there for this one
18:38:21 <pingou> (pkgs01)
18:38:30 <nirik> tflink: msuchy has been working on it. He now has the head node up and working... and was going to add compute nodes.
18:38:45 <nirik> tflink: hopefully we just add in a few more things and we can start using it.
18:38:53 <nirik> storage got figured out/is working
18:38:57 <smooge> cool
18:39:16 <nirik> of course I keep thinking this, but perhaps someday I will be right. ;)
18:39:32 <nirik> #info koji01/02 migration tomorrow sometime, shouldn't need an outage.
18:39:39 <nirik> #info 50 machines left in puppet currently
18:39:58 <nirik> #info proxy playbooks looking good in ansible, will start converting the rest of them
18:40:17 <nirik> #info pkgs01.stg is getting closer, once done we can roll out to pkgs01
18:40:26 <smooge> nirik, I expect that someday we will just say "tough fulsom dies." and will use whatever word they give X :)
18:40:42 <nirik> #topic nagios/alerts recap
18:40:54 * nirik looks for the url.. should put it in the meeting announcement
18:42:27 <nirik> nagios is being anoying... anyhow it was a heavy alert week...
18:42:42 <nirik> #topic Upcoming Tasks/Items
18:42:42 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:42:51 <nirik> anything upcoming anyone would like to schedule or note?
18:43:09 <nirik> I'll mention again that smooge and i will be at the main phx datacenter feb 9th to 12th
18:43:30 <pingou> just during branching
18:44:16 <nirik> indeed.
18:44:19 <nirik> we can work around that.
18:44:34 <nirik> pingou: you want to talk about one of our apps for a few? which one?
18:44:40 <pingou> I'll be there to handle the pkgdb side anyway
18:44:45 <pingou> sure I can
18:45:16 <pingou> I've got election, nuancier, pkgdb, fedocal, mirrormanager2 (this one might be a little complex to explain right now)
18:45:20 <pingou> any preferences?
18:45:41 <nirik> how about elections?
18:45:48 <pingou> sure
18:45:52 <nirik> #topic Overview of an app - elections
18:46:03 <pingou> so the election app is available at https://admin.fedoraproject.org/voting
18:46:19 <pingou> it's basically the web application that we use to do all the vote in the community
18:46:56 <pingou> We had an old turbogear 1 application which has been ported to turbogears 2 (never deployed), then to flask
18:47:09 <pingou> and it is this last version (using the flask framework) that is now running
18:47:33 <pingou> most of the heavy lifting of porting the application has been done by fchiulli
18:47:37 <pingou> and I took over after this
18:47:49 <pingou> the idea is that we have some sort of plugin system
18:47:56 <pingou> allowing different type of election
18:48:05 <pingou> for example: select 10 candidates in a list of 100
18:48:34 <pingou> or: assigned a score ranging from 10 (most preferred) to 1 (least preferred) to all the candidates
18:48:45 <pingou> or, pick one out of all the candidate
18:48:54 <nirik> is there other plugins now than range voting?
18:49:14 <pingou> nirik: all the exemple I just gave are present :)
18:49:30 <nirik> cool
18:49:35 <pingou> there is also an 'IRC' plugin supporting vote following the -1/0/+1 approach taken during IRC votes
18:49:57 <pingou> the UI adjusts itself depending on the type of election
18:50:30 <pingou> and after an election has ended, there are some stats available for it, for example https://admin.fedoraproject.org/voting/results/council-nov14
18:50:45 <Corey84> nice
18:51:09 <pingou> there is a graph coming in the next release showing the evolution of the votes during the voting period
18:51:23 <pingou> (so: are people voting more at the beginning or at the end of the voting period?)
18:51:45 <nirik> on the deployment side we have 2 app servers running the app talking to a database. So, if either is down the other takes all the traffic.
18:52:14 <nirik> pingou: there were some slowdown issues with the last election, we solved all those right? or were there still some?
18:52:16 <pingou> from a technology point of view it is: python, the flask web-framework, 1 database (postgresql) interfaced to using SQLAlchemy and jquery flot for the graph coming in the next release
18:52:27 <pingou> from an infra point of view: nirik already explained :)
18:52:35 <pingou> nirik: yes we solved it
18:52:55 <nirik> ok, just wanted to make sure, we have another election coming up. ;)
18:52:57 <pingou> the issue was that we were querying FAS to retrieve the real name of each candidate everytime someone wanted to vote
18:53:13 <pingou> since the real name doesn't change that often, we are now caching the info :)
18:53:24 <pingou> (thanks threebean for fixing this)
18:53:42 <nirik> ah yes. ;)
18:53:52 <pingou> any questions ?
18:54:47 * nirik asked his already
18:54:54 <nirik> oh, code is at: https://github.com/fedora-infra/elections
18:54:56 <pingou> Corey84: Mohamed_Fawzy all good?
18:55:01 <pingou> nirik: good point, thanks! :)
18:55:13 <Mohamed_Fawzy> Not 100% but i got some
18:55:16 <Corey84> yup all good here
18:55:29 <pingou> cool :)
18:55:49 <pingou> anyway, if you ever have a question about this app or another one, feel free to ask on -admin or -apps
18:55:56 <nirik> We are going to try and go over some app or setup each meeting. ;) Hopefully helps us all
18:55:56 <pingou> (that is #fedora-admin or #fedora-apps)
18:56:04 <Mohamed_Fawzy> i know it is first meet i must get some things fall
18:56:04 <nirik> thanks pingou
18:56:12 <nirik> #topic Open Floor
18:56:23 <nirik> anyone have any items for open floor? questions, comments?
18:56:32 <nirik> Mohamed_Fawzy: yeah, theres a lot to take in. :)
18:56:42 <smooge> nothing for me
18:56:46 <Mohamed_Fawzy> yea hahhahaha
18:57:18 <nirik> cool. Well, everyone do continue over in #fedora-admin, #fedora-noc and #fedora-apps.
18:57:18 <smooge> pingou, congrats on the elections thing!
18:57:28 <nirik> #endmeeting