18:00:53 <nirik> #startmeeting Infrastructure (2015-01-22) 18:00:53 <zodbot> Meeting started Thu Jan 22 18:00:53 2015 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:53 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:53 <nirik> #meetingname infrastructure 18:00:53 <zodbot> The meeting name has been set to 'infrastructure' 18:00:53 <nirik> #topic aloha 18:00:53 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:53 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:57 * pingou here 18:00:58 * puiterwijk is around 18:01:04 * fale here 18:01:07 <janeznemanic> hi 18:01:08 * relrod here 18:01:22 * bochecha is also here (for once) 18:01:42 * sayan here 18:02:26 <nirik> alrighty. welcome everyone 18:02:28 * tflink is here 18:02:40 <nirik> #topic New folks introductions and Apprentice tasks 18:02:49 <nirik> Any new folks like to introduce themselves? 18:02:56 <nirik> or any apprentices with questions or comments? 18:03:00 <andreasch> Hello everyone, my name is Andreas Christoforou and I am working as a system administrator also I am a fedora ambassador and I would like to help and contribute to the Fedora infrastructure. 18:03:04 <dgilmore> hola amigos 18:03:15 <nirik> andreasch: welcome. 18:03:24 <pingou> welcome andreasch 18:03:30 <nirik> are you more interested in sysadmin (seems likely) or application development type tasks? 18:03:33 <pingou> hóla dgilmore 18:03:51 <andreasch> sysadmin;) 18:04:01 <Corey84> not new to group but willing to take on apprenticeship tasks 18:04:24 <Mohamed_Fawzy> hi 18:04:26 <Corey84> both for me sysadmin is strong suit tho 18:05:08 <nirik> welcome to the fun Corey84. ;) 18:05:16 <Mohamed_Fawzy> hello sorry for late. my name is mohammed i work as linux sysadmin 18:05:23 <Corey84> nirik, lol 18:05:31 <Mohamed_Fawzy> also I am a fedora ambassador and I would like to help and contribute to the Fedora infrastructure 18:05:34 <nirik> welcome Mohamed_Fawzy. 18:05:41 * sborza is here 18:05:51 <Mohamed_Fawzy> sorry for late 18:06:00 <nirik> Please all of you see me after the meeting in #fedora-admin and I can get you added to the apprentice group and point you the way of outstanding tasks. ;) 18:06:13 <pingou> welcome Mohamed_Fawzy 18:06:36 <Mohamed_Fawzy> Wow good 18:06:40 <Mohamed_Fawzy> i like that 18:07:05 <nirik> ok, any other new folks or apprentices with questions before we move on? 18:07:16 <Corey84> nirik, seems adam3535 never was able to do o on ambassador side either last mtg possible we can hit both issues after this mtg? 18:07:48 <smooge> apologies.. late 18:07:53 <nirik> Corey84: not sure I understand the issue... but happy to look into it... 18:07:54 * danielbruno is late, but here 18:08:24 <Corey84> he was planning to look into why i had not got added to ambassadors group but was on ML 18:09:01 <nirik> ah, no idea. ambassadors control that... 18:09:12 <nirik> unless there's some technical issue with the account system. 18:09:45 <Corey84> worst case ill tag up with Southern_Gentlem later 18:09:59 <nirik> sure. 18:10:01 <nirik> #topic Applications status / discussion 18:10:09 <nirik> ok, any applications side news this week? or upcoming? 18:11:12 <sborza> nirik: on the application front re:graphite-web, we're looking into building out the proper spec files + rpms 18:11:23 <nirik> cool. 18:11:30 <sborza> once they're built i'll ping the group on getting it added for testing 18:11:47 <pingou> new pkgdb2 released, major release coming up 18:11:50 <sborza> is there issue that it uses the Apache 2.0 license? don't think so but wanted to confirm 18:12:34 <nirik> sborza: should not be, thats a acceptable license 18:12:48 <pingou> pkgdb-cli release and a new one coming up (related to the future pkgdb2 one) 18:12:57 <pingou> I also updated fedocal 18:13:20 <pingou> all the changes have been announced on the infrastructure list 18:13:39 <nirik> yep. I really appreciate that... nice to see whats changed and when a release comes out. 18:14:02 <nirik> I'd love such for all our apps. ;) 18:14:17 <pingou> I thought the same, if something break everyone knows what changed in the most recent version 18:14:21 <nirik> #info new pkgdb release this week 18:14:30 <nirik> #info new fedocal with some bugfixes 18:14:42 <pingou> and on these words I will keep pushing new releases in the evenings and on fridays :-p 18:14:49 * pingou ducks 18:14:49 <nirik> I know threebean has been working on fixing FMN... it's suffering a backlog again. ;( 18:15:11 <pingou> yeah FMN has an interesting problem: its caching strategy is too good 18:15:39 <Corey84> FMN ? 18:15:52 <Mohamed_Fawzy> i dnt knw or get wht u all talk about 18:15:53 <pingou> https://apps.fedoraproject.org/notifications/ 18:16:21 <nirik> It's our notification service. 18:16:23 * threebean here now 18:16:40 <nirik> it listens to messages and then according to a users preference notifies them... via email, irc, etc 18:17:00 <Mohamed_Fawzy> i got that but wht FMN 18:17:08 <pingou> FedMsg Notification 18:17:09 <bochecha> Mohamed_Fawzy, that's what FMN is 18:17:33 <tflink> how's bodhi2 coming along? 18:17:35 <pingou> FMN is the application running at https://apps.fedoraproject.org/notifications/ 18:17:37 <Mohamed_Fawzy> good thanks 18:18:03 <pingou> lmacken: update on bodhi2? 18:18:29 <nirik> I made a bodhi02.stg a bit ago... we were going to install bodhi2 on it and get it up in stg so we could see it. ;) 18:18:35 <pingou> I rebuild mirrormanager2 on stg today, the playbook went through nicely and all seems to run, which scares the hell out of me :) 18:18:38 <nirik> but not sure where we are there. 18:18:47 <pingou> rebuilt* 18:19:39 <lmacken> pingou: I'm working on polishing up the masher side of things at the moment 18:19:49 <lmacken> haven't had time to work on deploying/packaging the frontend stuff yet 18:19:52 <nirik> #info mirrormananger2 in staging reinstalled and all looks ok. 18:20:09 <nirik> pingou: whats the next steps for mirrormanager2? 18:20:16 <pingou> nirik: good question 18:20:27 <nirik> lmacken: is that something someone could assist with? it's making fedora packages? or upstream releases? 18:20:41 <pingou> I have no idea really how to test if stg is returning the same thing as prod 18:21:36 <nirik> as far as the end mirrorlists? 18:21:51 <bochecha> pingou, can we all set our yum configs to point to the stg mirror manager ? 18:21:56 <pingou> yes the mirrorlists, the UI itself I expect works 18:22:13 <bochecha> pingou, then we'll see if it at least returns something usable :) 18:22:14 <lmacken> nirik: yeah, I think we need to RPMify some more of the pyramid stack. Will have to touch base with threebean about it since I think he packaged up bodhi2 itself. 18:22:19 <pingou> bochecha: I expect this will always work, as most mirrors are up to date 18:23:09 <nirik> well, it may be then we just need to forge ahead and move things into production and fix any reported issues... 18:23:10 <threebean> yeah, i did lots of the pyramid stack. some pieces might be missing though (cornice?) 18:23:26 <bochecha> didn't I package cornice during the Denver FAD? 18:23:38 <lmacken> bochecha: I believe so 18:24:04 <pingou> nirik: when would be a good time 18:24:17 <bochecha> pingou, tomorrow evening, around 5:45pm ;) 18:24:35 <pingou> bochecha: oh cool, I was planning on leaving at 5:46 for 2 weeks off :) 18:24:42 <nirik> pingou: so, do we need to do it all at once? or can we do mirrorlists first, then frontend/backend? or is that mixing data from the current v1 one with v2 mirrorlists bad? 18:25:16 <pingou> nirik: I *think* we may be able to move mirrorlist first 18:25:23 <pingou> the pickle hasn't changed normally 18:25:28 <bochecha> threebean, cornice is in: https://admin.fedoraproject.org/pkgdb/package/python-cornice/ 18:25:34 <nirik> FYI, for folks who are new... mirrormanager is the thing that keeps track of our mirrors and part of it is what people query with yum to find packages. 18:25:43 <pingou> the mirrorlist itself also barely changed (just some optimization from lmacken ) 18:26:06 * lmacken crosses his fingers and hopes those don't cause any regressions 18:26:07 <pingou> (in other words, when mirrormanager breaks, we know it, fast! :D) 18:26:15 <nirik> pingou: but it's available in epel7/rhel7 right? (current ones are 6, which I want to upgrade from) 18:26:17 <pingou> lmacken: well, you have unit-tested it ;-) 18:26:25 <pingou> nirik: they are 18:26:47 <nirik> so I would say, lets try and do one of those, see how it goes, then the rest, then the backend/frontend. 18:26:53 <pingou> mirrorlists.stg is el7 18:27:26 <pingou> nirik: I'm ok with this, just that from next week friday timing will be short w/ FOSDEM and DevConf coming up 18:27:55 <pingou> so all joke aside, we may want to upgrade one mirrorlist next week Monday or so 18:27:57 <nirik> true... can try and do it monday and that gives a week to fix anything we see? 18:28:03 <nirik> right. +1 18:28:05 <pingou> leaving us 3 days before FOSDEM to fix things 18:28:24 <pingou> nirik: and let's announce it here and there :) 18:28:25 <nirik> I suspect we will see issues pretty quickly... 18:28:44 <pingou> not too many I hope 18:28:59 <nirik> yeah, I hope it will be smooth. ;) 18:29:08 <pingou> fingers crossed ;) 18:29:22 <nirik> #info will look at converting one mirrorlist server from mirrormanager1 to 2 on monday and add it to the rotation. 18:29:48 <nirik> ok, any other application news? 18:30:30 * pingou blogged a little about the new branch process coming up in pkgdb2 18:30:40 <nirik> oh yeah, I saw that. Much feedback? 18:30:46 <pingou> none :) 18:30:55 <nirik> figures. ;( 18:30:58 <nirik> #topic Sysadmin status / discussion 18:31:02 <pingou> so it's perfect! :) 18:31:06 <nirik> ok, on the sysadmin side of things.... 18:31:41 <nirik> I reinstalled out tummy01 server and then reinstalled proxy03 on it with threebean's excellent proxy ansible playbooks. 18:31:49 <smooge> cool 18:31:54 <nirik> There were some minor tweaks but overall it seems to be running great now. 18:32:00 <smooge> I need to take that and work on next people 18:32:41 <pingou> what is on tummy01? 18:32:45 <nirik> I also wrote playbooks for blockerbugs this morning and migrated blockerbugs01.stg to ansible/rhel7 18:33:04 <nirik> pingou: proxy03, smtp-mm-tummy01, unbound-tummy01 18:33:24 <pingou> and tummy is a location or ? 18:33:25 <pingou> (name sounds weird to me) 18:33:32 <nirik> yes. It's a donated server from tummy.com. ;) (who I used to work for) 18:33:41 <pingou> ok :) 18:33:46 <nirik> it's here in denver (at a datacenter) 18:33:50 <Mohamed_Fawzy> Good 18:33:59 <pingou> nirik: is it angry? 18:34:10 <nirik> no? 18:34:29 <pingou> hungry** 18:34:41 <nirik> ha. ;) No, it's full and content 18:34:45 <pingou> and this, ladies and gentlemen, is how you miss a joke :) 18:34:47 <Mohamed_Fawzy> lol 18:35:18 <nirik> so, with the blockerbugs01.stg move this morning, we are down to 50 hosts left in puppet. 18:35:20 <pingou> nirik: how many app/hosts left in puppet? 18:35:27 <nirik> ^ :) 18:35:31 <pingou> big ones? 18:35:50 <nirik> well, now that we have proxy playbooks we can do the proxies... there's a bunch of them 18:36:05 <smooge> so the other 40 we can just turn off? 18:36:09 <nirik> a few are machines that are going to go away before too long 18:36:19 * pingou _ó/\o_ threebean for the proxy playbok 18:36:21 <smooge> no one needs lockbox 18:36:28 <nirik> oh, reminds me... 18:36:32 <pingou> smooge: +1 18:36:42 <nirik> I have koji01/02 now setup and ready to go to replace koji03/04 18:36:58 <nirik> I was thinking of doing that tomorrow. It actually should only be a short blip... no real outage 18:37:13 <nirik> and that will be 2 less in puppet then. ;) 18:37:22 <bochecha> another one in puppet is pkgs01 :x 18:37:32 * nirik nods. yep. 18:37:54 <tflink> any updates on the new openstack? 18:37:58 <nirik> we also have some more remote sites we need to reinstall... and we have run into issues with rhel7 anaconda not wanting to nuke old installs right. ;( 18:38:17 <pingou> we're almost there for this one 18:38:21 <pingou> (pkgs01) 18:38:30 <nirik> tflink: msuchy has been working on it. He now has the head node up and working... and was going to add compute nodes. 18:38:45 <nirik> tflink: hopefully we just add in a few more things and we can start using it. 18:38:53 <nirik> storage got figured out/is working 18:38:57 <smooge> cool 18:39:16 <nirik> of course I keep thinking this, but perhaps someday I will be right. ;) 18:39:32 <nirik> #info koji01/02 migration tomorrow sometime, shouldn't need an outage. 18:39:39 <nirik> #info 50 machines left in puppet currently 18:39:58 <nirik> #info proxy playbooks looking good in ansible, will start converting the rest of them 18:40:17 <nirik> #info pkgs01.stg is getting closer, once done we can roll out to pkgs01 18:40:26 <smooge> nirik, I expect that someday we will just say "tough fulsom dies." and will use whatever word they give X :) 18:40:42 <nirik> #topic nagios/alerts recap 18:40:54 * nirik looks for the url.. should put it in the meeting announcement 18:42:27 <nirik> nagios is being anoying... anyhow it was a heavy alert week... 18:42:42 <nirik> #topic Upcoming Tasks/Items 18:42:42 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:42:51 <nirik> anything upcoming anyone would like to schedule or note? 18:43:09 <nirik> I'll mention again that smooge and i will be at the main phx datacenter feb 9th to 12th 18:43:30 <pingou> just during branching 18:44:16 <nirik> indeed. 18:44:19 <nirik> we can work around that. 18:44:34 <nirik> pingou: you want to talk about one of our apps for a few? which one? 18:44:40 <pingou> I'll be there to handle the pkgdb side anyway 18:44:45 <pingou> sure I can 18:45:16 <pingou> I've got election, nuancier, pkgdb, fedocal, mirrormanager2 (this one might be a little complex to explain right now) 18:45:20 <pingou> any preferences? 18:45:41 <nirik> how about elections? 18:45:48 <pingou> sure 18:45:52 <nirik> #topic Overview of an app - elections 18:46:03 <pingou> so the election app is available at https://admin.fedoraproject.org/voting 18:46:19 <pingou> it's basically the web application that we use to do all the vote in the community 18:46:56 <pingou> We had an old turbogear 1 application which has been ported to turbogears 2 (never deployed), then to flask 18:47:09 <pingou> and it is this last version (using the flask framework) that is now running 18:47:33 <pingou> most of the heavy lifting of porting the application has been done by fchiulli 18:47:37 <pingou> and I took over after this 18:47:49 <pingou> the idea is that we have some sort of plugin system 18:47:56 <pingou> allowing different type of election 18:48:05 <pingou> for example: select 10 candidates in a list of 100 18:48:34 <pingou> or: assigned a score ranging from 10 (most preferred) to 1 (least preferred) to all the candidates 18:48:45 <pingou> or, pick one out of all the candidate 18:48:54 <nirik> is there other plugins now than range voting? 18:49:14 <pingou> nirik: all the exemple I just gave are present :) 18:49:30 <nirik> cool 18:49:35 <pingou> there is also an 'IRC' plugin supporting vote following the -1/0/+1 approach taken during IRC votes 18:49:57 <pingou> the UI adjusts itself depending on the type of election 18:50:30 <pingou> and after an election has ended, there are some stats available for it, for example https://admin.fedoraproject.org/voting/results/council-nov14 18:50:45 <Corey84> nice 18:51:09 <pingou> there is a graph coming in the next release showing the evolution of the votes during the voting period 18:51:23 <pingou> (so: are people voting more at the beginning or at the end of the voting period?) 18:51:45 <nirik> on the deployment side we have 2 app servers running the app talking to a database. So, if either is down the other takes all the traffic. 18:52:14 <nirik> pingou: there were some slowdown issues with the last election, we solved all those right? or were there still some? 18:52:16 <pingou> from a technology point of view it is: python, the flask web-framework, 1 database (postgresql) interfaced to using SQLAlchemy and jquery flot for the graph coming in the next release 18:52:27 <pingou> from an infra point of view: nirik already explained :) 18:52:35 <pingou> nirik: yes we solved it 18:52:55 <nirik> ok, just wanted to make sure, we have another election coming up. ;) 18:52:57 <pingou> the issue was that we were querying FAS to retrieve the real name of each candidate everytime someone wanted to vote 18:53:13 <pingou> since the real name doesn't change that often, we are now caching the info :) 18:53:24 <pingou> (thanks threebean for fixing this) 18:53:42 <nirik> ah yes. ;) 18:53:52 <pingou> any questions ? 18:54:47 * nirik asked his already 18:54:54 <nirik> oh, code is at: https://github.com/fedora-infra/elections 18:54:56 <pingou> Corey84: Mohamed_Fawzy all good? 18:55:01 <pingou> nirik: good point, thanks! :) 18:55:13 <Mohamed_Fawzy> Not 100% but i got some 18:55:16 <Corey84> yup all good here 18:55:29 <pingou> cool :) 18:55:49 <pingou> anyway, if you ever have a question about this app or another one, feel free to ask on -admin or -apps 18:55:56 <nirik> We are going to try and go over some app or setup each meeting. ;) Hopefully helps us all 18:55:56 <pingou> (that is #fedora-admin or #fedora-apps) 18:56:04 <Mohamed_Fawzy> i know it is first meet i must get some things fall 18:56:04 <nirik> thanks pingou 18:56:12 <nirik> #topic Open Floor 18:56:23 <nirik> anyone have any items for open floor? questions, comments? 18:56:32 <nirik> Mohamed_Fawzy: yeah, theres a lot to take in. :) 18:56:42 <smooge> nothing for me 18:56:46 <Mohamed_Fawzy> yea hahhahaha 18:57:18 <nirik> cool. Well, everyone do continue over in #fedora-admin, #fedora-noc and #fedora-apps. 18:57:18 <smooge> pingou, congrats on the elections thing! 18:57:28 <nirik> #endmeeting