19:00:00 #startmeeting Infrastructure (2013-02-07) 19:00:00 Meeting started Thu Feb 7 19:00:00 2013 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:00 Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:01 #meetingname infrastructure 19:00:01 The meeting name has been set to 'infrastructure' 19:00:01 #topic welcome to all 19:00:01 #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken dgilmore mdomsch threebean 19:00:01 Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch nirik ricky skvidal smooge threebean 19:00:07 * skvidal is here 19:00:08 welcome everyone. 19:00:09 * puiterwijk 19:00:13 * mattdm is here 19:00:15 * pingou is here 19:00:22 * maayke is here 19:00:31 * relrod is here 19:00:42 * threebean is here 19:00:55 * abadger1999 here 19:01:01 * nirik will wait a min more for folks to wander in. ;) 19:01:05 hi 19:01:11 * herlo is here 19:01:15 * mdomsch 19:01:44 * lmacken 19:01:47 ok, lets go ahead and get started. ;) 19:01:50 #topic New folks introductions and Apprentice tasks. 19:02:01 any new folks like to say hi? or apprentices have questions or commends? 19:02:03 comments even 19:02:08 ! 19:02:31 cmullead: go ahead... 19:03:08 I am new to the group. I have been a user/admin of Red Hat/Fedora for many years 19:03:25 and I thought it was time to give something back 19:03:29 welcome. 19:03:29 eof 19:03:42 are you more interested in sysadmin type tasks? or application development? or both? :) 19:03:59 system admin 19:04:33 cool. ;) See me after the meeting over in #fedora-admin and I can point you in the right direction to get started. ;) 19:04:34 This is my first meeting. Im an intern with Gnome OPW program working on datanommer with threebean 19:04:48 ok 19:04:49 welcome housewifehacker1. 19:05:10 * nirik is very much looking forward to datanommer. ;) 19:05:16 \ó/ 19:05:28 ah ha! that answers a question I had about your nick 19:05:32 housewifehacker1: welcome 19:05:52 hmmm actually it doesn't really answer my question 19:06:01 but it is a not an interesting or useful question 19:06:11 :) 19:06:21 :-) 19:06:22 ok, any other new folks? Please do let us know if we can assist or direct you in helping out any. Thanks for contributing. 19:06:46 #topic Applications status / discussion 19:06:48 this is my first meeting as well. I'm a sysadmin willing to give something back to the community. I applied to the bug zappers group but would like to help out on anything infrastructure related as well. 19:06:53 #undo 19:06:54 Removing item from minutes: 19:07:24 dkanbier: welcome. Please do also see me after the meeting in #fedora-admin and I can point you in the right direction... ;) 19:07:35 will do :) 19:07:35 #topic Applications status / discussion 19:07:44 ok, any application news from this week/upcoming? 19:08:04 I want to call out for testers for FAS-OpenID again ,if I may? 19:08:08 copr-cli progressed over the week-end (even with a very interesting fosdem) 19:08:09 puiterwijk: please do. 19:08:17 * mdomsch fixed several bugs in MM 1.4 in staging; getting to the point where I need to start running cronjobs against live data to find more bugs 19:08:25 jenkins-master is also asking for testers 19:08:34 as I have not yet received any comments on FAS-OpenID last week, I would like to again ask for testers 19:08:52 puiterwijk: smooth for me 19:08:53 should we perhaps send out a devel-announce post asking for more testers? or is that too widespread? 19:08:54 #info OpenID ID for FAS-OpenID: http://.id.stg.fedoraproject.org/ 19:09:00 pingou: I'd like to migrate bodhi & kitch over at some point 19:09:04 **kitchen 19:09:13 lmacken: sounds great 19:09:20 nirik: I think that's a bit too wide 19:09:23 pingou: have you or slavek tried out copr-fe against it? 19:09:31 pingou: is there an easy way to migrate jobs? 19:09:32 or a test of copr-fe 19:09:37 skvidal: I didn't :] 19:09:45 puiterwijk: ok, how about infra list? 19:09:53 nirik: that would be a good one, I think 19:09:54 skvidal: just against local (and not for build) 19:09:59 nod 19:10:14 lmacken: I copy/pasted from jenkins to cloud when I did it for fedocal/f-r 19:10:33 pingou: cool 19:10:55 lmacken: the only thing to take care of is 1 build = 1 python 19:10:58 builder* 19:11:09 nirik: do you want me to send it after the meeting? 19:11:14 puiterwijk: please do. 19:11:16 (or do you want to do yourself?) 19:11:24 sure, will do 19:11:53 pingou: perhaps a jenkins post too... ? 19:12:26 nirik: I wanted to write down a wiki page on what's our idea of use-case for jenkins 19:12:37 but yeah after a general announcement is a good idea 19:12:41 thats also a good idea. agreed. 19:13:18 so, housewifehacker1 has put a ton of improvements into datanommer and we're looking to upgrade the instance in staging soon 19:13:24 mdomsch: I think early next week I might try and split out mm mirrorlists to ther own servers. Or should I wait on that until 1.4 is in prod? 19:13:27 here's our todo list for it https://github.com/fedora-infra/datanommer/issues/27 19:13:34 cool threebean housewifehacker1 19:13:47 nirik: we can split them out first 19:14:04 mdomsch: ok. I would think it would be pretty easy... 19:14:18 should be 19:14:22 if I'm not around and she happens to need help with various fi-apprentice things, any extra help or advice would be appreciated. 19:14:39 My door/irc client is always open. ;) 19:14:47 nirik: devel would be fine for fa-openid mail I think, devel-announce is supposed to be even lower volume, though. 19:15:20 abadger1999: so you think the email should go to devel or devel-announce? 19:15:24 sure, but this seemed like a one off announcement. ;) 19:15:27 puiterwijk: devel 19:15:33 not devel-announce 19:15:45 but devel sounds fine to me too 19:15:51 ah ok. but not just infra? 19:15:52 shouldn't wait for the design to announce it? 19:16:14 (lastly, on datanommer.. the upshots to the latest changes are three-fold: 1) we get a smooth upgrade path for the DB with python-alembic, 2) we get an API we need to write datagrepper, and 3) we get some CLI scripts we need for nice nagios checks ) 19:16:19 dvel-announce is for people who are too cranky to subscribe to devel :-/ 19:16:44 * SmootherFrOgZ is here now. 19:17:00 pingou: fine with that too -- if we want testing now, though, I think most of hte infra people are already aware 19:17:05 threebean / housewifehacker1: question: how is the db looking for datanommer so far? is it growing very big? should we split it in prod? or we don't know yet? 19:17:26 pingou: on jenkins? or fas-openid? 19:17:32 nirik: openid 19:18:06 ok, I'm fine waiting on a design writeup on those, I agree it would be good to have. more so on jenkins... 19:18:12 I tend to agree with pingou on that, but maybe someone could help me with getting the design 19:18:15 nirik: don't know yet -- we haven't checked. 19:18:34 nirik: you want a fedora design for jenkins?? 19:18:50 I could use help either with getting the design team a kick or with a temporary design until they are finally done 19:18:59 oh I see. I was misunderstanding 'design' here. 19:19:10 you are talking about theme/appearance? 19:19:12 nirik: oh, that's not true. A dump was 236M back at the end of january. 19:19:15 nirik: yes 19:19:21 * ianweller is here 19:19:38 I was talking about a wiki page with 'here's what this is, and how you can use it, and what problems it solves, etc' 19:19:52 puiterwijk: I can probably whip up at least a temporary design for it, poke me after the meeting. 19:20:17 relrod: ok, thanks :) (design would have been ready about two weeks back, but they're "still busy"...) 19:20:29 threebean: If it's postgres, point me at it and I can get numbers for you. 19:20:30 so, if we ask for testing we can point people at that and they can see how it's used, etc 19:20:49 threebean: or a df -k /var/lib/pgsql should do it if it's the only database on that server. 19:20:54 nirik: +1, that's kind of my idea for jenkins 19:21:10 abadger1999: it is.. I'll check here in a moment. 19:21:56 pingou: yeah. with how to add things, what things we can't add, etc. 19:22:21 nirik: I'll send it around for review/comments before I announce it :) 19:22:27 sounds good. 19:22:43 with fas-openid, it's much less a issue because thats "anywhere you want to use openid" :) 19:23:00 #info will work out a theme/skin for fas-openid and then ask for wider testing soon. 19:23:02 19:23:20 #info datanommer to be updated in stg and tested by threebean / housewifehacker1 19:23:31 abadger1999: its db01.phx2.fp.o.. so, there are other dbs there. 19:23:49 #info jenkins will get a faq doc and then more widespread testing. 19:24:06 Any other application news or plans? 19:24:18 oh, I've been fighting with askbot in stg with fedmsg plans. 19:24:26 * lmacken planning a bodhi update this week. masher changes only. 19:24:31 threebean: k. I'll get it with a query 19:24:33 sorting out fedora/epel django packaging issues for most of the week. 19:24:45 #info askbot upgrade being prepped in stg. 19:24:49 threebean and I have also been fighting with a mod_wsgi+fedmsg threading issue recently 19:24:51 threebean: thanks a lot for that. ;( 19:25:14 #info bodhi update for prod soon with bugfixes 19:25:20 threebean: if there is anything special it might be good to talk with Aurélien since HK is django 19:25:29 might prevent us from having the problem later 19:26:21 ok, moving on then... 19:26:22 * threebean nods 19:26:29 #topic Sysadmin status / discussion 19:26:50 So, we hopefully have some arm machines being racked today. Will need to get them up and running. 19:26:59 arg on apps, HK has made some really nice progress last week 19:27:00 nice! 19:27:20 Hopefully we can use a few of them for infra stuff too, which will be nice. 19:27:32 \o/ 19:27:33 nirik: need help on the arm boxes? 19:27:36 pingou: cool. I really want it to be ready. :) 19:27:37 pingou: wrt HK, we need to change the login form to https 19:27:55 lmacken: mentionned, forgot to open a ticket for that though 19:28:04 puiterwijk: well, we first need network and such. ;) Hopefully it will be pretty easy to get them setup 19:28:23 nirik: hehe, would be useful I guess yeah 19:28:25 I'd like to discuss a bit a few bigger picture plans... 19:28:34 a) splitting out apps from app servers. 19:28:50 I'm going to start on this likely next week... with mirrorlist server. 19:29:12 but at the end of it, it would be nice if we had no need for app boxes... 19:29:19 +billion 19:29:28 does anyone have concerns or questions about that being the way to go? :) 19:29:37 which cloud are we going to migrate the apps to? 19:29:47 are we migrating to the cloud? 19:29:51 no cloud yet, just seperate instances 19:29:54 I thought we were just going to split out instances 19:30:02 okay 19:30:12 so, make mirrorlist01 02 03 in phx2 and move that to those, etc. 19:30:28 threebean, nirik: datanommer on db01 is currently 305MB 19:30:41 select pg_database_size('datanommer'); 19:30:50 Down the road / rhel7 or something we might be able to use containers for smaller apps 19:31:03 but thats someday/sidetrack 19:31:17 nirik: works for me. 19:31:44 abadger1999: we're looking at a rate of ~1GB/year 19:31:51 short term this means more instances, but hopefully they can be smaller and help us with debugging/prevent one thing from messing up another 19:31:59 nirik: you going to just work on separating the web front ends right now? (db's at a later date)? 19:32:07 abadger1999: yeah. 19:32:14 cool. 19:32:23 I still hope someday we can figure out a better way to replicate/deal with dbs 19:32:28 do we still plan on sticking with i686 vms? 19:32:38 nirik: I have a replication setup in cloud I believe 19:32:39 nirik: oh something else to think about -- 19:32:53 lmacken: good question. we could try 64bit again... but the memory thing is anoying. 19:32:57 nirik: If we go with openid everywhere, I believe we'll be able to have SSO but different domains. 19:33:13 nirik: yeah :\ 19:33:29 abadger1999: completely correct 19:33:31 is there any new/cool way around it? perhaps we should ask dmalcolm again? 19:33:37 ie: packages.apps.fedoraproject.org, ask.fedoraproject.org, pkgdb.fedoraproject.org, etc will all be transparently logged into because of openid. 19:33:44 puiterwijk: cool. write it up and we can discuss it on list? 19:33:55 abadger1999: I do like that idea :) 19:33:57 nirik: sure, will try to find the servers back again :) 19:34:02 abadger1999: +1 19:34:05 nirik: I don't think there is any way around it (PyObject pointers double in size on 64bit). I'll ask dmalcolm though to make sure. 19:34:25 rhel doesn't ship 32bit python on x86_64 do they? 19:34:42 lmacken: there are ways but -- not as in, "oh, I'll just patch this and[...]" 19:35:10 nirik: not sure 19:35:17 yeah, I don't think so. 19:35:18 nirik: I believe they actually do, I'll check 19:35:32 nirik: they do 19:35:33 anyhow, worth looking into before we build a bunch of servers. ;) 19:35:42 it would require changing CPython data structures. One thing dmalcolm might know, though, is whether we could run on an alternate interpreter and gain memory. 19:35:49 but then if we use that, we need to build our stack of packages against that? 19:36:14 and if we're split out, we might be able to try that on just a few apps which have a compatible stack. 19:36:20 * lmacken asked dave about it in #fedora-python 19:36:34 nirik: RHEL does ship 32-bit python on x86_64 19:36:35 cool. lets continue looking at this out of meeting? 19:36:39 nirik: so.. .about building a bunch of servers.. 19:36:54 nirik: -- Anyhow -- I don't htink any of the potential solutions would be ready in time for this split to multiple app servers. 19:36:54 puiterwijk: yeah, but how many of the things we need are either noarch or have a 32bit version? 19:36:55 nirik: the koji builders are spun up using ansible + kickstart 19:37:19 nirik: we could think about implementing the new app-specific instances like that 19:37:25 nirik: ie - puppetless 19:37:26 skvidal: an _excellent_ segway into my next topic... 19:37:31 b) migration to ansible 19:37:33 So if memory is our constraint, I think we should stick with 32bit and think about 64bit+other solution in a year or more. 19:37:43 abadger1999: fair enough 19:37:54 dmalcolm | dmalcolm: or rather: yes: it will use nearly double the ram 19:38:11 so, I think we need to whip up a plan for our ansible migration... and I can work on that. Then start working away at the little parts... 19:38:14 skvidal: exciting prospect 19:38:28 short term we need: 19:38:32 nagios 19:38:44 it'd be nice to have fas-push 19:38:46 yeah, nagios was my c) topic. ;) 19:39:01 it would be nice to move builder ansible repo into main ansible repo 19:39:02 the cron runner stuff - which is on my plate and something I'm actively working on both parts of 19:39:06 yep. 19:39:17 agreed on moving builder ansible into main ansible 19:39:56 if anyone wants to do that and work on it 19:40:01 we will end up with some machines puppet and newer stuff ansible, but I think thats ok for migrating... and we can keep converting away until we get to where we can have a flag day for the rest. 19:40:04 and I'm the blocking point 19:40:08 don't block on me 19:40:10 yell at me 19:40:18 and take on the process 19:40:24 we have lots of builders 19:40:34 and taking one out to test a new rebuild playbook from the main repo 19:40:39 is super-duper easy 19:40:42 skvidal: I can look at doing the new mirrorlist boxes via ansible... that should possibly get us a global/whatever conversion to base others on 19:41:08 * mdomsch may need a 64-bit box for s3-mirror 19:41:08 nirik: I think I need to look at global again - istr it is full of crap 19:41:28 nirik: and by that I mean - it has a bunch of completely unnecessary system changes in it 19:41:29 skvidal: yes, we should only convert over the parts that make sense. ;) 19:41:33 nod 19:41:41 this is a good chance to clean that junk up 19:41:44 okay - sorry for telling you a bunch of things you already know :) 19:41:48 mdomsch: sad that it takes so much mem 19:41:55 not at all. :) 19:42:08 ok, so c) on my list is 'redo nagios' 19:42:08 nirik: it's tracking nearly 1M files 19:42:26 mdomsch: yeah, understandable. ;( 19:42:51 2M really, as it does so on each end 19:42:56 I'm going to whip up a mail on nagios and we can come up with a design... anyone interested in helping welcome. 19:43:46 ok, any other sysadminy stuff folks had? 19:44:05 #topic Private Cloud status update / discussion 19:44:22 so, we need/want to update and reboot clouds... possibly next week? 19:44:22 nirik: I'm interested in helping with nagios 19:44:31 puiterwijk: excellent. ;) 19:44:49 me too, quite some experience with zabbix 19:44:51 also, I'm interested in knocking off anything we have left before calling them production. 19:45:18 dkanbier: cool. 19:45:26 nirik: so all current instances will get killed? 19:45:43 puiterwijk: yep. we will need to restart them all. 19:45:54 nirik: restart or terminate? 19:46:17 terminated. ;) 19:46:23 since hosts will get rebooted. 19:46:36 ok 19:46:50 but we will provide notice on that, etc. 19:47:01 ah ok, great. 19:47:14 #topic Upcoming Tasks/Items 19:47:25 we now have an actual schedule for f19... so I added it in: 19:47:35 #info 2013-02-08 mass rebuild for f19 starts. 19:47:35 #info 2013-02-11 pkgdb update. 19:47:35 #info 2013-02-18 to 2013-02-19 smooge on site at phx2. 19:47:35 #info 2013-02-28 end of 4th quarter 19:47:36 #info 2013-03-29 - spring holiday. 19:47:37 #info 2013-04-02 to 2013-04-16 ALPHA infrastructure freeze 19:47:38 #info 2013-04-16 F19 alpha release 19:47:40 #info 2013-05-07 to 2013-05-21 BETA infrastructure freeze 19:47:42 #info 2013-05-21 F19 beta release 19:47:44 #info 2013-06-11 to 2013-06-25 FINAL infrastructure freeze. 19:47:46 #info 2013-06-25 F19 FINAL release 19:47:51 anything folks would like to schedule or note? 19:47:53 yes 19:48:01 mattdm: fire away. ;) 19:48:10 The cloud feature asks for a koji update in there at the end of march 19:48:29 (cloud image build feature) 19:48:46 ah ha. 19:48:47 ok. 19:48:57 Specifically given the current schedule, sometime between March 19th and March 26th. 19:49:03 Which isn't a huge window, I know. 19:49:12 mattdm: do you want me to pencil it in at a specific date? or ... ok. 19:49:37 It a little bit depends on the actual koji code development. 19:49:53 yeah. 19:49:58 ok, will add it in. ;) 19:50:01 March 26th is chosen as 1 week before alpha freeze, and it kinda works back from there. 19:50:10 thanks. that's all. :) 19:50:20 ok, keep us posted. ;) 19:50:26 yep, i will. 19:50:35 #topic Open Floor 19:50:42 anyone have anything for open floor... 19:50:57 almost made it this time... :/ 19:51:13 pingou: say, whats the status of fedocal? 19:51:30 abompard: welcome. ;) whats the word on hyperkitty/mm3? ;) 19:51:33 hey everyone, just joining late to say hello :-) 19:51:37 nirik: I have some RFE do cover, need to start packaging it 19:52:02 nirik: progress ! mm3test should be up-to-date, much debugging today thanks to pingou 19:52:08 nirik: basically, still in 0.1.0-alpha but little bugs were reported, so either it works fine or few people tested it :) 19:52:23 http://mm3test.fedoraproject.org/hyperkitty/ 19:52:32 pingou: thats another one I think we could move to wider testing soon/stg? 19:52:37 it has in-thread reply! 19:52:46 nirik: it would be nice for sue 19:53:38 lmacken: https://fedorahosted.org/hyperkitty/ticket/38 19:54:14 abompard: can you add openid to the login auth? 19:54:18 or is that planned. 19:55:03 nirik: hmm, I'm not sure we should open any type of auth, I'd rather keep it to the email providers and browserid 19:55:12 nirik: since it's a mailing-list app 19:55:26 pingou: thanks! 19:55:30 nirik: I need to make sure an email will be provided 19:55:30 abompard: fas provides openid, so does google/yahoo 19:56:00 abompard: when you would configure it for FAs-OpenID, you're sure the user has a working email 19:56:00 we can get email from fas openid I think... 19:56:05 * nirik nods. 19:56:07 pingou: I know, but any openid provider does not provide an email address (it's the other way around) 19:56:26 abompard: FAS-OpenID provides email addresses 19:56:31 abompard: and even if it wouldn 19:56:35 puiterwijk: HK isn't fedora specific 19:56:41 ah ok 19:57:01 sorry, then I was misled by the demo I got linked to last week ;) 19:57:05 can't you ask for email in the openid request and fail if you don't get one back? 19:57:09 yes 19:57:30 you can say that you require an email address, and the openid protocol will abort if the provider doesn't want to give it 19:57:32 but of course someone could send foo@bar.com... 19:57:53 nirik: why not, I also must be sure that the openid provider validates the email adress 19:57:55 nirik: then that's the provider providing that email. most providers check it 19:58:02 or you could impersonate anyone 19:58:36 abompard: in that case, we have the provider url and blacklist him? 19:58:39 wlel, it's already pretty easy to do with smtp, but yeah, there's issues. 19:58:54 anyhow, openid would be nice if we can figure a good way to do it. 19:59:14 abompard: I love HK, btw. nice work. 19:59:15 nirik: true. But the user has settings here, like his password that you could reset, etc. 19:59:20 threebean: thanks ! 19:59:40 yeah. 19:59:53 ok, we are almost out of time now. any last minute open floor items? 20:00:17 ok, thanks for coming everyone. ;) 20:00:20 #endmeeting