18:00:03 #startmeeting Infrastructure (2014-05-01) 18:00:03 Meeting started Thu May 1 18:00:03 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:03 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:03 #meetingname infrastructure 18:00:03 #topic greetings starfighters 18:00:03 #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:03 The meeting name has been set to 'infrastructure' 18:00:03 Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:10 * relrod waves 18:00:26 * pingou 18:00:31 * webpigeon waves 18:00:48 hi 18:01:04 * lmacken 18:01:33 I'm here, but if Konversation wigs out on me again 18:01:47 I'll be here instead 18:02:00 :) 18:02:06 ok, lets go ahead and get started.... 18:02:08 * threebean is here 18:02:13 #topic New folks introductions and Apprentice tasks 18:02:27 hi, i am new here 18:02:34 any new folks want to do a quick one line introduction of themselves? or apprentices with questions or comments? 18:02:46 yes, sure 18:02:48 I am sysadmin, my job is high load and high availability application as web, mail and databases services as well as bgp and ospf networking 18:03:00 I'll also jump in as one of the newbies 18:03:02 and i am interested in sysadmin things, if available 18:03:31 I've been using Linux for years and now want to contribute here. I think the testing FIG is my best starting point, with an eye toward sysadmin-main (eventually) 18:03:38 danrimal: welcome! sure thing... see me in #fedora-admin after the meeting and I can get you setup in our apprentice program... 18:03:52 ootbro: welcome again. ;) 18:03:59 thanks 18:04:22 As I introduced myself in the mailing, i'm a sysadmin/engineer from switzerland and very interested getting in charge here. And i search some fig to join. I think also fig testing or fig web is a good place for me. 18:05:00 nj0y: welcome also. ;) 18:05:16 * ghostalker is here 18:05:17 thanks, glad i'm here. 18:05:30 always good to have new folks around... do chime in with questions or comments anytime... 18:05:31 .fasinfo mohanprakash 18:05:32 mpduty: User: mohanprakash, Name: Mohan Prakash, email: mpduty@gmail.com, Creation: 2013-12-27, IRC Nick: mpduty, Timezone: Asia/Kolkata, Locale: en, GPG key ID: 0xAF620142, Status: active 18:05:36 mpduty: Unapproved Groups: l10n-editor l10n-commits marketing 18:05:39 mpduty: Approved Groups: fi-apprentice cvsl10n cla_done cla_fpca 18:06:26 I can assist anyone after the meeting over in #fedora-admin who wants to join our apprentice group or would like to be pointed at easyfix tickets, etc. :) 18:06:31 Welcome again everyone! 18:06:52 many thanks.... I could use the help in getting started 18:07:01 me too. 18:07:08 ok, thanks 18:07:12 http://fedoraproject.com/easyfix 18:07:23 * mattdm is lurking 18:07:35 * bwood09 is here 18:07:46 see also http://fedoraproject.org/wiki/Infrastructure/GettingStarted and https://fedoraproject.org/wiki/Infrastructure_Apprentice 18:08:02 #topic Applications status / discussion 18:08:14 any application side news from the previous week or upcoming? 18:08:27 * pingou has done a good chunk of work on mirrormanager2 this week 18:08:42 and over the last two days I am on the re-design of some the page of pkgdb2 18:08:48 * Daredel is late 18:08:48 including http://209.132.184.188/package/R-DBI/ 18:08:53 pingou: this is the flask re-write of it? or is it tg2? or ? 18:08:57 nirik: flask 18:09:10 but I only worked on the UI 18:09:14 have you contacted mdomsch any on it? (I know he's not been around) 18:09:24 pingou: that looks *much* nicer. 18:09:43 yeah, that looks nice :) 18:09:56 * nirik is waiting for load. ;) 18:10:08 I bet a bunch of us clicked it at the same time. 18:10:35 i waited a bit, came up fine for me ;) 18:10:48 or... it could be my pesky wireless. ;( 18:10:53 which is amazing, considering what I've been fighting with locally...... 18:11:33 pingou: how is 'package administrator' determined? 18:11:41 threebean: designed by mizmo, I can't beat that :) 18:11:47 * oddshocks here late, in lecture as usual 18:11:53 nirik: Contacts are the POC, Admins are the users with approveacls 18:12:56 pingou: ok, so anyone with any approveacls? 18:13:20 yes 18:13:36 nirik: or pending approveacls (then there is a (?) icon next to them) 18:13:51 ok, cool. 18:14:14 #info some work on a flask re-write of mirrormanager ongoing 18:14:24 #info ui work on pkgdb2 ongoing 18:14:48 #info hyperkitty came up in the news a few times this week in slashdot and lwn, pointing to our stg instance. 18:15:24 are we any closer to cutting another list over? 18:15:41 erm, by that I mean changing some existing lists from mailman2 to mailman3 18:15:54 I sent abompard some issues and he was going to fix them up... then we were going to see where we were. 18:16:05 hopefully soon tho. 18:16:09 cool. newly queued stuff.. 18:16:11 * threebean nods 18:16:12 I'd be happy to move the infra list. 18:16:20 yeah, agreed 18:16:36 there may also be some fixes from this recent press on it... 18:17:10 unrelated, janeznemanic and I have been working on some fedmsg monitoring stuff and made some progress this week 18:17:15 https://fedorahosted.org/fedora-infrastructure/ticket/4044 18:17:17 excellent. 18:17:19 http://threebean.org/blog/fedmsg-collectd-ng/ 18:17:34 collectd is in place and fun. nagios checks coming soon. 18:17:56 * bwood09 starts reading the entirety of threebean.org 18:18:17 any other application type news? or shall we move on to sysadmin? 18:18:47 #topic Sysadmin status / discussion 18:19:02 smooge got some of our new build virthosts up and running yesterday. 18:19:21 yay 18:19:22 Tuesday night we moved our backend storage from one netapp to another less loaded one... 18:19:28 hahah 18:19:29 but we have had some issues since then. ;( 18:19:34 boo 18:19:58 It's looking a lot like those issues are related to some virthosts having an emulated realtek network card instead of virtio. 18:20:07 nirik, will those new virthosts need to be added to nagios and the such? 18:20:09 something in the move caused them to start dropping packets like mad 18:20:16 bwood09: they will indeed. ;) 18:20:42 I can file a ticket on them after the meeting. 18:20:45 or smooge can 18:20:47 I'm going to go through today and tomorrow and take care of the nagios stuff, so if you drop a ticket for them in easyfix-- yeah 18:20:49 lol 18:20:51 or really anyone can. ;) 18:21:14 #info storage move had soe issues, but hopefully we have worked them out now. 18:21:20 #info new bvirthosts are on-line 18:21:57 * smooge opens an easyfix ticket that someone can open an easyfix ticket to add monitoring for several hosts 18:22:11 Hello all... i am late... already read previous messages 18:22:19 Also, not sure if this is the place to do this, but I want to get on with the sysadmin-hosted group 18:22:26 * pingou gtg 18:22:33 welcome henderbj 18:22:36 bye pingou 18:22:38 bye pingou 18:23:46 bwood09: what sorts of things do you want to work on there? any tickets in specific? or just adding new projects and such? 18:24:27 we had some plans in there we could look at again and see if you might want to work on them... 18:24:30 I'm going to look at the tickets today and see if there's anything I want to tackle. Recently, most of my experience has been git, svn, hg, and bzr so I figure I'd be a good fit 18:24:52 sure thing. Let me (or any other hosted sponsor know) and we can see about helping you along. 18:24:58 Alrighty 18:25:10 on nagios... we had a lot more alerts this last week I fear... 18:25:21 273 I see since last thursday. 18:25:45 oo 18:25:47 What's the norm for those? 18:25:47 the vast majority of which I think were related in one way or another to the storage move. 18:25:58 this is a fun new routine. :p 18:26:01 damn storage 18:26:32 * nirik looks back at the previous weeks 18:27:18 77 the week before 18:27:34 oh wow 18:28:18 I'd like to reduce them as much as we can... I fear it will be impossible to make them 0 without making them not alert when theres problems users will notice. 18:28:35 well, that's normal... a lot of alarms when someone touches anything! 18:29:13 well, most of the 'normal' ones are network related. We have a very wide network... so if our monitoring host can't reach some datacenter, it alerts. 18:30:07 some of the ones this last week were also from a datacenter where we started to see packet loss... they were being hit by a DDOS. 18:30:12 or where we aren't losing pings but they are taking close to a second to travel around the world 18:30:29 anyhow, if anyone wants to dig thru nagios logs and propose changes that would be lovely. ;) 18:30:53 i will be testing nagios on my own testing machine 18:31:04 we may be able to tune the network related ones down some, but not too far. 18:31:21 When get into it, i will pick something about nagios to help 18:31:42 question..... is there a way in nagios to not try a set of hosts if a "core" host is unreachable due to a network outage? 18:31:44 henderbj: sounds great. Feel free to ask in #fedora-noc or #fedora-admin if you have any questions about our setup 18:31:54 ootbro: yeah, it has dependencies... 18:32:01 Tnx, nirik, sure 18:32:09 I think they should be in pretty good shape now, I revamped them all a while back 18:32:32 so if say virthost01 is down, it will only alert about that, not the vm's running on it also 18:32:42 or a router is down, etc. 18:33:10 https://admin.fedoraproject.org/nagios/ is our main nagios 18:33:22 and https://admin.fedoraproject.org/nagios-external/ is a smaller one we have at a secondary datacenter 18:33:43 anyone should be able to login with their fedora account login/pass 18:34:29 ok, any other sysadmin related stuff? 18:35:09 #topic Upcoming Tasks/Items 18:35:09 https://apps.fedoraproject.org/calendar/list/infrastructure/ 18:35:13 good stuff 18:35:20 anything upcoming anyone would like to schedule or note? 18:35:34 * pingou has none 18:35:44 heh, kinda like a broken record... but we have the bodhi2 FAD upcoming in June 18:35:47 I'd like to note that I will be GONE from saturday until thursday (back late wed night) 18:35:48 just more hardware to install 18:35:54 https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014 18:36:02 nothing new to note.. just reminding that its happening. 18:36:06 #info nirik will be out saturday to next thursday. 18:36:11 oh, during the meeting I pushed the change the 'Manage ACL' page: see http://209.132.184.188/package/guake/acl/commit/ (replace guake by a package you own) 18:36:22 if you need me for anything before then, please find me today/tomorrow. ;) 18:36:43 nirik: if you have specific things you need taken care of while you're gone, feel free to tell us either here or offline. 18:36:43 during that time threebean will technically be in charge but in an undisclosed bunker. I will be available as Alexander Haig 18:37:16 * threebean promotes smooge 18:37:27 can do. ;) I will have cell saturday and wed, but won't even have that the rest of the time. Hurray wilderness! :) 18:37:29 I'm out from Sunday to Saturday next week 18:37:45 I'm probably going to be out for the same ^ 18:37:54 Supposed to be going to Georgia 18:38:00 I'll likely check on emails once in a while, but I'll try to stay away from irc :) 18:38:06 popular vacation week. ;) 18:38:35 nirik, threebean with you and pingou gone.. should we go to warm slush for changes? 18:39:12 well, I'd say to be carefull sure... dunno if we need anything formal 18:39:16 eg changes need at least a IRC +1 from someone else who can review it before commit/push 18:39:25 since I won't have phone, I don't care... can't bother me. ;) 18:39:45 you'll come back and we'll have a chef setup in place 18:39:53 :) 18:40:02 * smooge goes to find his contacts in the Smoke Jumpers to see if they can fix that 18:40:06 anyhow... 18:40:09 #topic Open Floor 18:40:20 anyone have anything for open floor? questions? comments? 18:40:41 nirik yeah I have one 18:40:43 I was able to get into nagios with my regular FP id 18:40:51 just filed https://fedorahosted.org/fedora-infrastructure/ticket/4350 18:40:51 i have one... a moment please 18:41:13 can apprentice members ssh to lockbox01? 18:41:17 colin walters requests a slightly-less ad hoc place to do ostree experimetnation fedora 18:41:39 hi, i get late for the New folks introductions and Apprentice tasks, i'm new and really exited about contributing to the community 18:41:46 mattdm: hum, ok, I already promised walters one of our old virthosts once we move a new one in... is that for this same thing or something different? 18:42:04 nirik I *think* this is the same thing? maybe he is just getting antsy? :) 18:42:15 henderbj: absolutely. See the ssh access link off the apprentice page. ;) 18:42:31 Daredel: welcome! are you interested in sysadmin or application devel or both? 18:42:32 * mattdm did not know about that. or forgot if i did 18:43:03 mattdm: ok. We have been backloged by heartbleed, then virthosts getting shipped the wrong place, then storage hell, etc. We are getting there tho. 18:43:11 i think both, but most of all devel 18:43:21 nirik ok I will relay that. 18:43:34 smooge: did we decide what 2 old virthosts we were going to save? one for ostree the other for cloud lockbox? 18:44:01 Daredel: great. See me after the meeting in #fedora-admin and I can help set you up with the apprentice group... #fedora-apps can help with application devel stuff. :) 18:44:13 I read it before.. but from bastion01 i get: Permission denied (publickey). 18:44:14 ok thanks :D 18:44:42 henderbj, how are you authenticating? And did you upload your public key to FAS? 18:44:54 henderbj: can assist you after the meeting in #fedora-admin, but you should be doing 'ssh lockbox01.phx2.fedoraproject.org' from your home machine, it should use bastion01 as a proxy... 18:45:02 nirik, I have not yet. I keep doing so and then forgetting which 2 I saved and start over 18:45:19 smooge: yeah, we should see if we can hurry on one for ostree stuff. 18:45:43 mattdm: we will try and hurry it along. 18:45:51 nirik thanks. :) 18:45:58 nirik is the previous ticket https://fedorahosted.org/fedora-infrastructure/ticket/4200 ? 18:46:09 I created the ~/.ssh/config file, then ssh to bastion01 , and from there, i did: ssh lockbox01.phx2.fedoraproject.org, and get as response: Permission denied (publickey). 18:46:16 mattdm: could be yeah 18:47:04 henderbj: you can't do it that way.. ;) bastion doesn't (and shouldn't) have your config and keys on it... you should run the 'ssh lockbox01.phx2.fedoraproject.org' from your home machine. The config takes care of the proxying part. 18:47:17 ok... i will trying to connect after the meeting 18:47:50 we will get it working. :) 18:47:51 mattdm, we are having to do a lot of yak shaving to get these boxes available. it may be mid may 18:49:07 anyhow, we will get there as soon as we can. 18:49:23 smooge: lets both go over them and come up with a pair... 18:50:26 ok, anything else? or shall we call it a meeting? 18:51:05 well, about easyfix tickets 18:51:36 sure, shoot... 18:51:37 are those easyfix tickets from 2011-2012 really need any work done? 18:51:50 if they are still open, yes. 18:52:17 they may have been things that weren't urgent enough for someone else to do... 18:52:41 henderbj: if you have one or two in particular in mind, drop a link to them in channel 18:52:56 if they don't need anything anymore, we can close them. ;) 18:53:06 otherwise, I can only guess... 18:53:49 i reviewed this one: https://fedorahosted.org/fedora-infrastructure/ticket/3617 18:54:50 yeah, I'm pretty sure that one still needs work 18:54:50 After my "quick" review, i didn0t find anything to do... i left it because it was too old ;) 18:54:51 yeah, probibly needs the current output added, but I can do that if you want to work on it. ;) 18:56:00 ok, lets all move over to #fedora-admin, #fedora-noc and #fedora-apps... 18:56:04 Ok... if any question i will post it on the ticket to get going to close it 18:56:15 thanks for coming everyone. And welcome again to all the new folks. ;) 18:56:19 henderbj: sounds great. 18:56:22 #endmeeting