19:00:17 #startmeeting Infrastructure (2011-09-01) 19:00:17 Meeting started Thu Sep 1 19:00:17 2011 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:17 Useful Commands: #action #agreed #halp #info #idea #link #topic. 19:00:17 #meetingname infrastructure 19:00:17 The meeting name has been set to 'infrastructure' 19:00:17 #topic Robot Roll Call 19:00:17 #chair smooge skvidal codeblock ricky nirik abadger1999 lmacken 19:00:17 Current chairs: abadger1999 codeblock lmacken nirik ricky skvidal smooge 19:00:23 * abadger1999 here 19:00:40 * athmane is kinda here 19:01:06 here 19:01:26 * herlo here 19:01:37 * mahrud here 19:01:49 * ricky listens in 19:02:15 cool. Lets go ahead and dive in. 19:02:22 #topic New folks introductions and apprentice tasks/feedback 19:02:43 any new folks want to say hi? or apprentice folks want to talk about/look at any tickets or feedback? 19:03:27 guess not. ;) 19:03:39 #topic Upcoming new machine plans. 19:03:49 I posted some upcoming new machine/migration plans to the list. 19:03:59 here 19:04:17 Feedback welcome there. Assistance building machines from sysadmin-main folks welcome. ;) 19:04:31 I am planning on trying to migrate bastion02 and db02 next week. 19:04:39 I will be sending out an outage notice on that. 19:04:49 * CodeBlock was going to look at working on some of those this afternoon with smooge 19:05:06 I'm not sure if it makes sense to do the outage early in the day, or later at night after us folks are gone home. 19:05:16 CodeBlock: appreciated. ;) 19:06:19 On the new releng boxes I am waiting for a few tickets to get done... we need to move bvirthost04's vlan on secondary interface, and the netapp needs to allow them to mount it. 19:06:30 once those get done, we can migrate some releng stuff too. 19:07:49 anyone have thoughts on the db02/bastion02 outage timing? It will basically take most things out, as it takes down the vpn and fas. 19:08:48 is this thing on? :) 19:09:30 * skvidal is here 19:09:31 sorry 19:09:33 isn't there any backup for bastion02? 19:09:37 * skvidal had someone at the dor :) 19:09:39 err door, even :) 19:10:11 nirik: you said the problem is with selinux, right? 19:10:32 sure, we can go to bastion01... it would still be a blip in vpn tho. 19:10:38 mahrud: which problem? 19:10:49 but if we are migrating db02, fas will be down. 19:11:21 not sure, i think you said some host has problem with selinux ... 19:11:31 and you wanted too reboot it 19:11:36 oh, fas01.stg? that is fixed. ;) 19:11:46 aha, ok 19:12:12 I guess I should do a dump/reload on db01 of db02 content and see how long it takes. :) 19:12:42 #topic Upcoming Tasks/Items 19:12:54 Any other upcoming items folks are working on or want to talk about. 19:13:08 note that the beta freeze is coming up: 19:13:10 2011-09-13 - 27: Beta change freeze 19:13:24 nirik: between, I want to work on ticket 1084, but not sure about what are some hosts doing exactly, also, not sure about which hosts do we have! 19:13:40 I'm going to be upgrading python-fedora soon -- lots of changes to the fas auth providers for both TG1 and TG2 apps. 19:13:42 .ticket 1084 19:13:45 nirik: #1084 (Fix proxy -> app docs) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1084 19:14:29 mahrud: yeah, I cleaned up logs on log02, so that should look much nicer. ;) as for the apps/proxies you will have to dig around some and ask questions... 19:14:39 and thanks for looking at updating those docs. 19:15:06 abadger1999: cool. when might this land? before freeze? 19:15:29 nirik: I'm going to try to push to stg tomorrow (possibly tonight) and prod on Monday. 19:15:44 nirik: I need the update for the raffle app which I'm trying to deploy to prod before freeze. 19:16:02 cool. 19:16:07 nirik: thanks for cleaning log02, it much better, but yet if you change a hostname or ... you'll need to update log02 19:16:10 lmacken: you were going to push out a bodhi update soon too? 19:16:21 nirik: i was thinking about some script based on ip addresses 19:16:50 well, as long as we change hosts / add new ones consistently moving forward we should be fine. 19:17:02 nirik: this is another place where I think we need to be autogenerating some info for other hosts based on the info in puppet/func 19:17:03 There are also some hosts not logging to log02 yet, which we need to add 19:17:25 nirik: we have vpn on 192.168.0.0/16 (right?), and --i think-- phx2 hosts are on 10.5.125-126-127.0/24 19:17:41 nirik: I'm not sure, yet, how we should go about deploying autogenerated information to other systems off of lockbox01, though 19:17:48 nirik: yeah, i'm trying to wrap up a couple of fixes and will hopefully push a new release out soonish 19:18:01 mainly b/c there is no way to nicely automatically check in something to puppet as a non-user 19:18:40 mahrud: yep. 192.168.x.x is vpn. 10.5.125.0/24 is builder network, 10.5.126.0/24 is main network, 10.5.127.0/24 is storage network, 10.5.124.0/24 is qa/community network. 19:18:58 nirik: then we can write a script to scan the net and report online hostnames 19:19:01 yeah, we should leverage infra-hosts as much as we can. ;) 19:19:14 mahrud: well, sure, but lets back up a second. 19:19:57 I think we should use infra-hosts for this... should it not contain all hosts? 19:20:04 nirik: it does, yes 19:20:12 nirik: but again -we have to check things in there, too, right? 19:20:23 yeah, true. 19:20:27 nirik: oh you were replying to mahrud 19:20:32 so I have a couple of thoughts on that 19:20:38 we could use infrahosts as we have been 19:20:40 yes, but I agree if it was automated it could be nice too. ;) 19:20:49 but generate some lists to another location 19:20:52 that is still a git repo 19:20:55 in case of any hostname change, we need something to automatically list hosts 19:21:00 nirik: is 10.5.*.* only for phx2 or all of them? 19:21:05 but t is just a single committer 19:21:08 so we can tell what changes 19:21:16 but not change it ourselves 19:21:19 if you see what I mean 19:21:19 mahrud: only phx2. 19:21:51 well, it seems like we are creating lots of repos... more means it's difficult to remember which one to do what in... 19:22:07 we really shouldn't be renaming stuff much if at all should we? 19:22:49 nirik: right 19:22:55 I'm not suggesting we make this a new repo 19:22:56 for US 19:23:10 I'm saying we make it so a root/system process commits changes to it 19:23:17 every hour 19:23:38 ok, as a dump of func hosts? 19:23:59 skvidal: you mean an script to commit its changes to repo? 19:24:12 mahrud: yes 19:24:18 the point of the commit 19:24:21 on each host? 19:24:23 is just so we can walk back through waht it changed 19:24:27 that sounds nice ... 19:24:29 not _on_ each host no 19:24:41 it would be located on lockbox01 19:24:42 hmm, so where? 19:24:47 into infa-hosts? :) 19:25:09 nirik: right - that's the problem 19:25:15 it is sorta like this 19:25:18 hmm 19:25:22 maybe we can do this with infra hosts 19:25:23 hmmm can someone see whats up with fedorapeople.org 19:25:36 Southern_Gentlem: looks fine from here 19:26:02 nirik: I really just want a 'space where info is autogenerated' and a 'space where admins edit things' that has the same list of hosts 19:26:19 I know! lets make it branches in the same repo! 19:26:21 * nirik runs away 19:27:21 nirik: you know how to hurt a guy, don't you? 19:27:37 anyhow, lets discuss this out of meeting? and come up with a plan/ 19:27:40 yah 19:27:42 sounds fine w/me 19:27:46 we could do it like virt-hosts 19:27:46 ok 19:27:57 hosts and it mails when it changes. 19:28:16 nirik: nod - thats sorta what I was thinking - but we could put the results into either a repo or into a path accessible on infrastructure.fp.o 19:28:25 nirik: so then other hosts could use/mine the information for their own processes 19:28:52 yeah. 19:29:11 ok, any other upcoming work people are looking at? 19:29:28 ow 19:29:36 nirik: about that certificate ... 19:29:48 mahrud: the koji one? 19:29:54 yeah 19:30:16 I'd prefer if we get dgilmore to change that and pkgs... just in case there are things we are not thinking of. ;) 19:30:35 what was the ticket # on that one? 19:30:54 umm 19:31:04 .ticket 1929 19:31:06 nirik: #1929 (https://koji.fedoraproject.org server certificate is signed with MD5) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/1929 19:31:30 yeah, I will see what we can do there. If you could add your testing and thoughts to the ticket that would be great. 19:31:47 ok 19:32:09 #topic Meeting tagged tickets: 19:32:10 https://fedorahosted.org/fedora-infrastructure/report/10 19:32:17 I cleaned up our meeting tagged tickets. 19:32:21 we currently have 0. ;) 19:32:36 if anyone has a specific ticket they want to bring up moving forward, add the 'meeting' keyword to it. 19:33:33 #topic Open Floor 19:33:39 Anyone have anything for open floor? 19:33:54 I opened a new ticket for fi-apprentice to look at. 19:34:03 Just wanted to check that it's what we want to happen. 19:34:11 abadger1999: did you stick the easyfix keyword on it? 19:34:27 the new_repo script for fedorapeople repos take a "group or user to own the repo" 19:34:32 nirik: yeah, I did 19:34:39 cool. 19:34:42 https://fedorahosted.org/fedora-infrastructure/ticket/2931 19:34:58 currently, that's just a freeform string. 19:35:03 easyfix 19:35:07 not EasyFix :D 19:35:09 cool. 19:35:17 I think we should make that confirm that it's either a username on fedorapeople or a group on fedorapeople. 19:35:28 Sound good to everyone else? 19:35:44 yep. Sounds good to me. 19:36:05 +1 19:37:01 * abadger1999 updates ticket. 19:37:02 ok, anything else? or shall we close on up and get back to work? 19:37:12 hmm 19:37:16 before that 19:37:16 oh, FYI, all servers should have the updated httpd and have been restarted... 19:37:41 oh, cool 19:37:48 can I ask to put audit log on log02 too? 19:38:12 is audit logged via syslog? 19:38:19 I thought it logged directly for some reason 19:38:29 mahrud: well, I think we have talked about that... but yeah, it does it's own logging. 19:38:44 but I agree it would be good to get going on there too... 19:38:49 no it isn't via syslog, but there must be some way ... 19:39:53 in the worst case, a script to read it and send it with nc should work :) 19:39:54 yeah, I think it's possible/doable... we can look into doing so. 19:40:50 #info look into audit logging to log02 19:41:23 also, I think I might like a log03 to have a sync/backup copy of all logs thats ro/locked down. Just to have another copy in the audit trail. ;) 19:42:01 anyhow, thanks for coming everyone! 19:42:22 nice meeting. 19:42:27 short and sweet 19:42:41 oh, hey herlo. Any news on paste? ;) 19:42:53 and I forgot news on ask. 19:43:04 #topic quick RFR roundup 19:43:20 #info ask is making some last minute packaging changes to make it easier to deploy. 19:43:40 #info There's a puppet commit I have been getting ready to add ask01.stg once thats done. 19:43:43 nirik: no, been crazy busy the past couple weeks. I will probably have some time in sept though 19:44:00 no worries at all. 19:44:19 ok, thanks again for coming everyone. 19:44:22 #endmeeting