18:00:07 <nirik> #startmeeting Infrastructure (2015-07-23)
18:00:08 <zodbot> Meeting started Thu Jul 23 18:00:07 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:08 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:08 <nirik> #meetingname infrastructure
18:00:08 <zodbot> The meeting name has been set to 'infrastructure'
18:00:08 <nirik> #topic aloha
18:00:08 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson
18:00:08 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:08 <nirik> #topic New folks introductions / Apprentice feedback
18:00:36 * Corey84 sneaks in
18:00:39 <p_klos> I would like to introduce myself,so Hi :) My name is Pawel. I live in Warsaw, Poland. I work as a developer (C, C++, Java, Python, HTML, CSS, JS) and admin/DBA (RedHat, CentOS, Fedora, Oracle DB, PostgreSQL, LDAP, RADIUS). I use Fedora since Werewolf and would be happy to be a part of the team and to contribute to the Fedora future ;-)
18:00:52 * puiterwijk is here, but can get called back by Dell any moment
18:00:54 <nirik> might be a quick meeting today, as many folks are on vacation or otherwise out or busy. ;)
18:00:56 <smooge> here
18:01:06 <nirik> p_klos: welcome!
18:01:17 <Corey84> puiterwijk,  fun times  sounds like my day
18:01:23 <sonalkr132> @nirik?
18:01:26 <p_klos> nirik: Welcome Nirik!
18:01:40 <nirik> p_klos: are you more interested in our sysadmin work or application development side of things? or both? :)
18:01:45 <nirik> sonalkr132: ?
18:01:56 <p_klos> nirik: I can do both :)
18:02:14 <Corey84> p_klos,  which do you / would you prefer is the ?
18:02:32 <sonalkr132> nirik Sorry. I didn't see any thing for a while, thought no one is here :P Must be lag.
18:02:48 <Corey84> trying to do both is likely  a trainwreck schedule wise
18:02:54 <p_klos> Corey84, I would prefer development but feel comfortable with sysadmin too ;)
18:02:59 <nirik> sonalkr132: no worries. freenode has been a bit netsplitty of late. ;(
18:03:11 <Corey84> a bit? :p
18:03:21 <Corey84> 4  today for me alone
18:03:23 <nirik> p_klos: cool. the fine folks in #fedora-apps can help you get started. There's tons of things to look at.
18:03:24 <Corey84> since 4am
18:03:49 <nirik> anyone else new like to introduce themselves? or apprentices with questions? if not will move on to GSoC checkins...
18:04:07 <Corey84> been working with Southern_Gentlem  and the gang  on torrents of late  looking to get back in saddle on -docs too
18:04:08 <p_klos> nirik, thanks. I will join #fedora-apps. Is there anything I can help you with the infrastructure?
18:04:53 <nirik> p_klos: likely. ;) If you are interested in the sysadmin side see me in #fedora-admin after the meeting and I can get you started.
18:05:00 <Corey84> p_klos,  you are on the mailing list yes?  if so check the easy_fix list  see if anything fits skills / time allowances
18:05:08 <nirik> #topic GSoC student update - kushal
18:05:19 <nirik> kushal: you around? any GSoC folks like to check in on their work this week?
18:05:34 <p_klos> nirik, ok, I'll see you after meeting
18:05:34 <dgilmore> hey all
18:05:46 <p_klos> Corey84, yes, I am on the list :)
18:05:58 <sonalkr132> On glittergallery, I added keys model, made changes to web ui so that users can add keys to their profile and added an api for git shell to use.
18:06:06 <AnuradhaW> Hi all, I have done some integration with the main and badges page of AskFedora testing instance which you can see some screen shots here: http://anuradhanotes.blogspot.com/2015/07/feedback-from-fudcon-and-progress-with.html
18:06:07 <sonalkr132> #link https://chasingcrazydreams.wordpress.com/2015/07/19/git-over-ssh/
18:06:29 <AnuradhaW> and have also gone through the feedback from FUDCon.
18:06:31 <nirik> nice. :)
18:06:35 <Corey84> if not  before the august  monthly  push to all,   check the list of  easy_fix stuff  p_klos
18:06:43 <nirik> sonalkr132: is the cloud instance still working well for your testing?
18:06:52 <prth> hi nirik, I've implemented server side cropping for nuancier this week
18:07:21 <nirik> cool
18:07:33 <Corey84> prth,  nuancier?
18:07:54 <nirik> Corey84: it's our wallpaper voting application.
18:07:55 <sonalkr132> nirik: I haven't done much on it since last week. I need do the implementation first. I guess I can update you about it next week.
18:08:06 <nirik> sonalkr132: no worries, just wondered
18:08:11 <prth> Corey84, it's a python flask app for fedora's supplementary wallpaper election
18:08:16 <Corey84> never see my desktop makes sense i didnt know  it
18:08:38 <prth> thanks nirik
18:09:14 <AnuradhaW> But I have met up with an error when I access the site as an anonymous user which is also stated in my blog post. This error does not come when I am logged into the site. Do you know anyone who can help me with this?
18:10:00 <Corey84> AnuradhaW,  mind  posting a snippet  somewhere I'll take a look if oyu want
18:10:07 <nirik> AnuradhaW: not sure off hand I am afraid. perhaps puiterwijk might know, but he's pretty busy...
18:10:54 <puiterwijk> AnuradhaW: just ping me aftert the meeting and I'll see what I can do
18:11:10 <nirik> ok, any other GSoC checkins?
18:11:12 <^conner> (sorry for missing the call for introductions) Hello! My name is Josh and I'm interested in both some general SA work and infrastructure development.  I live in Tucson so I'm @UTC-7 year round.  I did send an intro email to the list but my background includes sysadmin, systems programming, a lot of storage, some time as a network engineer and these days I'm called a "devops engineer"
18:11:30 <nirik> welcome ^conner. ;)
18:11:31 <AnuradhaW> Corey84: It says UndefinedError at /badges/  'django.contrib.auth.models.AnonymousUser object' has no attribute 'is_read_only' when try to access the site http://askbotfedoratest-anuradhaw.rhcloud.com/
18:11:54 <Corey84> will look as well later
18:12:04 <nirik> ^conner: I can also get you setup for the apprentice group/sysadmin stuff after the meeting over in #fedora-admin if you like...
18:12:18 <AnuradhaW> puiterwijk: Okay, thank you :)
18:12:19 <^conner> nirik, roger
18:12:31 <nirik> ok, on to announcements/info dump
18:12:34 <nirik> #topic announcements and information
18:12:34 <nirik> #info cloud migration moving along, possibly done end of next week - kevin/patrick
18:12:35 <nirik> #info mass updates/reboots done. kevin/patrick/relrod/smooge/tflink
18:12:35 <nirik> #info openstack update again a nightmare, but patrick got it working - patrick
18:12:35 <nirik> #info production bodhi2 instances ready to go - kevin
18:12:36 <nirik> #info flock coming up soon, be thinking of things you want to bring up there - kevin
18:12:47 <Corey84> not as bad as i expected
18:12:59 <nirik> anything there anyone would like to discuss or get more info or items that wen't added.
18:12:59 <smooge> it was the best of times.. it was the worst of times
18:13:19 <smooge> patrick is a hero
18:13:30 <nirik> yep. :)
18:13:35 <Corey84> what was up with openstack
18:13:41 <^conner> I do have some minor openstack experience
18:13:42 <nirik> We really should try and avoid the epic 14 hour openstack fixes tho
18:13:44 <^conner> that might be something I can help with
18:13:47 <smooge> that is all I have to say which can be said without my tourets syndrome filter kicking in
18:13:59 <AadityaNair> I wanted to know, is implementing HSTS still open ?
18:14:00 <nirik> it's very fragile
18:14:21 <nirik> or as someone recently described it: "a bag of razor blades"
18:14:25 <puiterwijk> Corey84: I can explain it more after the meeting, but it was annoying
18:14:43 <nirik> AadityaNair: yeah, we are still moving forward on that. There's an open ticket still.
18:14:54 <Corey84> puiterwijk, looked it  sure  may be better in email form today tho tbh  busy as crap myself
18:15:03 <puiterwijk> basically, Neutron first didn't create tunnels, and then it didn't create flow rules. The causes are interresting
18:15:03 <^conner> is this rdo or seperate packaging?
18:15:07 <Corey84> .fas corey84
18:15:08 <zodbot> Corey84: corey84 'Corey Sheldon' <sheldon.corey@gmail.com>
18:15:15 <Corey84> to that one puiterwijk
18:15:15 <puiterwijk> ^conner: as of last week, we are using RHOS
18:15:19 <nirik> ^conner: it was rdo, but we switched to rhos
18:15:20 <puiterwijk> (Red Hat OpenStack Platform)
18:15:23 <AadityaNair> nirik: I would like to work on that.
18:15:39 <^conner> I saw that RH has some rdo packaging jobs open ;)
18:15:49 <nirik> AadityaNair: I can dig up the ticket after the meeting. ;) tyll has been doing most of the work there, but we could definitely use more checking and patches. ;)
18:16:08 <nirik> ok, we had one item up for discussion:
18:16:16 <nirik> #topic Centralized logging - jcvicelli
18:16:17 <nirik> https://fedorahosted.org/fedora-infrastructure/ticket/3627
18:16:17 <nirik> looks like it was all done, should we close this ticket?
18:16:22 <nirik> jcvicelli: you around?
18:16:33 <nirik> I think this may be done, but I am not 100% sure...
18:16:40 <nirik> lmacken: you around? is the above done ^
18:17:30 <Corey84> looks  ready to close  at  quick glance
18:17:50 <lmacken> nirik: hmm
18:17:59 <lmacken> as far as shipping app logs over to log01, I think so
18:18:11 <lmacken> actually, things like badges-web/backend aren't doing it afaik
18:18:12 <jcvicelli> What do i miss
18:18:17 <nirik> yeah, I know we have a apps.log there, but not sure if everything is using it. ;)
18:18:17 <lmacken> so we might need to do a once-over again
18:18:49 <Corey84> can look at  some of the  ask stuff
18:19:01 <lmacken> we may want to look at https://apps.fedoraproject.org and make sure most of them are shipping them over
18:19:04 <Corey84> maybe tagger too
18:19:14 <lmacken> there are definitely a few stragglers...
18:19:55 <nirik> ok, so we should check apps.log against apps.fedoraproject.org and see whats missing?
18:20:20 <lmacken> sounds good. apps.log might be a bit crazy, so it may be easier to look at the logs for each host?
18:20:22 <nirik> I wish we had a better / complete list of applications... apps I don't think has everything
18:20:26 <nirik> but it might be close I guess
18:21:26 <nirik> lmacken: one problem with checking httpd logs... you need sudo.
18:21:38 <jcvicelli> Yep
18:21:43 <Corey84> not many have that right?
18:22:00 <jcvicelli> I don't
18:22:04 <Corey84> i know I  dont at least
18:22:27 <nirik> there was a thought a while back making them readable to all apprentices, but we never figured out a sane way to do that.
18:22:50 <Corey84> not sure sadly there would be a "sane" way
18:22:51 <smooge> nirik, I was expecting that the rsync script which gets it over could od it?
18:23:15 <Corey84> smooge,  how so?
18:23:19 <lmacken> yeah, not sure about apache logs... a lot of tracebacks bubble up to the error_log I'm guessing
18:23:26 <nirik> smooge: well, for the things we sync httpd logs for... but thats a smaller subset of all machines.
18:23:44 <smooge> Corey84, it sort of knows which directories it copied over so at the end it just does a
18:23:57 <smooge> chmod 644 $date/$files
18:24:09 <^conner> could dump it into an ELK stack
18:24:09 <Corey84> im familiar with rsync  just not what you were getting at
18:24:19 <^conner> that just pushes the privs problem around though
18:24:36 <smooge> it would require us to build an ELK stack also :)
18:24:40 <nirik> ^conner: then we have an ELK stack. ;)
18:25:04 <Corey84> layers and layers  fun times
18:25:18 <nirik> anyhow, I think the best for now would be check apps.fedoraproject.org apps against the apps.log on log01 and anything missing from there needs fixing.
18:26:05 <jcvicelli> Ok
18:26:12 <nirik> sound reasonable?
18:26:13 <Corey84> sounds good to me  I'll try to  get eyes on what i can over weekend and check
18:26:58 <nirik> do coodinate, so people aren't checking whats already been checked. ;)
18:27:06 <Corey84> ofc
18:27:09 <nirik> perhaps a list in ticket and people could say which ones they are checking?
18:27:27 <jcvicelli> Sounds good
18:27:40 <Corey84> seems reasonable
18:27:41 <nirik> I think we can scrape it from the .json from apps.
18:27:54 <Corey84> have to go there anyway to see what's outstanding anywya
18:28:14 <Corey84> brb
18:28:23 <nirik> #topic Open Floor
18:28:41 <nirik> so, we didn't have any other discssion items and I didn't line up someone to teach today...
18:28:45 <nirik> which brings us to open floor.
18:29:17 <nirik> I was thinking instead of or alternating with the learning topic, we could pick a SOP and fix it up each week.
18:29:26 <nirik> our documentation is pretty out of date in places.
18:29:39 <nirik> but I can propose this on the list and we can try it next week if people want to
18:30:24 <smooge> ok cool
18:31:03 <nirik> anything else? If not, will close out in a minute.
18:31:11 <smooge> not from me.. need coffee
18:31:35 <smooge> jcvicelli, where do you hang out I would like to talk to you about the centralized logging at some point.. when I am more awake
18:31:45 <Corey84> im  good
18:31:57 <jcvicelli> Sure
18:32:18 <jcvicelli> Im on admin and apps
18:34:02 <nirik> ok, thanks for coming everyone!
18:34:04 <nirik> #endmeeting