18:00:02 <nirik> #startmeeting Infrastructure (2016-03-24)
18:00:02 <zodbot> Meeting started Thu Mar 24 18:00:02 2016 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:02 <zodbot> The meeting name has been set to 'infrastructure_(2016-03-24)'
18:00:02 <nirik> #meetingname infrastructure
18:00:02 <nirik> #topic aloha
18:00:02 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson
18:00:02 <zodbot> The meeting name has been set to 'infrastructure'
18:00:02 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:02 <nirik> #topic New folks introductions / Apprentice feedback
18:00:20 <nirik> welcome all.
18:00:26 * aikidouke here
18:00:33 <nirik> any new folks like to give a 1 line introduction of themselves?
18:00:40 <nirik> or apprentices with questions or comments?
18:01:45 * linuxmodder here and there till about  bottom of hour
18:02:04 * doteast present
18:02:54 <nirik> alright, lets go on to status/info I guess...
18:03:09 <nirik> #topic announcements and information
18:03:09 <nirik> #info Fedora 24 Alpha is go for release next tuesday (2016-03-29) - kevin
18:03:09 <nirik> #info Freeze ends next wed morning (2016-03-30) - kevin
18:03:09 <nirik> #info Friday (2016-03-25) is a holiday in the us, and Monday (2016-03-28) is in the eu - everyone
18:03:09 <nirik> #info Setup cachefilesd on download05, but it didn't seem to help us much - kevin
18:03:10 <nirik> #info Storage upgrade scheduled for the night of 2016-03-29 - kevin
18:03:12 <nirik> Throttling starting late afternoon, then upgrade starting 7pm EDT
18:03:14 <nirik> Upgrade done by around 2am.
18:03:18 <nirik> #info GSoC deadline for applicants is  Tomorrow @ 1900 UTC (thats 1200 US/PST | 1500 US/PST    0030 IST Saturday Morning)
18:03:24 <nirik> anyone have anything else they want to note or share?
18:03:59 <linuxmodder> that should b 1500 us/dst  sorry
18:04:22 <nirik> ok
18:04:35 <nirik> wait... tomorrow? or saturday?
18:05:03 <linuxmodder> tomorrow (Friday the 25th)
18:05:27 <linuxmodder> for those on the far side of the  world  its  0030 am saturday
18:05:39 <nirik> ok.
18:05:57 * nirik just keeps to utc at all times. :)
18:06:14 <linuxmodder> me  too but  some hate to convert manually
18:06:38 <nirik> date -d is your friend. ;)
18:06:42 <nirik> #topic badges-backend cron jobs - aikidouke
18:06:42 <nirik> #info https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/L5B3OIYP6JJTV3LV3ZHWT73P473MKRC2/
18:06:43 <linuxmodder> anyone able and willing to mentor see decause  or myself as well
18:06:54 <aikidouke> thanks nirik
18:06:55 <nirik> aikidouke: you had some questions about this cron ?
18:07:03 <aikidouke> well sort of
18:07:16 <aikidouke> threebean helped me with the question i had about stage
18:07:21 <aikidouke> so I can test in stage
18:07:35 <aikidouke> but I think I will run into a resource issue here as well
18:07:50 <dgilmore> hola
18:07:53 <nirik> so it runs out of memory? or ?
18:08:07 <aikidouke> so If I start getting alerts from tahrir or fas.stg
18:08:14 <puiterwijk> aikidouke: in stg, that's no issue as long as you give a headsup.
18:08:24 <puiterwijk> Staging shouldn't have monitoring with paging
18:08:38 <aikidouke> oh good puiterwijk - do you remember the alarm i got last night?
18:08:39 <nirik> we can give prod more resources if we know how much and what kind. ;)
18:08:50 <puiterwijk> aikidouke: yeah. That was prod though
18:08:55 <aikidouke> right
18:09:05 <puiterwijk> nirik: yeah, that's what I said :)
18:09:19 <nirik> perhaps collectd can tell us...
18:09:20 * nirik looks
18:09:52 <aikidouke> anyway - pulling in ALL of the usernames from fas for running the cron job seems I dunno a bit wasteful I guess, but maybe it wont be so bad once we get caught up
18:10:00 <nirik> https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=badges-backend01.phx2.fedoraproject.org&plugin=memory&timespan=604800&action=show_selection&ok_button=OK
18:10:40 <nirik> cpu looks ok too.
18:10:54 <nirik> so not sure what was the constraint... might be threads on the server or something...
18:11:04 <puiterwijk> nirik: it looked like the database was the issue
18:11:09 <puiterwijk> Since the frontend was alos down
18:11:18 <nirik> hum, odd.
18:11:32 <nirik> it uses db01 right?
18:11:37 <puiterwijk> Yes
18:12:09 <puiterwijk> Well, .... aikidouke: is the script available anywhere? I'd like to check what it runs to see which locks it's trying to get on the database
18:12:26 <nirik> yeah, I see a cpu spike on db01
18:12:27 <aikidouke> ok, that helps
18:12:32 <nirik> https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=db01.phx2.fedoraproject.org&plugin=cpu&timespan=86400&action=show_selection&ok_button=OK
18:12:37 <nirik> around 00:00 utc yesterday?
18:12:49 <nirik> but it wasn't pegged.
18:12:56 <puiterwijk> Yeah, I think that's around the time
18:13:05 <aikidouke> puiterwijk yes, I will create a branch in ansible so you can look at it
18:13:08 <puiterwijk> Right, but things start to timeout
18:13:10 <puiterwijk> aikidouke: thanks
18:13:34 <nirik> ok it might be we could throttle it somehow too...
18:14:14 <aikidouke> i think I will actually put some pauses in the loop when I test it, maybe even break the searches/pulls from fas down further
18:14:44 <nirik> so it gathers a list of all account names, then looks up when they were made, and awards based on that?
18:14:51 <aikidouke> yes
18:16:03 <aikidouke> I think it would probably be more effecient to create a job on the database that drops a list of fedora birthdays and sends it to badges, but I would like to get this bandaid working and promoted first I guess
18:16:33 <nirik> sure.
18:16:41 <aikidouke> anyway - this helps alot - thanks guys
18:16:51 <nirik> there's always ways to make such things better. it may be we can add a index or two to the db also to make it faster.
18:17:13 <nirik> ok, anything else on this?
18:17:24 <aikidouke> cool, if we get through getting caught up and it's still super slow, thats probably the path we should take
18:18:00 <aikidouke> I'm good, thanks for the assistance
18:18:00 <nirik> or once we are caught up perhaps we could change it to have 1 script that awards only the 1st year ones and then another that awards based on how long ago the previous one was awarded.
18:18:09 <nirik> but I guess thats not quite right. oh well, anyhow.
18:18:21 <aikidouke> yeah - there's a better way - we'll find it
18:18:37 <nirik> so, I had puiterwijk down to discuss RFR still, but it's been a crazy few weeks and there's no hurry...
18:18:48 <nirik> puiterwijk: you want to go today or later sometime?
18:18:53 <puiterwijk> aikidouke: I have an idea for a better way
18:19:00 <puiterwijk> nirik: later sometime please
18:19:06 <nirik> sure, no problem.
18:19:11 <aikidouke> i'm all ears
18:19:19 <puiterwijk> aikidouke: how about I give you a script that gives you a csv of username,creationdate?
18:19:33 <nirik> If there's something else people would like to have a learn about session on, please let me know and I can pontificate on it or find someone else who can. ;)
18:19:43 <puiterwijk> That's trivial to dump from the database, without thousands of querys
18:19:59 <aikidouke> yes - that would be great!
18:20:47 <nirik> ok, we have a bunch of things we could discuss, but few people around today... so that would take us to...
18:20:50 <nirik> #topic Open Floor
18:21:09 <nirik> anyone have any ideas/suggestions/nerf gun recommendations?
18:21:50 <puiterwijk> "Stay away from kernel source unless you want to waste entire weekends on it"?
18:21:57 <nirik> ha
18:22:10 <aikidouke> let it snow?
18:22:26 <nirik> it seems to be melting actually.
18:23:04 <aikidouke> :)
18:23:12 <nirik> hey cool... another under 30min meeting. ;)
18:23:28 <nirik> ok, everyone continue in #fedora-admin, #fedora-noc and #fedora-apps
18:23:30 <nirik> #endmeeting