18:00:02 <nirik> #startmeeting Infrastructure (2016-03-24) 18:00:02 <zodbot> Meeting started Thu Mar 24 18:00:02 2016 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:02 <zodbot> The meeting name has been set to 'infrastructure_(2016-03-24)' 18:00:02 <nirik> #meetingname infrastructure 18:00:02 <nirik> #topic aloha 18:00:02 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson 18:00:02 <zodbot> The meeting name has been set to 'infrastructure' 18:00:02 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:02 <nirik> #topic New folks introductions / Apprentice feedback 18:00:20 <nirik> welcome all. 18:00:26 * aikidouke here 18:00:33 <nirik> any new folks like to give a 1 line introduction of themselves? 18:00:40 <nirik> or apprentices with questions or comments? 18:01:45 * linuxmodder here and there till about bottom of hour 18:02:04 * doteast present 18:02:54 <nirik> alright, lets go on to status/info I guess... 18:03:09 <nirik> #topic announcements and information 18:03:09 <nirik> #info Fedora 24 Alpha is go for release next tuesday (2016-03-29) - kevin 18:03:09 <nirik> #info Freeze ends next wed morning (2016-03-30) - kevin 18:03:09 <nirik> #info Friday (2016-03-25) is a holiday in the us, and Monday (2016-03-28) is in the eu - everyone 18:03:09 <nirik> #info Setup cachefilesd on download05, but it didn't seem to help us much - kevin 18:03:10 <nirik> #info Storage upgrade scheduled for the night of 2016-03-29 - kevin 18:03:12 <nirik> Throttling starting late afternoon, then upgrade starting 7pm EDT 18:03:14 <nirik> Upgrade done by around 2am. 18:03:18 <nirik> #info GSoC deadline for applicants is Tomorrow @ 1900 UTC (thats 1200 US/PST | 1500 US/PST 0030 IST Saturday Morning) 18:03:24 <nirik> anyone have anything else they want to note or share? 18:03:59 <linuxmodder> that should b 1500 us/dst sorry 18:04:22 <nirik> ok 18:04:35 <nirik> wait... tomorrow? or saturday? 18:05:03 <linuxmodder> tomorrow (Friday the 25th) 18:05:27 <linuxmodder> for those on the far side of the world its 0030 am saturday 18:05:39 <nirik> ok. 18:05:57 * nirik just keeps to utc at all times. :) 18:06:14 <linuxmodder> me too but some hate to convert manually 18:06:38 <nirik> date -d is your friend. ;) 18:06:42 <nirik> #topic badges-backend cron jobs - aikidouke 18:06:42 <nirik> #info https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/L5B3OIYP6JJTV3LV3ZHWT73P473MKRC2/ 18:06:43 <linuxmodder> anyone able and willing to mentor see decause or myself as well 18:06:54 <aikidouke> thanks nirik 18:06:55 <nirik> aikidouke: you had some questions about this cron ? 18:07:03 <aikidouke> well sort of 18:07:16 <aikidouke> threebean helped me with the question i had about stage 18:07:21 <aikidouke> so I can test in stage 18:07:35 <aikidouke> but I think I will run into a resource issue here as well 18:07:50 <dgilmore> hola 18:07:53 <nirik> so it runs out of memory? or ? 18:08:07 <aikidouke> so If I start getting alerts from tahrir or fas.stg 18:08:14 <puiterwijk> aikidouke: in stg, that's no issue as long as you give a headsup. 18:08:24 <puiterwijk> Staging shouldn't have monitoring with paging 18:08:38 <aikidouke> oh good puiterwijk - do you remember the alarm i got last night? 18:08:39 <nirik> we can give prod more resources if we know how much and what kind. ;) 18:08:50 <puiterwijk> aikidouke: yeah. That was prod though 18:08:55 <aikidouke> right 18:09:05 <puiterwijk> nirik: yeah, that's what I said :) 18:09:19 <nirik> perhaps collectd can tell us... 18:09:20 * nirik looks 18:09:52 <aikidouke> anyway - pulling in ALL of the usernames from fas for running the cron job seems I dunno a bit wasteful I guess, but maybe it wont be so bad once we get caught up 18:10:00 <nirik> https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=badges-backend01.phx2.fedoraproject.org&plugin=memory×pan=604800&action=show_selection&ok_button=OK 18:10:40 <nirik> cpu looks ok too. 18:10:54 <nirik> so not sure what was the constraint... might be threads on the server or something... 18:11:04 <puiterwijk> nirik: it looked like the database was the issue 18:11:09 <puiterwijk> Since the frontend was alos down 18:11:18 <nirik> hum, odd. 18:11:32 <nirik> it uses db01 right? 18:11:37 <puiterwijk> Yes 18:12:09 <puiterwijk> Well, .... aikidouke: is the script available anywhere? I'd like to check what it runs to see which locks it's trying to get on the database 18:12:26 <nirik> yeah, I see a cpu spike on db01 18:12:27 <aikidouke> ok, that helps 18:12:32 <nirik> https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=db01.phx2.fedoraproject.org&plugin=cpu×pan=86400&action=show_selection&ok_button=OK 18:12:37 <nirik> around 00:00 utc yesterday? 18:12:49 <nirik> but it wasn't pegged. 18:12:56 <puiterwijk> Yeah, I think that's around the time 18:13:05 <aikidouke> puiterwijk yes, I will create a branch in ansible so you can look at it 18:13:08 <puiterwijk> Right, but things start to timeout 18:13:10 <puiterwijk> aikidouke: thanks 18:13:34 <nirik> ok it might be we could throttle it somehow too... 18:14:14 <aikidouke> i think I will actually put some pauses in the loop when I test it, maybe even break the searches/pulls from fas down further 18:14:44 <nirik> so it gathers a list of all account names, then looks up when they were made, and awards based on that? 18:14:51 <aikidouke> yes 18:16:03 <aikidouke> I think it would probably be more effecient to create a job on the database that drops a list of fedora birthdays and sends it to badges, but I would like to get this bandaid working and promoted first I guess 18:16:33 <nirik> sure. 18:16:41 <aikidouke> anyway - this helps alot - thanks guys 18:16:51 <nirik> there's always ways to make such things better. it may be we can add a index or two to the db also to make it faster. 18:17:13 <nirik> ok, anything else on this? 18:17:24 <aikidouke> cool, if we get through getting caught up and it's still super slow, thats probably the path we should take 18:18:00 <aikidouke> I'm good, thanks for the assistance 18:18:00 <nirik> or once we are caught up perhaps we could change it to have 1 script that awards only the 1st year ones and then another that awards based on how long ago the previous one was awarded. 18:18:09 <nirik> but I guess thats not quite right. oh well, anyhow. 18:18:21 <aikidouke> yeah - there's a better way - we'll find it 18:18:37 <nirik> so, I had puiterwijk down to discuss RFR still, but it's been a crazy few weeks and there's no hurry... 18:18:48 <nirik> puiterwijk: you want to go today or later sometime? 18:18:53 <puiterwijk> aikidouke: I have an idea for a better way 18:19:00 <puiterwijk> nirik: later sometime please 18:19:06 <nirik> sure, no problem. 18:19:11 <aikidouke> i'm all ears 18:19:19 <puiterwijk> aikidouke: how about I give you a script that gives you a csv of username,creationdate? 18:19:33 <nirik> If there's something else people would like to have a learn about session on, please let me know and I can pontificate on it or find someone else who can. ;) 18:19:43 <puiterwijk> That's trivial to dump from the database, without thousands of querys 18:19:59 <aikidouke> yes - that would be great! 18:20:47 <nirik> ok, we have a bunch of things we could discuss, but few people around today... so that would take us to... 18:20:50 <nirik> #topic Open Floor 18:21:09 <nirik> anyone have any ideas/suggestions/nerf gun recommendations? 18:21:50 <puiterwijk> "Stay away from kernel source unless you want to waste entire weekends on it"? 18:21:57 <nirik> ha 18:22:10 <aikidouke> let it snow? 18:22:26 <nirik> it seems to be melting actually. 18:23:04 <aikidouke> :) 18:23:12 <nirik> hey cool... another under 30min meeting. ;) 18:23:28 <nirik> ok, everyone continue in #fedora-admin, #fedora-noc and #fedora-apps 18:23:30 <nirik> #endmeeting