18:00:00 <nirik> #startmeeting Infrastructure (2016-04-14) 18:00:00 <zodbot> Meeting started Thu Apr 14 18:00:00 2016 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:00 <zodbot> The meeting name has been set to 'infrastructure_(2016-04-14)' 18:00:01 <nirik> #meetingname infrastructure 18:00:01 <zodbot> The meeting name has been set to 'infrastructure' 18:00:01 <nirik> #topic aloha 18:00:01 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson 18:00:01 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean 18:00:01 <nirik> #topic New folks introductions / Apprentice feedback 18:00:14 * pingou here 18:00:47 <nirik> Any new folks want to give a short (one line) introduction? Or any apprentices with questions or comments? 18:01:01 * puiterwijk here but traveling to home 18:01:16 * tflink here 18:01:16 <cverna> yeah 18:01:19 * doteast present 18:01:21 * aikidouke here 18:01:24 <cverna> hi everyone 18:01:36 <cverna> I have started playing with python begining of the year and did a few easyfix on pagure. I would like now to get a bit more involved within the infra team mostly development side at the begining 18:01:37 <linuxmodder> .fas corey84 18:01:37 <zodbot> linuxmodder: corey84 'Corey Sheldon' <sheldon.corey@gmail.com> 18:02:08 <nirik> cverna: welcome! 18:02:24 <nirik> lots of easyfixes to choose from. :) 18:02:24 <linuxmodder> cverna, packaging wise or testing? 18:02:55 <cverna> bit of both I guess :) 18:03:00 * pingou waves to cverna 18:03:35 <cverna> I haven't had much experience in packaging 18:04:15 <linuxmodder> luckily for you we have great packagers that mentor :) 18:04:35 <nirik> yep. Always fun. ;) 18:04:43 <nirik> any other new folks? or apprentices with questions? 18:04:46 <gkadam> nirik, could you please help with https://fedoraproject.org/easyfix #4485 run POC-Change script for EPEL ? 18:04:58 <linuxmodder> ! something for open floor 18:05:08 * sayan is here 18:05:15 <nirik> .ticket 4485 18:05:17 <zodbot> nirik: #4485 (run POC-Change script for EPEL) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/4485 18:06:00 <nirik> ok, so pingou might be the best one to ask about that... and/or on ticket... 18:06:36 <nirik> https://github.com/pypingou/fedora-owner-change/blob/master/fedora-owner-change.py would be the thing to change 18:06:44 <nirik> right now it only reports fedora changes 18:07:22 <linuxmodder> not schooled enough in epel to know what to change but cverna I can help if oyu need help 18:07:35 <pingou> I'm thinking we could had a CLI flag to do EPEL as well 18:07:48 <pingou> do epel or fedora or both actually 18:07:51 <linuxmodder> pingou, what as a var or something 18:08:22 <gkadam> nirik, may ask a bit of more basic qestion ? 18:08:31 <gkadam> *question? 18:08:35 <pingou> linuxmodder: as fedora-owner-change.py --target=epel or so 18:08:41 <nirik> I'd be fine with just both and sending to eppel-devel too, or more specific and sending to each, either way 18:08:53 <cverna> linuxmodder: thx 18:08:54 <nirik> gkadam: absolutely. ;) 18:08:59 <nirik> ask away 18:09:02 <pingou> nirik: then we can drop the flag I guess :) 18:09:39 <nirik> pingou: I think a number of epel maintainers are not on epel-devel but are on fedora devel list, so sending it all seems ok to me 18:09:55 <gkadam> nirik, everytime i see easyfix tickets , its already taken by someone or other..also i find it difficult sometimes to start...i mean where to start and what all concepts and understandings are pre requisites for the same 18:10:06 <gkadam> nirik, how should i tackle this ? 18:10:30 <pingou> nirik: agreed, it seems more sensitive to send the fedora stuff to the epel list 18:12:01 <aikidouke> may I chime in nirik? 18:12:46 <pingou> aikidouke: of course 18:13:09 <aikidouke> gkadam - as a relatively new person, it is hard to find a place to start 18:13:15 <pingou> nirik: I think the easiest is to either: ask on the ticket or find the persons who contributed to it and ping them on IRC 18:13:29 <aikidouke> I started by reading as many of our SOPs as I could 18:13:30 <pingou> gkadam: sorry this was actually for you ^ :) 18:13:37 <nirik> sigh. XWayland crashed on me, so I had to reboot. 18:13:39 * nirik reads back 18:13:50 <gkadam> pingou, :) thanks! 18:13:58 <aikidouke> then trying to filter through our playbooks - especially things I was interested in 18:14:09 <nirik> gkadam: so, that ticket's last comment was 8 months ago. Its very safe to say it's open... anything older than a week or so I would say comment on the ticket that you are going ot start looking at it and go 18:14:25 <gkadam> nirik, ack 18:14:35 <aikidouke> go through old tickets and find something that hasnt been touched in a while - as nirik said, leave a comment 18:15:03 <gkadam> aikidouke, thanks! :) 18:15:10 <nirik> and yeah, it can be hard to find a place to start... we could do much better if we did direct mentoring, but if we did that everything would blow up because we don't all have time to do that... so we hope people will learn stuff, ask questions and get more involved. 18:15:16 <aikidouke> hang out and ask questions - if there are fires - understand you might not get an immediate answer 18:15:50 <pingou> don't be afraid to ask 18:15:54 <nirik> I wish I had a better answer... the period from "I want to help" to "I am doing things" is a anoying gap sometimes. ;( 18:16:09 <aikidouke> pick a small thing, that you know well and are or want to be good at, look at it, improve it, etc 18:16:10 <pingou> we're all more than happy to answer question (w/ the time exception of fire burning) 18:16:25 <nirik> yeah. 18:16:40 <aikidouke> +1 nirik -> that period of ummmmm... 18:16:41 <nirik> especially if you can hang out in irc channels and ask about things as others are talking about them... 18:16:41 * cverna asked a lot of question to pingou about pagure 18:16:47 <doteast> yeah, I had a topic for today in relation to this, may it will help 18:17:03 <linuxmodder> ask == learn staying quiet == why are you here :) 18:17:04 <nirik> doteast: oh yeah, saw that and was curious. ;) 18:17:19 <nirik> lets move on to status/info and then discussion... 18:17:48 <nirik> #topic announcements and information 18:17:49 <nirik> #info mass update/reboot cycle mostly done, just a few groups left - kevin/patrick/smooge 18:17:49 <nirik> #info bodhi02.stg reinstalled as bodhi01.stg - kevin 18:17:49 <nirik> #info buildvm-02/03/04/05.stg added in staging - kevin 18:17:49 <nirik> #info bvirthost01 and autosign02 added in build network - kevin 18:17:50 <nirik> #info spammers still trying to get in, mostly being blocked by basset - everyone 18:17:51 <nirik> #info noc01/02 upgraded to nagios 4.0.8 - kevin/patrick 18:18:00 <nirik> any other status/info items? or things people would like to add? 18:18:25 <smooge> congrats on the nagios upgrade 18:18:34 <smooge> that was a major version update? 18:18:37 <pingou> and the reboots :) 18:18:42 <nirik> it's still a mess config wise, but it's working. 18:18:49 <nirik> smooge: yeah, 3.0.x to 4.0.x 18:18:59 <linuxmodder> noc hosts are what again? 18:19:09 <nirik> we really need to scrap it and make it all automated with ansible. ;) 18:19:13 <smooge> they are our nagios servers 18:19:20 <nirik> linuxmodder: noc01 and noc02... /nagios and /nagios-external 18:19:29 <linuxmodder> ah 18:19:52 <doteast> improvements? 18:20:13 <nirik> well, I think it's faster, it launches workers to do checks instead of a single thread. 18:20:22 <nirik> the web interface is a little nicer I guess, but about the same 18:20:40 <doteast> fair enough 18:21:01 <nirik> mostly we updated because the nagios maintainer in epel pushed it out... so, go with the flow. 18:21:30 <nirik> anyhow, anything else status or info wise? shall we move on to discussion items? 18:21:41 <nirik> #topic - Aging tickets - Aikidouke 18:21:49 <aikidouke> thanks 18:21:50 <nirik> aikidouke: take it away. what did you want to do here? 18:21:55 <aikidouke> .ticket 2859 18:21:56 <zodbot> aikidouke: #2859 (Add clamav filter for mailman) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/2859 18:22:19 * nirik ponders. 18:22:21 <aikidouke> its old, references mailman instead of hyperkitty - should we check if its still an issue? 18:22:40 <nirik> I guess I'd be fine just closing it. 18:22:48 <aikidouke> ok 18:22:56 <aikidouke> .ticket 1958 18:22:57 <zodbot> aikidouke: #1958 (phpESP/limesurvey) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/1958 18:23:07 <aikidouke> another old one - ok to close?/ 18:23:22 <smooge> close it 18:23:22 <nirik> yeah, never happened. We got close to a package, but then never got it finished. 18:23:24 <smooge> I thought I did 18:23:35 <smooge> I thought I did close it 18:23:37 <nirik> someone could try again, but not going to be me. ;) 18:23:43 <smooge> nor me 18:23:51 <aikidouke> ok 18:23:59 <aikidouke> and finally .ticket 2996 18:24:02 <smooge> it can go in the cloud with all the other php balls of yarn 18:24:12 <nirik> and perhaps there's some better one around, would need some reinit of requirements/needs 18:24:17 <smooge> .ticket 2996 18:24:18 <zodbot> smooge: #2996 (linux-pam.git hooks request) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/2996 18:24:24 <aikidouke> oh shoot 18:24:39 <aikidouke> anyway - looks like nirik fixed it but didnt get a response to close 18:24:55 <nirik> I think there was still one part of this pending. 18:24:57 * nirik looks 18:25:12 <smooge> well nirik was the last person on it 18:25:21 <aikidouke> it looks like you had asked for some feedback 18:25:24 <smooge> or what aikidouke said :) 18:25:39 <aikidouke> :) 18:25:40 <nirik> yeah, so I guess close and say open a new ticket if you still need anything here. 18:25:47 <aikidouke> cool 18:26:17 <aikidouke> just one fyi if I can slip it in - PS1 - good idea added to the new ticket, will try it out today/submit patch 18:26:25 <aikidouke> that's all I had - TY! 18:26:32 <nirik> yeah, except I don't think it solves the problem. ;) 18:26:36 <nirik> I have a TERM set. 18:26:42 <aikidouke> oh well darn, ok 18:26:42 <nirik> it still spews the tput stuff for me 18:26:56 <nirik> so, I am not sure what the solution is... 18:27:10 <aikidouke> well - that may be further down the rabbit hole than we want to go right now? 18:27:22 <nirik> ah, I bet I know why... 18:27:24 <aikidouke> I'll keep poking it at when I can tho 18:27:38 <nirik> if I ssh to bastion it's fine, if I ssh to an invernal machine via bastion nc it's not. 18:27:45 <nirik> so yeah, the term thing might fix it. 18:28:00 <nirik> let me know when you commit something and I can push it out 18:28:01 * aikidouke realizes I just went off agenda 18:28:05 <nirik> no biggie. 18:28:07 <aikidouke> ok thanks again 18:28:21 <nirik> anything else on this? or move on? 18:28:33 <aikidouke> i am good 18:28:57 <nirik> cool. thanks aikidouke. You ok to close those tickets? or did you want me or smooge to? 18:29:47 <aikidouke> doing it now :) 18:29:50 <nirik> thanks 18:30:02 <nirik> #topic - fi sysadmin potluck - doteast 18:30:14 <nirik> doteast: take it away, what are you meaning here?/ ;) 18:30:25 <doteast> thanks nirik 18:30:29 <doteast> so, this came-up in fi-apprentice monthly status update 18:30:38 <doteast> the idea is that mentoring in the strict sense is very difficult in fi, particularly, when it comes to sysadmin work. 18:30:47 <doteast> instead the idea is to bring along a set of tasks and a followup "habbit" :) on a regular basis to apprentices. 18:31:06 <doteast> this could help in the placement and focus of apprentices in areas where they benefit fi and help them grow 18:31:20 <gkadam> doteast, sounds really great! 18:31:28 <doteast> things can range from simple patches/enhancements to ongoing or foreseen projects/plans, or even checking-out a how a new piece of software that might benefit fi. may fi repo packaging..etc. 18:31:51 <doteast> observing fi for some time now, even during crazy times or crises, new idea appear about improvements and/or investigations 18:32:24 <nirik> these would be easyfixes? or ? 18:32:27 <doteast> for this, two things have been proposed so far: 1) weekly apprent. meeting 2) weekly meeting agenda item, along with apprent. intro, etc 18:32:47 <nirik> ah right, so more a chance to talk about/meet about things... 18:33:18 <doteast> hmm, and for fi-sysadmins to bring to the meeting free food (work) :) 18:33:30 <nirik> another related thing I have seen some groups do (like docs) is 'office hours'... 18:33:49 <nirik> ie, a time period when people block off to try and not be busy with other things so they can answer questions, etc. 18:34:18 <aikidouke> does threebean have a thing like this for sysadmin-dev stuff? 18:34:34 <aikidouke> thought I saw something on his blog - wanted to try to attend one 18:34:46 <aikidouke> I like the idea though 18:34:50 <nirik> Yeah, they were doing a hangouts or something? 18:35:01 <doteast> I guess, yeah...but may encourage steady creation of tasks for apprent. 18:35:23 <aikidouke> yeah - there are some new options there as far as webrtc I think - jitsi has a new web client 18:35:24 <pingou> jit.si 18:35:35 <pingou> it was the weekly web-clinic 18:35:44 <nirik> yeah, I'd prefer irc personally, but could go with other if everyone else wanted. 18:35:55 <pingou> the idea was that everyone could bring an idea/problem to discuss 18:36:17 <pingou> and otherwise just be available there for a little while if someone was to show up 18:36:24 <doteast> pingou, spot on 18:36:24 <aikidouke> maybe instead of new tasks, someone could break down an existing ticket into smaller chunks? 18:36:26 * nirik thinks video usually means someone has problems connecting and there is a lot of waiting to speak and no logs 18:36:39 <aikidouke> ^yeah 18:37:11 * doteast thinks irc is fine 18:37:23 <nirik> I like the idea of office hours/weekly time/infrastructure open house/whatever. 18:37:48 <nirik> should we try something next week? if so when? 18:38:06 <pingou> the web-clinic was on Monday 18:38:14 <nirik> IMHO, later in the day works better for me so I have time to put out the normal fires. 18:38:16 <pingou> iirc around 1pm UTC 18:38:25 <doteast> may be start with 2 ^ then if goes well we can assign a specific time 18:38:50 <nirik> doteast: ah true... sure. 18:39:12 <nirik> should we put it at the end of meeting? I haven't really been able to line up many learn about sessions of late... 18:39:51 <doteast> so, on our weekly meeting, officers bring "free" food, pots gets assigned... 18:39:56 <aikidouke> thats a good idea 18:40:25 <nirik> willing to give it a go. ;) 18:40:29 <doteast> yeah, as simple as: hey I want someone to work on this.... 18:40:30 <aikidouke> maybe an apprentice could pick a ticket - add it to the gobby doc? 18:40:49 * pingou onboard as well 18:41:01 <nirik> sure. 18:42:32 <nirik> ok, should we move on? or anything else on this? 18:43:03 <doteast> or, something is giving us a headeache what can we do about it 18:43:48 <nirik> yep. Happy to brainstorm things to work on, go over old tickets to work on, etc. 18:45:01 <nirik> #topic Open Floor 18:45:08 <nirik> anyone have anything for open floor? 18:45:31 <pingou> linuxmodder: you said you had something no? 18:45:53 <nirik> oh, I guess I could also talk about the great kojidb issues... 18:47:16 <nirik> #topic Kojidb issues 18:47:35 <nirik> so, we have had koji db issues of late... load goes high, io goes high, things slow down... 18:47:46 <nirik> with help from jberkus we have: https://fedoraproject.org/wiki/Infrastructure/KojiDBIssues 18:48:06 <nirik> basically there's some bad db queries that are locking things. we have a script to log and kill them now. 18:48:15 <nirik> but we need to track down whats making those queries and fix them 18:48:37 <nirik> Hopefully we can do that and go back to normal. ;) 18:48:42 <nirik> thats all. 18:48:47 <nirik> #topic Open Floor 2 18:48:51 <puiterwijk> nirik: how about changing the account password, and see what crashes? That way we know which client does it 18:49:11 <nirik> puiterwijk: it's not using passwords. 18:49:32 <nirik> all the stuff hits the hub, the hub in turn makes db calls. If you denied the hub... nothing would work 18:49:42 <puiterwijk> Ah, right. 18:49:50 <puiterwijk> I thought you meant there was something else connected to the database 18:49:56 <nirik> I see in the bad queries 'id' and 'session id' so we might be able to find out which client by that 18:50:36 <nirik> but I don't know the koji code at all, so hard to say 18:50:46 * doteast bids the old days when queries were human written :) 18:51:54 <nirik> I think also I may move the db server this weekend... 18:52:20 <nirik> anyhow... 18:52:29 <nirik> linuxmodder: you had something for open floor? 18:52:32 <nirik> or does anyone else? 18:53:52 * nirik will close out in a minute or two if nothing else. 18:54:19 <dgilmore> all sql queries are in hub code 18:56:30 * nirik nods. 18:56:36 <nirik> ok, thanks for coming everyone! 18:56:38 <nirik> #endmeeting