18:00:00 <nirik> #startmeeting Infrastructure (2016-04-14)
18:00:00 <zodbot> Meeting started Thu Apr 14 18:00:00 2016 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:00 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:00 <zodbot> The meeting name has been set to 'infrastructure_(2016-04-14)'
18:00:01 <nirik> #meetingname infrastructure
18:00:01 <zodbot> The meeting name has been set to 'infrastructure'
18:00:01 <nirik> #topic aloha
18:00:01 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore threebean pingou puiterwijk pbrobinson
18:00:01 <zodbot> Current chairs: abadger1999 dgilmore lmacken nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:01 <nirik> #topic New folks introductions / Apprentice feedback
18:00:14 * pingou here
18:00:47 <nirik> Any new folks want to give a short (one line) introduction? Or any apprentices with questions or comments?
18:01:01 * puiterwijk here but traveling to home
18:01:16 * tflink here
18:01:16 <cverna> yeah
18:01:19 * doteast present
18:01:21 * aikidouke here
18:01:24 <cverna> hi  everyone
18:01:36 <cverna> I have started playing with python begining of the year and did a few easyfix on pagure. I would like now to get a bit more involved within the infra team mostly development side at the begining
18:01:37 <linuxmodder> .fas corey84
18:01:37 <zodbot> linuxmodder: corey84 'Corey Sheldon' <sheldon.corey@gmail.com>
18:02:08 <nirik> cverna: welcome!
18:02:24 <nirik> lots of easyfixes to choose from. :)
18:02:24 <linuxmodder> cverna,  packaging wise or  testing?
18:02:55 <cverna> bit of both I guess :)
18:03:00 * pingou waves to cverna
18:03:35 <cverna> I haven't had much experience in packaging
18:04:15 <linuxmodder> luckily  for you we have  great packagers that  mentor :)
18:04:35 <nirik> yep. Always fun. ;)
18:04:43 <nirik> any other new folks? or apprentices with questions?
18:04:46 <gkadam> nirik, could you please help with https://fedoraproject.org/easyfix  #4485 run POC-Change script for EPEL ?
18:04:58 <linuxmodder> !  something for open floor
18:05:08 * sayan is here
18:05:15 <nirik> .ticket 4485
18:05:17 <zodbot> nirik: #4485 (run POC-Change script for EPEL) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/4485
18:06:00 <nirik> ok, so pingou might be the best one to ask about that... and/or on ticket...
18:06:36 <nirik> https://github.com/pypingou/fedora-owner-change/blob/master/fedora-owner-change.py would be the thing to change
18:06:44 <nirik> right now it only reports fedora changes
18:07:22 <linuxmodder> not schooled enough in epel to know what to change but cverna  I can help if oyu need help
18:07:35 <pingou> I'm thinking we could had a CLI flag to do EPEL as well
18:07:48 <pingou> do epel or fedora or both actually
18:07:51 <linuxmodder> pingou,  what as a var or something
18:08:22 <gkadam> nirik, may ask a bit of more basic qestion ?
18:08:31 <gkadam> *question?
18:08:35 <pingou> linuxmodder: as fedora-owner-change.py --target=epel or so
18:08:41 <nirik> I'd be fine with just both and sending to eppel-devel too, or more specific and sending to each, either way
18:08:53 <cverna> linuxmodder: thx
18:08:54 <nirik> gkadam: absolutely. ;)
18:08:59 <nirik> ask away
18:09:02 <pingou> nirik: then we can drop the flag I guess :)
18:09:39 <nirik> pingou: I think a number of epel maintainers are not on epel-devel but are on fedora devel list, so sending it all seems ok to me
18:09:55 <gkadam> nirik, everytime i see easyfix tickets , its already taken by someone or other..also i find it difficult sometimes to start...i mean where to start and what all concepts and understandings are pre requisites for the same
18:10:06 <gkadam> nirik, how should i tackle this ?
18:10:30 <pingou> nirik: agreed, it seems more sensitive to send the fedora stuff to the epel list
18:12:01 <aikidouke> may I chime in nirik?
18:12:46 <pingou> aikidouke: of course
18:13:09 <aikidouke> gkadam - as a relatively new person, it is hard to find a place to start
18:13:15 <pingou> nirik: I think the easiest is to either: ask on the ticket or find the persons who contributed to it and ping them on IRC
18:13:29 <aikidouke> I started by reading as many of our SOPs as I could
18:13:30 <pingou> gkadam: sorry this was actually for you ^ :)
18:13:37 <nirik> sigh. XWayland crashed on me, so I had to reboot.
18:13:39 * nirik reads back
18:13:50 <gkadam> pingou, :) thanks!
18:13:58 <aikidouke> then trying to filter through our playbooks - especially things I was interested in
18:14:09 <nirik> gkadam: so, that ticket's last comment was 8 months ago. Its very safe to say it's open... anything older than a week or so I would say comment on the ticket that you are going ot start looking at it and go
18:14:25 <gkadam> nirik, ack
18:14:35 <aikidouke> go through old tickets and find something that hasnt been touched in a while - as nirik said, leave a comment
18:15:03 <gkadam> aikidouke, thanks! :)
18:15:10 <nirik> and yeah, it can be hard to find a place to start... we could do much better if we did direct mentoring, but if we did that everything would blow up because we don't all have time to do that... so we hope people will learn stuff, ask questions and get more involved.
18:15:16 <aikidouke> hang out and ask questions - if there are fires - understand you might not get an immediate answer
18:15:50 <pingou> don't be afraid to ask
18:15:54 <nirik> I wish I had a better answer... the period from "I want to help" to "I am doing things" is a anoying gap sometimes. ;(
18:16:09 <aikidouke> pick a small thing, that you know well and are or want to be good at, look at it, improve it, etc
18:16:10 <pingou> we're all more than happy to answer question (w/ the time exception of fire burning)
18:16:25 <nirik> yeah.
18:16:40 <aikidouke> +1 nirik -> that period of ummmmm...
18:16:41 <nirik> especially if you can hang out in irc channels and ask about things as others are talking about them...
18:16:41 * cverna asked a lot of question to pingou about pagure
18:16:47 <doteast> yeah, I had a topic for today in relation to this, may it will help
18:17:03 <linuxmodder> ask == learn   staying quiet ==  why are you here :)
18:17:04 <nirik> doteast: oh yeah, saw that and was curious. ;)
18:17:19 <nirik> lets move on to status/info and then discussion...
18:17:48 <nirik> #topic announcements and information
18:17:49 <nirik> #info mass update/reboot cycle mostly done, just a few groups left - kevin/patrick/smooge
18:17:49 <nirik> #info bodhi02.stg reinstalled as bodhi01.stg - kevin
18:17:49 <nirik> #info buildvm-02/03/04/05.stg added in staging - kevin
18:17:49 <nirik> #info bvirthost01 and autosign02 added in build network - kevin
18:17:50 <nirik> #info spammers still trying to get in, mostly being blocked by basset - everyone
18:17:51 <nirik> #info noc01/02 upgraded to nagios 4.0.8 - kevin/patrick
18:18:00 <nirik> any other status/info items? or things people would like to add?
18:18:25 <smooge> congrats on the nagios upgrade
18:18:34 <smooge> that was a major version update?
18:18:37 <pingou> and the reboots :)
18:18:42 <nirik> it's still a mess config wise, but it's working.
18:18:49 <nirik> smooge: yeah, 3.0.x to 4.0.x
18:18:59 <linuxmodder> noc  hosts are  what again?
18:19:09 <nirik> we really need to scrap it and make it all automated with ansible. ;)
18:19:13 <smooge> they are our nagios servers
18:19:20 <nirik> linuxmodder: noc01 and noc02... /nagios and /nagios-external
18:19:29 <linuxmodder> ah
18:19:52 <doteast> improvements?
18:20:13 <nirik> well, I think it's faster, it launches workers to do checks instead of a single thread.
18:20:22 <nirik> the web interface is a little nicer I guess, but about the same
18:20:40 <doteast> fair enough
18:21:01 <nirik> mostly we updated because the nagios maintainer in epel pushed it out... so, go with the flow.
18:21:30 <nirik> anyhow, anything else status or info wise? shall we move on to discussion items?
18:21:41 <nirik> #topic - Aging tickets - Aikidouke
18:21:49 <aikidouke> thanks
18:21:50 <nirik> aikidouke: take it away. what did you want to do here?
18:21:55 <aikidouke> .ticket 2859
18:21:56 <zodbot> aikidouke: #2859 (Add clamav filter for mailman) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/2859
18:22:19 * nirik ponders.
18:22:21 <aikidouke> its old, references mailman instead of hyperkitty - should we check if its still an issue?
18:22:40 <nirik> I guess I'd be fine just closing it.
18:22:48 <aikidouke> ok
18:22:56 <aikidouke> .ticket 1958
18:22:57 <zodbot> aikidouke: #1958 (phpESP/limesurvey) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/1958
18:23:07 <aikidouke> another old one - ok to close?/
18:23:22 <smooge> close it
18:23:22 <nirik> yeah, never happened. We got close to a package, but then never got it finished.
18:23:24 <smooge> I thought I did
18:23:35 <smooge> I thought I did close it
18:23:37 <nirik> someone could try again, but not going to be me. ;)
18:23:43 <smooge> nor me
18:23:51 <aikidouke> ok
18:23:59 <aikidouke> and finally .ticket 2996
18:24:02 <smooge> it can go in the cloud with all the other php balls of yarn
18:24:12 <nirik> and perhaps there's some better one around, would need some reinit of requirements/needs
18:24:17 <smooge> .ticket 2996
18:24:18 <zodbot> smooge: #2996 (linux-pam.git hooks request) – Fedora Infrastructure - https://fedorahosted.org/fedora-infrastructure/ticket/2996
18:24:24 <aikidouke> oh shoot
18:24:39 <aikidouke> anyway - looks like nirik fixed it but didnt get a response to close
18:24:55 <nirik> I think there was still one part of this pending.
18:24:57 * nirik looks
18:25:12 <smooge> well nirik was the last person on it
18:25:21 <aikidouke> it looks like you had asked for some feedback
18:25:24 <smooge> or what aikidouke said :)
18:25:39 <aikidouke> :)
18:25:40 <nirik> yeah, so I guess close and say open a new ticket if you still need anything here.
18:25:47 <aikidouke> cool
18:26:17 <aikidouke> just one fyi if I can slip it in - PS1 - good idea added to the new ticket, will try it out today/submit patch
18:26:25 <aikidouke> that's all I had - TY!
18:26:32 <nirik> yeah, except I don't think it solves the problem. ;)
18:26:36 <nirik> I have a TERM set.
18:26:42 <aikidouke> oh well darn, ok
18:26:42 <nirik> it still spews the tput stuff for me
18:26:56 <nirik> so, I am not sure what the solution is...
18:27:10 <aikidouke> well - that may be further down the rabbit hole than we want to go right now?
18:27:22 <nirik> ah, I bet I know why...
18:27:24 <aikidouke> I'll keep poking it at when I can tho
18:27:38 <nirik> if I ssh to bastion it's fine, if I ssh to an invernal machine via bastion nc it's not.
18:27:45 <nirik> so yeah, the term thing might fix it.
18:28:00 <nirik> let me know when you commit something and I can push it out
18:28:01 * aikidouke realizes I just went off agenda
18:28:05 <nirik> no biggie.
18:28:07 <aikidouke> ok thanks again
18:28:21 <nirik> anything else on this? or move on?
18:28:33 <aikidouke> i am good
18:28:57 <nirik> cool. thanks aikidouke. You ok to close those tickets? or did you want me or smooge to?
18:29:47 <aikidouke> doing it now :)
18:29:50 <nirik> thanks
18:30:02 <nirik> #topic - fi sysadmin potluck - doteast
18:30:14 <nirik> doteast: take it away, what are you meaning here?/ ;)
18:30:25 <doteast> thanks nirik
18:30:29 <doteast> so, this came-up in fi-apprentice monthly status update
18:30:38 <doteast> the idea is that mentoring in the strict sense is very difficult in fi, particularly, when it comes to sysadmin work.
18:30:47 <doteast> instead the idea is to bring along a set of tasks and a followup "habbit" :) on a regular basis to apprentices.
18:31:06 <doteast> this could help in the placement and focus of apprentices in areas where they benefit fi and help them grow
18:31:20 <gkadam> doteast, sounds really great!
18:31:28 <doteast> things can range from simple patches/enhancements to ongoing or foreseen projects/plans, or even checking-out a how a new piece of software that might benefit fi. may fi repo packaging..etc.
18:31:51 <doteast> observing fi for some time now, even during crazy times or crises, new idea appear about improvements and/or investigations
18:32:24 <nirik> these would be easyfixes? or ?
18:32:27 <doteast> for this, two things have been proposed so far: 1) weekly apprent. meeting 2) weekly meeting agenda item, along with apprent. intro, etc
18:32:47 <nirik> ah right, so more a chance to talk about/meet about things...
18:33:18 <doteast> hmm, and for fi-sysadmins to bring to the meeting free food (work) :)
18:33:30 <nirik> another related thing I have seen some groups do (like docs) is 'office hours'...
18:33:49 <nirik> ie, a time period when people block off to try and not be busy with other things so they can answer questions, etc.
18:34:18 <aikidouke> does threebean have a thing like this for sysadmin-dev stuff?
18:34:34 <aikidouke> thought I saw something on his blog - wanted to try to attend one
18:34:46 <aikidouke> I like the idea though
18:34:50 <nirik> Yeah, they were doing a hangouts or something?
18:35:01 <doteast> I guess, yeah...but may encourage steady creation of tasks for apprent.
18:35:23 <aikidouke> yeah - there are some new options there as far as webrtc I think - jitsi has a new web client
18:35:24 <pingou> jit.si
18:35:35 <pingou> it was the weekly web-clinic
18:35:44 <nirik> yeah, I'd prefer irc personally, but could go with other if everyone else wanted.
18:35:55 <pingou> the idea was that everyone could bring an idea/problem to discuss
18:36:17 <pingou> and otherwise just be available there for a little while if someone was to show up
18:36:24 <doteast> pingou, spot on
18:36:24 <aikidouke> maybe instead of new tasks, someone could break down an existing ticket into smaller chunks?
18:36:26 * nirik thinks video usually means someone has problems connecting and there is a lot of waiting to speak and no logs
18:36:39 <aikidouke> ^yeah
18:37:11 * doteast thinks irc is fine
18:37:23 <nirik> I like the idea of office hours/weekly time/infrastructure open house/whatever.
18:37:48 <nirik> should we try something next week? if so when?
18:38:06 <pingou> the web-clinic was on Monday
18:38:14 <nirik> IMHO, later in the day works better for me so I have time to put out the normal fires.
18:38:16 <pingou> iirc around 1pm UTC
18:38:25 <doteast> may be start with 2 ^ then if goes well we can assign a specific time
18:38:50 <nirik> doteast: ah true... sure.
18:39:12 <nirik> should we put it at the end of meeting? I haven't really been able to line up many learn about sessions of late...
18:39:51 <doteast> so, on our weekly meeting, officers bring "free" food, pots gets assigned...
18:39:56 <aikidouke> thats a good idea
18:40:25 <nirik> willing to give it a go. ;)
18:40:29 <doteast> yeah, as simple as: hey I want someone to work on this....
18:40:30 <aikidouke> maybe an apprentice could pick a ticket - add it to the gobby doc?
18:40:49 * pingou onboard as well
18:41:01 <nirik> sure.
18:42:32 <nirik> ok, should we move on? or anything else on this?
18:43:03 <doteast> or, something is giving us a headeache what can we do about it
18:43:48 <nirik> yep. Happy to brainstorm things to work on, go over old tickets to work on, etc.
18:45:01 <nirik> #topic Open Floor
18:45:08 <nirik> anyone have anything for open floor?
18:45:31 <pingou> linuxmodder: you said you had something no?
18:45:53 <nirik> oh, I guess I could also talk about the great kojidb issues...
18:47:16 <nirik> #topic Kojidb issues
18:47:35 <nirik> so, we have had koji db issues of late... load goes high, io goes high, things slow down...
18:47:46 <nirik> with help from jberkus we have: https://fedoraproject.org/wiki/Infrastructure/KojiDBIssues
18:48:06 <nirik> basically there's some bad db queries that are locking things. we have a script to log and kill them now.
18:48:15 <nirik> but we need to track down whats making those queries and fix them
18:48:37 <nirik> Hopefully we can do that and go back to normal. ;)
18:48:42 <nirik> thats all.
18:48:47 <nirik> #topic Open Floor 2
18:48:51 <puiterwijk> nirik: how about changing the account password, and see what crashes? That way we know which client does it
18:49:11 <nirik> puiterwijk: it's not using passwords.
18:49:32 <nirik> all the stuff hits the hub, the hub in turn makes db calls. If you denied the hub... nothing would work
18:49:42 <puiterwijk> Ah, right.
18:49:50 <puiterwijk> I thought you meant there was something else connected to the database
18:49:56 <nirik> I see in the bad queries 'id' and 'session id' so we might be able to find out which client by that
18:50:36 <nirik> but I don't know the koji code at all, so hard to say
18:50:46 * doteast bids the old days when queries were human written :)
18:51:54 <nirik> I think also I may move the db server this weekend...
18:52:20 <nirik> anyhow...
18:52:29 <nirik> linuxmodder: you had something for open floor?
18:52:32 <nirik> or does anyone else?
18:53:52 * nirik will close out in a minute or two if nothing else.
18:54:19 <dgilmore> all sql queries are in hub code
18:56:30 * nirik nods.
18:56:36 <nirik> ok, thanks for coming everyone!
18:56:38 <nirik> #endmeeting