15:25:57 <nirik> #startmeeting Infrastructure (2019-01-10)
15:25:57 <zodbot> Meeting started Thu Jan 10 15:25:57 2019 UTC.
15:25:57 <zodbot> This meeting is logged and archived in a public location.
15:25:57 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:25:57 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:25:57 <zodbot> The meeting name has been set to 'infrastructure_(2019-01-10)'
15:25:57 <nirik> #meetingname infrastructure
15:25:57 <nirik> #topic aloha
15:25:57 <nirik> #chair nirik pingou puiterwijk relrod smooge tflink threebean cverna mkonecny
15:25:57 <zodbot> The meeting name has been set to 'infrastructure'
15:25:57 <zodbot> Current chairs: cverna mkonecny nirik pingou puiterwijk relrod smooge tflink threebean
15:27:06 * relrod waves
15:27:14 <nirik> morning everyone
15:27:29 <nirik> #topic New folks introductions
15:27:29 <nirik> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:27:29 <nirik> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:27:33 <nirik> any new folks around today?
15:28:37 * cverna waves
15:29:15 * mkonecny waves
15:30:28 <nirik> on to announcements...
15:30:44 <nirik> #topic announcements and information
15:30:44 <nirik> #info Anitya (release-monitoring.org) 0.14.0 released - deployment on production planned to next week
15:30:44 <nirik> #info tflink is on extended PTO
15:30:44 <nirik> #info devconf is happening from 2019-01-25 -> 2019-01-27
15:30:45 <nirik> #info major outage: s390 2019-01-11 -> 2019-01-14
15:30:46 <nirik> #info Bodhi stakeholders' meeting next Tuesday https://apps.fedoraproject.org/calendar/meeting/9429/
15:30:49 <nirik> #info Proposed backwards incompatible changes in upcoming Bodhi 4.0.0: https://github.com/fedora-infra/bodhi/projects/6
15:30:53 <nirik> any other announcements or items to note?
15:30:56 <tflink> actually, I'm back now
15:31:04 <nirik> oh hey! welcome back tflink!
15:31:32 <cverna> #info whatcanidoforfedora now runs on openshift
15:31:34 <relrod> #info proxies are done being upgraded to f29 \o/
15:32:45 <smooge> *cough*
15:32:48 <smooge> sorry everyone
15:32:52 <smooge> thank you nirik
15:33:03 <tflink> nirik: thanks, still trying to get caught up with the stuff I missed
15:33:10 <nirik> #info builders are all upgraded to f29 except for all the armv7 ones (f27) and ppc64 ones (f28)
15:33:28 <nirik> hey smooge
15:33:28 <smooge> hi tflink
15:33:45 <smooge> you can keep running meeting. I just wanted to say hi and thanks for doing so
15:34:06 <smooge> I realized my *cough* could mean I could take over...
15:34:25 <cverna> fedora-packages still runs on f27, I have fixed Turbogears2 in f29 and it should land in stable tomorrow, so we can move these boxes to f29
15:34:28 <nirik> oh, so I was supposed to be on PTO this week (for moving) but thats not happening... so I instead now will be off the week after devconf at the end of the month
15:34:40 <nirik> cverna: great
15:35:15 <smooge> I have a question on ppc64be .. when F28 goes away do we need to keep an EL7 one?
15:35:42 <nirik> yep, we will. ;(
15:35:48 <nirik> for epel6/7
15:36:21 <nirik> but it can be just 2 or something...
15:36:54 <nirik> #topic Oncall
15:36:54 <nirik> #info relrod is on call from 2019-01-10 -> 2019-01-17
15:36:54 <nirik> #info ??? is on call from 2019-01-17 -> 2019-01-24
15:36:54 <nirik> #info ??? is on call from 2019-01-24 -> 2019-01-31
15:36:54 <nirik> #info Summary of last week: (from smooge )
15:37:19 <creaked> .hello2
15:37:20 <zodbot> creaked: creaked 'Will Chellman' <creaked@gmail.com>
15:37:25 <smooge> Last week was pretty quiet. We have a problem with pkgs01.sg
15:37:44 <smooge> but otherwise most everything is the usual fedmsg memory leak
15:38:22 <tflink> a bunch of the taskotron machines are still F27, we're working on it but are currently stuck on upgrading to a new buildbot version
15:38:43 * nirik plays the pipes for the transfer of the oncall from smooge to relrod
15:39:25 <relrod> Next week classes are starting for me. So there might be a few periods of me not being around throughout the day.
15:39:48 <nirik> theres a few random other machines still on 27, I will try and upgrade/redeploy those soon
15:40:07 <nirik> relrod: ok. I should be around if needed.
15:40:36 <nirik> #topic Monitoring discussion
15:40:36 <nirik> #info https://nagios.fedoraproject.org/nagios
15:40:36 <nirik> #info Go over existing out items and fix
15:41:01 <nirik> we have some proxy03 alerts, but that should go away when it gets reinstalled.
15:41:19 <nirik> 2 machines have drives out of raid: ppc8-02 and virthost06
15:42:01 <nirik> the check for fedmsg plugin in zodbot is alerting again... likely a change on freenode we need to adjust the plugin for.
15:42:20 <mizdebsk> pkgs01.stg alerts are annoying, perhaps we can add more ram or swap to it?
15:42:45 <nirik> mizdebsk: I think puiterwijk needs to sort out some repospanner issues there...
15:42:53 <nirik> but yeah, we could try I guess.
15:42:55 <puiterwijk> nirik: not really, the box just needs more mem
15:43:08 <nirik> ok, how much?
15:43:32 <puiterwijk> Dunno? 128GB?
15:43:36 <nirik> it's got 8gb now...
15:44:08 <mizdebsk> or reduce number of httpd processes or something?
15:44:21 <mizdebsk> httpd seems to be responsible for most of memory consumption
15:44:25 <puiterwijk> Sure. That could work too
15:44:38 <nirik> we have 64gb free on that virthost.
15:44:44 <nirik> so we could bump it to 32?
15:44:57 <nirik> or 48?
15:45:14 <puiterwijk> I'd say 32, but maybe first just lower httpd?
15:45:39 <nirik> ok, someone want to take on this? I can if no one else has time...
15:45:51 <mizdebsk> i can do it
15:46:14 <nirik> thanks mizdebsk!
15:46:36 <nirik> #topic Tickets discussion
15:46:36 <nirik> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:47:22 <nirik> we have one item in meeting tickets, but mizdebsk's comment is fine with me there.
15:47:33 <smooge> so the issue on the pkgs01.stg was that several times I found that the swap was full but free ram was empty
15:48:11 <mizdebsk> smooge, yes, i think it would be good to check for sum of free swap and memory, not just swap
15:48:12 <nirik> yeah, I think that happens when there's a big httpd request or two, it swaps and then the request is over and memory goes down
15:49:19 <nirik> anyhow on tickets... if everyone could look at tickets they are assigned to or said they would do something on and do that, I would be happy. We have quite a lot of tickets building up...
15:49:24 <smooge> ah ok when I had seen this on our other servers it was usually some sort of memory leak where the ram was seen as not needed so the kernel never swapped it back out
15:49:41 <smooge> that is all on it and will not interrupt :)
15:50:05 <smooge> next week is DO ALL THE TICKETS!
15:50:44 <nirik> CLOSED->KTHXBYE :)
15:50:51 <mizdebsk> smooge, often it is a leak, but also services like rsyslogd like to preallocate like 2g ram in advance, which stays in swap
15:51:06 <nirik> #topic Apprentice Open office minutes
15:51:06 <nirik> #info A time where apprentices may ask for help or look at problems.
15:51:24 <nirik> any apprentices with questions or ideas or anything?
15:52:29 <chris787> i'd like to join the apprentice group i'm following meetings for quite a time now and like to implicate a little further
15:52:56 <nirik> chris787: excellent. welcome. See us all after the meeting in #fedora-admin and we can get you setup
15:53:05 <chris787> coool !
15:53:27 <chris787> thx
15:53:42 <nirik> #topic Open Floor
15:53:51 <nirik> anyone have anything for open floor?
15:54:31 <cverna> not sure if we want to discuss about the council report ?
15:54:34 <mizdebsk> there is an interesting infra-related thread on council-discuss list in case someone hasn't seen it yet
15:54:42 <cverna> or if there is anything to say about it :)
15:54:54 <nirik> ah yeah... we could note it for sure.
15:55:06 <cverna> https://lists.fedoraproject.org/archives/list/council-discuss@lists.fedoraproject.org/thread/GOUIBXVVVSE33ZMGV6JNFZIR6K5WZMG7/
15:55:10 <cverna> ^^
15:56:14 <cverna> we are supposed to work with FESCO to identify all the services we are running
15:56:17 <nirik> I guess we will see how it goes over the next few weeks... I'm fine with part of it, but not sure about other parts
15:56:23 <cverna> if I understood correctly
15:56:40 <nirik> yeah, which seems a bit odd, but ok.
15:56:59 <smooge> I think we need to do a bit more than that
15:57:03 <cverna> this is will be a long list :)
15:57:12 <smooge> because it will turn into a long loop like the other times we do that
15:57:33 <smooge> we need to outline all the services we run and should also outline what those services need to be 'close' to
15:57:35 <nirik> the thought of moving things that are working fine and use very little of our time means it... will use a lot more of our time
15:57:38 <smooge> or are related to
15:58:17 <smooge> it would also be good to get a good estimate of how much time we actually spend on each of those
15:58:42 <cverna> agreed with smooge
15:58:44 <nirik> indeed. But thats hard without all of us doing a good job recording it.
15:59:03 <smooge> so I think that should be our next goal. Pick a tool and make us all use it for a month
15:59:20 <nirik> well, it looks like that tool may be taiga...
15:59:44 <smooge> well then problem A solved
15:59:51 <smooge> onto problem B.. using it
16:00:01 <nirik> yep. :)
16:00:10 <smooge> going onto problem C. really using it
16:00:26 <smooge> and onto problem D.. I don't care how much of a time sink it is to use it... USE IT
16:00:44 <nirik> we skip one step as we will not be running it... it will be a hosted version by the taiga folks.
16:00:51 <mkonecny> I'm using taiga as my personal tool, but more like a TODO list than anything else
16:00:57 <smooge> me too
16:01:15 * nirik hasn't used it much, but can learn
16:01:58 <cverna> do we roughly know when it will be available ?
16:01:58 <mkonecny> But it's still duplication with github issues, but at least it's all in one place
16:02:23 <nirik> cverna: waiting to hear what amazon instances they need... soon I hope
16:02:37 <cverna> ok :)
16:02:39 <nirik> mkonecny: and pagure issues, and bugzilla and... ;)
16:03:02 <mizdebsk> yeah, https://xkcd.com/927
16:03:05 <nirik> but it should replace our cpe-planning thing
16:03:18 <nirik> mizdebsk++
16:03:41 <nirik> anyhow, will close out in a minute if nothing else...
16:04:01 <mkonecny> nirik: You are right, it will be nice to have this automatized
16:04:23 <mkonecny> At least to list issues from other sources when creating new issue in taiga
16:04:56 <nirik> it's not an easy problem. ;( but hopefully taiga will help us at least at a high level...
16:05:01 <mkonecny> The best situation is to have taiga only tracking what you will tell it to track and don't need to do anything with it
16:05:17 <nirik> yeah.
16:05:34 <mkonecny> But I understand that is not possible at 100%
16:06:10 <nirik> yeah, at least initially... but hopefully we can make it better over time
16:06:39 <nirik> anyhow, thanks for coming everyone!
16:06:42 <nirik> #endmeeting