19:02:38 <smooge> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting
19:02:38 <zodbot> Meeting started Wed Feb  5 19:02:38 2020 UTC.
19:02:38 <zodbot> This meeting is logged and archived in a public location.
19:02:38 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:38 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:02:38 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting'
19:02:38 <smooge> #chair smooge nirik relrod cverna
19:02:38 <smooge> #info meeting is 30 minutes MAX. At the end of 30, its stops
19:02:38 <smooge> #info agenda is at https://board.net/p/fedora-infra-daily
19:02:38 <zodbot> Current chairs: cverna nirik relrod smooge
19:02:38 <smooge> #topic Tickets needing review
19:02:39 <smooge> #info https://pagure.io/fedora-infrastructure/issues
19:02:49 <smooge> good morning
19:03:06 <smooge> https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1
19:05:06 <smooge> .ticket 8615
19:05:06 <zodbot> smooge: Issue #8615: Retire keys.fedoraproject.org 2020-02-07 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8615
19:05:24 <smooge> This is in progress. I have sent an email to the mailing lsits and taken over ticket.
19:05:33 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8615 https://pagure.io/fedora-infrastructure/issue/8615
19:05:55 <fm-admin> pagure.issue.assigned.added -- smooge assigned ticket fedora-infrastructure#8617 to smooge https://pagure.io/fedora-infrastructure/issue/8617
19:06:07 <smooge> .ticket 8617
19:06:08 <zodbot> smooge: Issue #8617: Rack, stack, and install new retrace server - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8617
19:06:20 <nirik> ok, here now
19:06:22 <nirik> .ticket 8617
19:06:23 <zodbot> nirik: Issue #8617: Rack, stack, and install new retrace server - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8617
19:06:24 <nirik> huh, why does zodbot hate me. ;)
19:06:26 <nirik> anyhow, we have the new hardware?
19:06:34 <smooge> The system has been racked. it just needs to start doing the opendcim changes
19:06:47 <nirik> ok
19:06:47 <smooge> nirik, we have it for the retrace box.
19:06:48 <fm-admin> pagure.issue.comment.added -- lucarval commented on ticket fedora-infrastructure#8248: "MTS is now running in prod in dry_run mode" https://pagure.io/fedora-infrastructure/issue/8248#comment-624602
19:06:55 <smooge> the copr boxes arrive next week
19:07:13 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8617 https://pagure.io/fedora-infrastructure/issue/8617
19:07:20 <nirik> that ordering got sorted? and they are going to phx2?
19:07:30 <smooge> yep
19:07:41 <smooge> lots o emails
19:07:48 <nirik> ok, that answers the question I sent you in email then
19:08:01 <nirik> .ticket 8620
19:08:02 <zodbot> nirik: Issue #8620: Access requested - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8620
19:08:05 <smooge> and a long spread sheet I am now trakcking
19:09:17 <nirik> so, I can do this now... we don't actually have a team for that project/app tho
19:09:51 <smooge> i have no knowledge of what is needed for htis so I was going to ask you
19:09:54 <nirik> relrod: you around?
19:10:18 <nirik> looks like there's a authdev team, but not sure thats where we want to add them...
19:11:02 * cverna around
19:11:26 <nirik> It may be authdev, but that might be broader...
19:11:44 <cverna> looking at the people already in the authdev it looks like people working on AAA
19:11:58 <nirik> yeah, and it has the right repos...
19:12:00 <cverna> but yeah we can ask relrod to ack :)
19:12:16 <nirik> just wanted to make sure so we don't add to a group that has access to a bunch of security related things.
19:12:33 <nirik> proposal: ask relrod to ack in ticket and add them if thats the right group
19:13:01 <nirik> .ticket 8621
19:13:02 <zodbot> nirik: Issue #8621: XML parser error when building two flatpak containers - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8621
19:13:26 <nirik> cverna: you were gonna look at this one? might move it to waiting on asignee?
19:13:58 <cverna> Yes
19:14:08 <fm-admin> pagure.issue.edit -- kevin edited the priority fields of ticket fedora-infrastructure#8620 https://pagure.io/fedora-infrastructure/issue/8620
19:14:09 <fm-admin> pagure.issue.comment.added -- kevin commented on ticket fedora-infrastructure#8620: "Access requested" https://pagure.io/fedora-infrastructure/issue/8620#comment-624604
19:14:15 <cverna> I have assigned it to myself but forgot to update the status
19:14:24 <nirik> easy to do
19:14:25 <cverna> would be nice if that could be automated :)
19:14:39 <fm-admin> pagure.issue.edit -- cverna edited the priority fields of ticket fedora-infrastructure#8621 https://pagure.io/fedora-infrastructure/issue/8621
19:14:45 <nirik> ok, thats all the needs review.
19:14:45 <cverna> done
19:14:51 <nirik> any other ones to discuss today?
19:15:06 <smooge> so I have some old ones
19:15:10 <cverna> I pinged, updated quite a few tickets this morning
19:15:24 <cverna> go on smooge
19:15:34 <nirik> cverna: I saw. thanks for that. ;)
19:15:39 <smooge> .ticket 5887
19:15:40 <zodbot> smooge: Issue #5887: Log spew messages from /var/log/merged/messages.log - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/5887
19:16:02 <nirik> so, this was orig a ticket to give apprentices something to work on... but it never really panned out.
19:16:22 <smooge> so I was going to close it as we will need to work out something different when we move to RHEL-8
19:16:47 <nirik> right, epylog sadly will need replacement
19:17:05 <smooge> yeah and the replacements are basically 10 systems to run elasticsearch
19:17:14 <fm-admin> pagure.issue.edit -- smooge edited the close_status and status fields of ticket fedora-infrastructure#5887 https://pagure.io/fedora-infrastructure/issue/5887
19:17:28 <nirik> and they all suck/are not as simple/good
19:17:36 <smooge> .ticket 7115
19:17:37 <zodbot> smooge: Issue #7115: We need application monitoring against openshift routers - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/7115
19:18:25 <nirik> so that needs the other apps added I think?
19:18:28 <smooge> so I was going to unassign myself to this and let someone else start defining things to it
19:18:45 <smooge> I don't want to give the impression I can work on it soon
19:18:50 <nirik> sure, please do.
19:18:53 <cverna> +1
19:19:01 <fm-admin> pagure.issue.assigned.reset -- smooge reset the assignee of ticket fedora-infrastructure#7115 https://pagure.io/fedora-infrastructure/issue/7115
19:19:11 <nirik> in the ideal world we would only be assigned to 1 ticket at a time
19:19:38 <cverna> 2 or 3 would be a good start :)
19:19:51 <nirik> BTW, on log spew, here's the top 5 machines by log lines:
19:19:53 <nirik> 1674624 autosign01.stg.phx2.fedoraproject.org
19:19:53 <nirik> 676575 notifs-backend01.phx2.fedoraproject.org
19:19:53 <nirik> 375268 oci-registry01.phx2.fedoraproject.org
19:19:53 <nirik> 374878 oci-registry02.phx2.fedoraproject.org
19:19:54 <nirik> 278561 pagure01.vpn.fedoraproject.org
19:20:02 <smooge> oof
19:20:10 <nirik> I think autosign01.stg is still set to DEBUG level of spew
19:20:24 <nirik> notifs has always been noisy
19:20:26 <smooge> that is per day correct? or hour
19:20:38 <nirik> and the registry logs every single access to the journal.
19:20:44 <nirik> per day.
19:21:04 <smooge> ok that makes it 24 times less ugly
19:21:17 <nirik> still comically bad
19:21:21 <nirik> anyhow.
19:21:23 <cverna> can we put that in the ticket ? that might give an idea to where to look first ? happy to do it
19:21:41 <nirik> cverna: we closed the ticket... :)
19:21:52 <nirik> perhaps we could be more targeted?
19:21:53 <cverna> ha ha then forget it :P
19:22:03 <nirik> open a new one 'reduce log spew on autosign01.stg' ?
19:22:05 <smooge> cverna if you want to do it then reopen 5887
19:22:11 <smooge> or I can do that
19:22:15 <nirik> or reopen,ywah
19:22:45 <cverna> how bad is it ? is it worth to spend a lot of time on it ?
19:22:59 <cverna> if not I would not bother too much
19:23:08 <nirik> well, I am ok punting it now, since we are going to have to address it when we replace epylog
19:23:32 <cverna> then let's move on :)
19:24:14 <smooge> cverna, it is several gigabytes per day
19:24:27 <nirik> any other tickets to look at in the last 5min?
19:24:31 <smooge> we are currently only able to have about a month of data uncompressed
19:24:46 <smooge> no the rest of my 30 tickets I am going to wait til later
19:25:00 <smooge> #endmeeting