18:00:32 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:00:32 <zodbot> Meeting started Thu Apr 2 18:00:32 2020 UTC. 18:00:32 <zodbot> This meeting is logged and archived in a public location. 18:00:32 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:32 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:32 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:32 <nirik> #chair cverna mboddu nirik smooge 18:00:32 <zodbot> Current chairs: cverna mboddu nirik smooge 18:00:33 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops 18:00:33 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily 18:00:33 <nirik> #topic Tickets needing review 18:00:33 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:00:42 <cverna> ha nice did not know about the sop 18:00:55 <mboddu> .hello mohanboddu 18:00:55 <nirik> mattdm: https://docs.pagure.org/infra-docs/sysadmin-guide/sops/freenode-irc-channel.html 18:00:56 <zodbot> mboddu: mohanboddu 'Mohan Boddu' <mboddu@bhujji.com> 18:01:20 <smooge> hllo 18:01:28 <smooge> just made it bck 18:01:31 <nirik> .ticket 8802 18:01:32 <zodbot> nirik: Issue #8802: Fedora planet is not updating (again) - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8802 18:01:34 <cverna> hell 18:01:36 <nb> mattdm, what is the difference between #fedora-social-hour and #fedora-social? 18:01:52 <nirik> this just got stuck again... I don't know if we want to come up with a longer term solution or not here. 18:02:28 <fm-admin> pagure.issue.edit -- kevin edited the close_status and status fields of ticket fedora-infrastructure#8802 https://pagure.io/fedora-infrastructure/issue/8802 18:02:29 <fm-admin> pagure.issue.comment.added -- kevin commented on ticket fedora-infrastructure#8802: "Fedora planet is not updating (again)" https://pagure.io/fedora-infrastructure/issue/8802#comment-639699 18:02:43 <nirik> I guess we could stick in a 'pkill -f planet' in cron. ;) 18:02:48 <cverna> not sure how much effort it would need 18:02:56 <cverna> do we know what is failing ? 18:03:15 <nirik> it's hanging... some bad/crap web server is just hanging and it never finishes it's run. 18:04:18 <nirik> I guess we can ponder and revisit if it keeps happening more? 18:04:19 <smooge> I think it is multiple ones 18:04:30 <nirik> .ticket 8803 18:04:31 <zodbot> nirik: Issue #8803: Link sonarqube to noggin - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8803 18:04:34 <cverna> ok yeah I guess the cron might not be a bad idea 18:04:37 <nirik> this is done? 18:04:41 <cverna> if we don't want to spend much more time on it :) 18:04:43 <nirik> I did this a while back. 18:05:12 <nirik> or perhaps something else. 18:05:34 <nirik> I guess a webhook? 18:05:56 <cverna> I could check with james tomorrow, and try to clarify what is needed 18:06:01 <nirik> I can do this but we need the info for the webhook. 18:06:02 <cverna> I ll assign this to me 18:06:06 <nirik> or you could cverna, sure. 18:06:15 <fm-admin> pagure.issue.assigned.added -- cverna assigned ticket fedora-infrastructure#8803 to cverna https://pagure.io/fedora-infrastructure/issue/8803 18:06:24 <nirik> .ticket 8804 18:06:25 <zodbot> nirik: Issue #8804: Playing with packages app // CommunityShift access - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8804 18:06:38 <fm-admin> pagure.issue.tag.added -- cverna tagged ticket fedora-infrastructure#8803: low-trouble and medium-gain https://pagure.io/fedora-infrastructure/issue/8803 18:06:39 <fm-admin> pagure.issue.edit -- cverna edited the priority fields of ticket fedora-infrastructure#8803 https://pagure.io/fedora-infrastructure/issue/8803 18:06:40 <nirik> I can do this one, or someone else can, but we should note to them the upcoming downtime. 18:06:48 <nirik> so, move to waiting on assignee 18:07:06 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8804 https://pagure.io/fedora-infrastructure/issue/8804 18:07:15 <nirik> .ticket 8805 18:07:16 <cverna> yeah I believe smooge did on the ticket and I did mentioned it on the mailing list 18:07:16 <zodbot> nirik: Issue #8805: Break backups into smaller volumes - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8805 18:07:31 <cverna> how do we give access to communishift ? a FAS group ? 18:07:32 <smooge> my first unscheduled meeting of the day 18:07:35 <nirik> smooge: where did this request get to you? 18:07:43 <smooge> nirik, chat.google.com to you and me 18:07:48 * nirik sighs 18:08:23 <nirik> so... this can be probibly reduced to less than 10TB 18:08:30 <nirik> I even suggested I could do that. 18:08:40 <nirik> but then everyone said "no no no, not before the move!" 18:08:51 <mattdm> nb I guess we could bridge to #fedora-social, but this is kind of meant to be a event-based thing rather than ongoing 18:09:11 <nirik> so anyhow, assign it to me, waiting on assignee, and I can look at it. 18:09:27 <smooge> so they need the various volumes down to 10 TB because we are now causing an impact. they can be up to 15 but no larger 18:09:28 <smooge> will do so 18:09:50 <nb> mattdm, I would suggest re-using #fedora-social, to cut down on making yet another channel, although if you want another channel, sure, that can be done 18:10:04 <nirik> ok, I can look at this (again) :) 18:10:05 <nb> you'd probably get more people participating if you used -social 18:10:10 <nb> since people are already there 18:10:17 <nirik> .ticket 8806 18:10:18 <zodbot> nirik: Issue #8806: MBS builds fail with "[Errno 12] Cannot allocate memory" - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8806 18:10:35 <fm-admin> pagure.issue.assigned.added -- smooge assigned ticket fedora-infrastructure#8805 to kevin https://pagure.io/fedora-infrastructure/issue/8805 18:10:36 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8805 https://pagure.io/fedora-infrastructure/issue/8805 18:10:36 <nirik> I fixed this (I think) by restarting fedmsg-hub on mbs-backend01. 18:10:48 <nirik> however, I think there's a ticket to fix this so we don't have to do that all the time... 18:10:51 <smooge> ok will ask in ticket 18:11:05 <cverna> still using fedmsg :( 18:11:18 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#8806: mbs https://pagure.io/fedora-infrastructure/issue/8806 18:11:19 <fm-admin> pagure.issue.assigned.added -- smooge assigned ticket fedora-infrastructure#8806 to kevin https://pagure.io/fedora-infrastructure/issue/8806 18:11:20 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#8806 https://pagure.io/fedora-infrastructure/issue/8806 18:11:58 <nirik> Iam looking for it. 18:12:08 <nirik> smooge: we should just close 8806 I think 18:12:21 <nirik> .ticket 8445 18:12:23 <zodbot> nirik: Issue #8445: Provide newer versions of python-kerberos and python-requests-kerberos in epel7-infra-stg - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8445 18:12:31 <nirik> this is the one ^ we need some rpms to make it not leak 18:12:47 <fm-admin> pagure.issue.edit -- smooge edited the close_status and status fields of ticket fedora-infrastructure#8806 https://pagure.io/fedora-infrastructure/issue/8806 18:13:05 <nirik> so, perhaps we should try and get that done soon... 18:13:21 <nirik> I can try or smooge was going to? or does someone else want it? :) 18:13:51 <kalev> MBS just fell over again, I think -- getting error 500 when submitting a build 18:14:00 <cverna> could it be something we can try to do during the packaging hackfest tomorrow 18:14:16 <smooge> I was supposed to do so and it fell into a hole 18:14:46 <smooge> kalev, there is an ongoing outage in proxies at the moment 18:14:56 <nirik> kalev: it's ipsilon. I am working on it. 18:15:07 <nirik> cverna: oh, yeah, +1 18:15:23 <nirik> kalev: might be back... try again? 18:16:19 <nirik> so, thats all the needs review. 18:16:21 <kalev> nirik: smooge: yep, works now, thanks! 18:16:29 <nirik> any blockers, reviews, other tickets to discuss? 18:17:10 <cverna> nothing here that I can think of 18:17:20 <cverna> do we have any releng tickets ? 18:17:43 * nirik looks 18:17:57 <nirik> .releng 9380 18:18:04 <zodbot> nirik: Issue #9380: Clean up side-tags in koji.stg - releng - Pagure.io - https://pagure.io/releng/issue/9380 18:18:14 <nirik> this one should be easy to do... just fedpkg remove-side-tag on them (if you are a admin there) 18:18:28 <nirik> .releng 9364 18:18:31 <zodbot> nirik: Issue #9364: drop fedora-release and fedora-repos from the secure-boot channel - releng - Pagure.io - https://pagure.io/releng/issue/9364 18:18:41 <nirik> this is done, just waiting on more history, but I think we can just close it. 18:18:54 <pingou> nirik: what does one need for 9380? 18:18:56 <nirik> 2 unretirement requests 18:18:59 <pingou> koji admin priv? 18:19:06 <nirik> pingou: yeah, I think so. 18:19:08 <pingou> ok 18:20:08 <nirik> you can list them easily with 'koji list-targets' too 18:21:19 <nirik> so my updates are still running on some s390x builders. ;( probibly stuck. I will clean them up... 18:21:27 <nirik> and we can start reboot later / tomorrow. 18:21:34 <nirik> I don't have anything else off hand. 18:21:48 <kalev> I keep running into transient s390x builder issues all the time 18:22:00 <kalev> should I note what builders are failing and file tickets or is it expected? 18:22:33 <kalev> it's not much of a trouble to resubmit builds, just wondering if it would be helpful to gather failing host names or not 18:22:35 <nirik> depends... whats the issue? 18:22:41 <nirik> unpacking the src.rpm? 18:22:59 <kalev> I don't know, they have been failing so much that I've not bothered to even look lately :) it used to be that, yes 18:23:23 <nirik> well, thats still the issue I am aware of. it's not fixed. 18:23:28 <nirik> https://pagure.io/koji/issue/1974 is the upstream issue 18:23:46 * kalev nods. 18:23:59 <nirik> I have had 0 time to apply that debugging. 18:24:01 <kalev> anyway, I didn't at all want to cause more work here, just wondered if it would be helpful :) 18:24:32 <nirik> if you could look and see if it's the src.rpm or something else we don't know about yet that could be handy 18:24:44 <kalev> ok, will do 18:24:45 <nirik> if it's the src.rpm thing... we know. ;) 18:24:48 <kalev> ok 18:25:37 <kalev> MBS fell over again :( 18:25:37 <kalev> Reason: [Errno 12] Cannot allocate memory 18:25:49 <nirik> ok, if nothing else I have pleanty to do and we can end here? or anything from anyone else? 18:26:23 <smooge> #endmeeting