18:00:30 <nirik> #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:00:30 <zodbot> Meeting started Mon Aug 3 18:00:30 2020 UTC. 18:00:30 <zodbot> This meeting is logged and archived in a public location. 18:00:30 <zodbot> The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:30 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:30 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:30 <nirik> #chair mboddu nirik smooge pingou 18:00:30 <zodbot> Current chairs: mboddu nirik pingou smooge 18:00:30 <nirik> #meetingname fedora_infrastructure_ops_daily_standup_meeting 18:00:30 <zodbot> The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:00:30 <nirik> #info meeting is 30 minutes MAX. At the end of 30, its stops 18:00:31 <nirik> #info agenda is at https://board.net/p/fedora-infra-daily 18:00:32 <nirik> #topic Tickets needing review 18:00:33 <nirik> #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:00:49 <smooge> HELLO VIET_ADMIN 18:00:58 <nirik> good morning 18:02:01 <smooge> ok tickets 18:02:08 <nirik> you want to mod tickets? 18:02:14 <smooge> ok will do so 18:02:20 <nirik> .ticket 9189 18:02:21 <zodbot> nirik: Issue #9189: Fedora docs not being built anymore - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9189 18:02:28 <nirik> so, lets fix this now. ;) 18:02:51 <nirik> I am going to run: 18:03:12 <nirik> ansible -a 'pkill -9 rsync' proxies 18:03:26 <nirik> unless there's a more elegant way? 18:03:40 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#9189: groomed, high-gain, and 2 others https://pagure.io/fedora-infrastructure/issue/9189 18:03:41 <fm-admin> pagure.issue.assigned.added -- smooge assigned ticket fedora-infrastructure#9189 to kevin https://pagure.io/fedora-infrastructure/issue/9189 18:03:42 <fm-admin> pagure.issue.edit -- smooge edited the priority fields of ticket fedora-infrastructure#9189 https://pagure.io/fedora-infrastructure/issue/9189 18:03:43 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#9189: "Fedora docs not being built anymore" https://pagure.io/fedora-infrastructure/issue/9189#comment-668882 18:04:01 <smooge> nirik, sounds good. I don't know of a better way 18:04:31 * mboddu is here 18:04:58 <nirik> done and I am running a sync. I think we can close that ticket now? or do we want to wait until the sync finishes? 18:06:03 <smooge> lets wait until the sync finishes 18:06:17 <smooge> I don't like closing 'fixed' and finding out it wasn't 18:06:19 <nirik> .ticket 9091 18:06:20 <zodbot> nirik: Issue #9091: Very slow s390x builder - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9091 18:06:26 <nirik> wait, thats wrong 18:06:34 <nirik> .ticket 9191 18:06:35 <zodbot> nirik: Issue #9191: ELN bodhi updates closing Fedora rawhide bugzillas - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9191 18:06:45 <smooge> no its right.. just not the ticket you wanted :P 18:07:13 <nirik> IMHO, this should be refiled upstream... it's bodhi behavior that was just added, and it needs adjusting. 18:07:28 <nirik> which of course brings up... who is bodhi upstream anymore. 18:07:35 <smooge> we are 18:08:19 <nirik> anyhow, we could ask them to refile upstream or do so for them and close? or ? 18:09:28 <smooge> ok filing upstream 18:10:21 <nirik> I don't think that change was properly thought out... but perhaps it's just me 18:11:28 <mboddu> I asked the same question myself couple of days :D 18:11:35 <mboddu> days back* 18:12:10 <nirik> .ticket 9193 18:12:11 <zodbot> nirik: Issue #9193: Review-stats pod stuck - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/9193 18:12:18 <fm-admin> pagure.issue.edit -- smooge edited the close_status and status fields of ticket fedora-infrastructure#9191 https://pagure.io/fedora-infrastructure/issue/9191 18:12:19 <fm-admin> pagure.issue.tag.added -- smooge tagged ticket fedora-infrastructure#9191: bodhi https://pagure.io/fedora-infrastructure/issue/9191 18:12:20 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#9191: "ELN bodhi updates closing Fedora rawhide bugzillas" https://pagure.io/fedora-infrastructure/issue/9191#comment-668885 18:12:25 <nirik> I can likewise do this one right now, it's just a oc delete away 18:12:54 <smooge> can you walk me through this.. my openshift knowledge is crap 18:13:05 <nirik> pod "review-stats-make-html-pages-1596013200-bvgct" deleted 18:13:09 <nirik> sure. 18:13:19 <nirik> login to os-master01.iad2 (or any os-master) 18:13:24 <nirik> oc project review-stats 18:13:27 <fm-admin> pagure.issue.assigned.added -- smooge assigned ticket fedora-infrastructure#9193 to smooge https://pagure.io/fedora-infrastructure/issue/9193 18:13:35 <nirik> (note that you can use tab for the project, it will complete it) 18:13:53 <nirik> oc delete pod/review-stats-make-html-pages-1596013200-bvgct 18:14:01 <smooge> Already on project "review-stats" on server "https://os-masters.iad2.fedoraproject.org:443". 18:14:19 <smooge> Error from server (NotFound): pods "review-stats-make-html-pages-1596013200-bvgct" not found 18:14:21 <nirik> yeah, it will be on the project the last person to run as root on there was/is 18:14:29 <smooge> oh you did this already 18:14:30 <nirik> right, because I already deleted it 18:14:43 <nirik> I did look at logs and the last thing in it was: 18:14:46 <nirik> 2020-07-29 09:00:22,025 review_stats.py INFO Quering Bugzilla for the blockers list... 18:14:54 <nirik> so I guess it got stuck talking to bz 18:15:09 <fm-admin> pagure.issue.edit -- smooge edited the close_status and status fields of ticket fedora-infrastructure#9193 https://pagure.io/fedora-infrastructure/issue/9193 18:15:10 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#9193: "Review-stats pod stuck" https://pagure.io/fedora-infrastructure/issue/9193#comment-668888 18:15:20 <mboddu> bz became huge pain lately, ftbfs are also failing randomly with 502's 18:15:20 <nirik> thats all in needs-review in infra 18:15:33 <nirik> mboddu: huh, I thought it was fixed now.... 18:15:40 <fm-admin> pagure.issue.edit -- smooge edited the close_status and status fields of ticket fedora-infrastructure#9182 https://pagure.io/fedora-infrastructure/issue/9182 18:15:41 <fm-admin> pagure.issue.comment.added -- smooge commented on ticket fedora-infrastructure#9182: "Build hosts trying to connect to port 9940 to ci.centos.org and others" https://pagure.io/fedora-infrastructure/issue/9182#comment-668890 18:16:06 <nirik> oh, and BTW: F32F-testing : 2 updates (failed) 18:16:08 <mboddu> nirik: Sorta, its just random, ftbfs is filing few tickets and then 502, then resume the script, file some bugs and then 502... 18:16:09 <nirik> :( 18:16:20 <mboddu> nirik: I resumed it today morning 18:16:26 <mboddu> And thats another thing I want to discuss 18:16:34 <nirik> well, apparently it's broken. ;) 18:17:05 <nirik> fire away with releng stuff. ;) 18:17:28 <mboddu> There is a ticket filed by kalev about it(https://pagure.io/fedora-infrastructure/issue/9186) but I am not sure if its a duplicate of 18:17:33 <mboddu> https://pagure.io/fedora-infrastructure/issue/9177 18:18:02 <mboddu> Anyway, there is only 1 releng ticket 18:18:12 <nirik> yeah, it could well be. needs investigation. 18:18:15 <mboddu> .releng 9650 18:18:16 <zodbot> mboddu: Issue #9650: No logs attached to FTBFS bugzilla - releng - Pagure.io - https://pagure.io/releng/issue/9650 18:18:30 <mboddu> The logs are missing for s390x builds 18:18:45 <mboddu> "BuildrootError: Requested repo (1785390) is DELETED" 18:18:58 <nirik> so... let me try and explain. 18:19:28 <nirik> 1. the FTBFS bugzilla bug filier is attaching the logs from the FIRST mass rebuild pass. Not the second one. 18:19:48 <nirik> I don't know if that can easily be fixed. 18:20:12 <nirik> There's a number of s390x failures that happened at different times. 18:20:15 <mboddu> huh... 18:21:00 <nirik> a) The repo is DELETED thing. That was monday morning. I had set kojira to expire repos really quickly to remove space. I fixed it monday. It should only afftect some range of packages monday morning and shouldn't affect anything after that 18:21:50 <nirik> b) There was a problem with the varnish cache on wed night. It messed up a bunch of builds, it's fixed and shouldn't have affected any after 18:22:31 <nirik> c) The random transitory 'can't read header' or 'failed to download package' cache issues. Those still affected some builds even after both passes, but decathrope is resubmitting them. 18:23:37 <mboddu> Thanks nirik 18:24:02 <nirik> so, I'd say close that ticket... it's just the way koji works... sometimes if builds fail before it makes any logs... we don't have any logs to attache 18:24:08 <mboddu> I am looking at the ftbfs script, but there is no latest failed builds in the tag :( 18:24:09 <mboddu> https://pagure.io/releng/blob/master/f/scripts/find_failures.py#_67 18:25:08 <nirik> you could change the start date to the start date of the second pass? 18:25:40 <nirik> because all the ones that failed should have failed after that? 18:25:55 <mboddu> nirik: Yeah, but what about the failed one's from first pass? 18:26:17 <nirik> if they worked after that, we don't want to file them or treat them as failed do we? 18:26:40 <nirik> since we redid all the failed ones... 18:26:49 <nirik> if we only did some of them that wouldn't work, but we did all of them... 18:27:03 <mboddu> Yeah, but there might be failed one's from the first pass as well, if we change the date, the failed one's from first pass wont be picked 18:27:25 <mboddu> Oh right 18:27:28 <nirik> but if it's still failed we would pick them from the second pass 18:27:30 <mboddu> I understood, what you mean 18:27:39 <nirik> and if not... all's well 18:28:24 <mboddu> jednorozec: ^ can you change it if the file ftbfs failed again with bz 502? 18:28:50 <nirik> anything else from releng land? 18:28:56 <mboddu> Nope, thats all 18:29:24 <nirik> ok, a few infos: 18:29:49 <nirik> #info mass update/reboots next week: monday - staging, tuesday non outage causing stuff, wed - the rest 18:30:09 <nirik> mboddu: oh, do we need to do any prep work for branching? stuff we meant to fix last time? 18:30:46 <mboddu> nirik: Nope, it should be all good, but I will recheck, there is a rust change that is needed, iirc 18:31:08 <nirik> how long did it take last time? 18:31:39 <nirik> #info koji update / outage later this week 18:31:48 <nirik> (open to when... tomorrow? wed?) 18:33:38 <nirik> anyhow, I think thats all I had. 18:33:45 <darknao> nirik: docs.fp.o is up to date now :) thanks 18:33:51 <nirik> darknao: great! 18:34:19 <mboddu> nirik: How long as in mass rebuild? 18:34:40 <nirik> mboddu: no, how long to do branching? 18:35:16 <mboddu> nirik: Just a day + mass rebuild (branching) of modules which took couple of days 18:35:40 <nirik> ok 18:35:47 <fm-admin> pagure.issue.edit -- kevin edited the close_status and status fields of ticket fedora-infrastructure#9189 https://pagure.io/fedora-infrastructure/issue/9189 18:35:48 <fm-admin> pagure.issue.comment.added -- kevin commented on ticket fedora-infrastructure#9189: "Fedora docs not being built anymore" https://pagure.io/fedora-infrastructure/issue/9189#comment-668897 18:35:54 <nirik> #endmeeting