18:05:35 #startmeeting Fedora Infrastructure Ops Daily Standup Meeting 18:05:35 Meeting started Mon May 25 18:05:35 2020 UTC. 18:05:35 This meeting is logged and archived in a public location. 18:05:35 The chair is cverna. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:05:35 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:05:35 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:05:35 #chair cverna mboddu nirik smooge 18:05:35 #meetingname fedora_infrastructure_ops_daily_standup_meeting 18:05:35 Current chairs: cverna mboddu nirik smooge 18:05:35 The meeting name has been set to 'fedora_infrastructure_ops_daily_standup_meeting' 18:05:50 #info meeting is 30 minutes MAX. At the end of 30, its stops 18:05:51 #info agenda is at https://board.net/p/fedora-infra-daily 18:06:04 #topic Tickets needing review 18:06:04 #info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=1 18:07:42 * nirik isn't really here since it's a holiday, but I can be lurking. ;) 18:08:18 * cverna wait and see who is here then :P 18:08:34 we can do a short meeting. I am just working on iad2... 18:08:52 .ticket 8937 18:08:53 nirik: Issue #8937: Rawhide builds are not pushed to stable - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8937 18:09:02 that seemed to fix itself? 18:10:12 yeah I did not do anything to this, I tried to look at kibana but the logs were gone 18:10:30 the logging pods keep getting OOMed. 18:10:31 :( 18:10:41 :( 18:10:45 I made the nodes in iad2 with more memory, hopefully that will help that 18:11:01 I ll try to monitor the pods and to see if I see some specific traceback 18:11:16 is there an upstream ticket on this? or ? 18:11:54 I don't think this is upstream, I think this is more of a problem with celery and rabbitmq but I am not 100% sure 18:12:53 hello sorry was poking disk space 18:13:32 I think we can close that and I ll keep an eye on the bodhi in the next few days 18:13:38 ok, so keep it open and loook a while, then close if nothing can be found? 18:13:40 I ll update that ticket 18:13:43 or that 18:13:44 +1 18:14:18 pagure.issue.new -- kevin opened a new ticket fedora-infrastructure#8941: "get ipad01/02.iad2 replicating with ipa01/02.phx2" https://pagure.io/fedora-infrastructure/issue/8941 18:14:27 .ticket 8939 18:14:28 nirik: Issue #8939: centos-ci not processing fedora-messaging messages? - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8939 18:14:58 this does not seem to be fixed. 18:15:01 RABBITMQ_QUEUE WARNING - messages WARNING (24480), messages_ready OK (24480) messages_unacknowledged OK (0) consumers OK (0) 18:15:22 pagure.issue.edit -- cverna edited the close_status and status fields of ticket fedora-infrastructure#8937 https://pagure.io/fedora-infrastructure/issue/8937 18:15:23 pagure.issue.comment.added -- cverna commented on ticket fedora-infrastructure#8937: "Rawhide builds are not pushed to stable" https://pagure.io/fedora-infrastructure/issue/8937#comment-654181 18:15:52 pagure.issue.comment.added -- kevin commented on ticket fedora-infrastructure#8939: "centos-ci not processing fedora-messaging messages?" https://pagure.io/fedora-infrastructure/issue/8939#comment-654183 18:16:11 I guess we can move it to wiating on asignee and see what they say to my last comment there? 18:16:53 ok doing it 18:17:18 pagure.issue.edit -- cverna edited the priority fields of ticket fedora-infrastructure#8939 https://pagure.io/fedora-infrastructure/issue/8939 18:18:21 pagure.issue.new -- kevin opened a new ticket fedora-infrastructure#8942: "rabbitmq cluster in iad2 not clustering" https://pagure.io/fedora-infrastructure/issue/8942 18:18:33 ok, the next set are iad2 related. 18:19:04 .ticket 8940 18:19:05 nirik: Issue #8940: port python-tag2distrepo koji hub plugin to python3 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8940 18:19:07 .ticket 8941 18:19:08 nirik: Issue #8941: get ipad01/02.iad2 replicating with ipa01/02.phx2 - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8941 18:19:08 .ticket 8942 18:19:11 nirik: Issue #8942: rabbitmq cluster in iad2 not clustering - fedora-infrastructure - Pagure.io - https://pagure.io/fedora-infrastructure/issue/8942 18:19:35 I'm hoping puiterwijk could help with some or all of these, but if someone else can, feel free. 18:19:55 do we need help from abompard for the rabbitmq cluster ? 18:20:15 we can see if he has a bit of time to help us 18:20:28 yeah, I mentioned him there too. if he has time, great! 18:21:05 will he have access to the box tho ? 18:21:41 yep. Just use bastion-iad01.fedoraproject.org in your .ssh/config for iad hosts... and the same people who have access to stuff in phx2 should have access to the iad2 counterpart 18:21:59 cool 18:22:09 I have almost all the non build stuff built now. The exceptions are: mailman01 and notifs 18:22:15 I ll try to catch him tomorrow and see if he can help us or not :) 18:22:25 I was thinking we might want ti migrate those vm's and reconfigure them rather than reinstall... 18:23:19 we could also try doing new installs... but I am not sure how far we will get... mailman01 has some... interesting deps. notifs-backend01 is.... fedora 27 I think? 18:23:45 so I am sure modern fedora would break it due to python2 if nothing else. 18:23:49 oh my 18:24:17 I think nils started to look at mailman but this are quite some work there to package it 18:24:34 I'm working on build sys side now... hopefully will have that setup in the next few days. 18:24:57 then I want to make a checklist and have people go and validate things. 18:25:24 nirik: for tag2distrepo is it the package that needs to be python3 or there is an error in the code base ? 18:25:55 the package. its python2 currently. 18:26:06 https://koji.fedoraproject.org/koji/buildinfo?buildID=1272405 18:26:12 ok 18:26:21 the code base looks sane 18:26:26 for python3 18:27:06 https://pagure.io/releng/tag2distrepo/commits/master 18:27:13 yeah, hopefully it will be easy... 18:27:14 I ll check with siddharthvipul if he wants to do a PR for the package to have it build for python3 18:27:28 thats blocking me configuring the koji hubs 18:28:01 ok yeah would be nice to have soon 18:28:11 cverna: you want me to do osbs in iad2? or could you take that on? 18:28:40 happy to do it 18:28:53 cool. Or I can start it and hand it to you to finish? 18:29:08 yeah sure that works too 18:29:25 I would like to move prod to rhel7 instead of fedora 18:29:25 that way I can get all the ips/vmhosts all set... 18:29:39 in phx2 I did stg but no prod 18:29:48 ok. I thought there was some blocker for that... 18:29:50 +1 18:29:53 but I can't recall what it was.. 18:30:14 no the blocker is for rhel8 because no docker 18:30:23 rhel7 seem to work fine :) 18:30:30 pagure.issue.comment.added -- msrb commented on ticket fedora-infrastructure#8939: "centos-ci not processing fedora-messaging messages?" https://pagure.io/fedora-infrastructure/issue/8939#comment-654187 18:30:41 ahhh. ok. 18:30:50 fine with me then. 18:31:24 cverna, no docker as in Moby yes. But crio and podman are there. You should be able to run/translate what-ever you have to use podman very easily (just alias docker => podman) 18:32:55 plarsen: yep but no :) OSBS was writting using the docker python binding API and moving to something else is a lot of work. 18:33:07 plarsen: Last I talked with the upstream this was not there priority for now 18:33:14 their* 18:34:05 ok we are done for the 30mins 18:34:10 #endmeeting