15:04:43 <smooge> #startmeeting Infrastructure (2019-05-30)
15:04:43 <zodbot> Meeting started Thu May 30 15:04:43 2019 UTC.
15:04:43 <zodbot> This meeting is logged and archived in a public location.
15:04:43 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:04:43 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:04:43 <zodbot> The meeting name has been set to 'infrastructure_(2019-05-30)'
15:04:43 <smooge> #meetingname infrastructure
15:04:43 <smooge> #topic aloha
15:04:43 <zodbot> The meeting name has been set to 'infrastructure'
15:04:43 <smooge> #chair nirik pingou puiterwijk relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:04:43 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou puiterwijk relrod smooge tflink
15:04:46 <relrod> morning
15:04:47 <mizdebsk> .hello2
15:04:48 <zodbot> mizdebsk: mizdebsk 'Mikolaj Izdebski' <mizdebsk@redhat.com>
15:05:07 <nirik> morning
15:05:09 <smooge> hello
15:05:52 <mkonecny> .hello zlopez
15:05:53 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:06:06 <tflink> morning
15:06:19 <smooge> #topic New folks introductions
15:06:19 <smooge> #info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
15:06:19 <smooge> #info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
15:06:28 <smooge> any new people this week
15:06:38 <cverna> hi all o/
15:07:17 <smooge> #topic announcements and information
15:07:18 <smooge> #info bowlofeggs will be going on extended leave in June
15:07:18 <smooge> #info nirik will be on leave from 2019-06-03 to 2019-06-07
15:07:18 <smooge> #info many people will be at a community platform engineering f2f 2019-06-10 to 14
15:07:18 <smooge> #info Flock2Fedora 2019-08-08 -> 2019-08-11
15:07:19 <smooge> #info bodhi 4.0.0 was deployed to production. There is a known incompatibility between the bodhi 3 client and bodhi 4 server: https://pagure.io/fesco/issue/2137
15:07:31 <smooge> congrats to bowlofeggs for getting bodhi-4.0.0 out the door
15:07:52 <mkonecny> \o/
15:08:50 <cverna> #info cverna will be going on extended leave in June too :)
15:09:02 <smooge> any other announcements this weke?
15:09:06 <smooge> cverna will add
15:09:12 <nirik> hopefully we keep the lights on in june. ;)
15:09:29 <cverna> :)
15:10:00 <mizdebsk> hopefully the servers won't be on extended leave
15:10:48 <mkonecny> They don't deserve the leave?
15:11:09 <cverna> mkonecny: depends if they behave well or not :D
15:11:20 <relrod> #info broke up the s3 mirror sync into a bunch of smaller syncs and we've gone from 48 hours per sync to about ~5 minutes per individual portion synced.
15:11:22 <bowlofeggs> .hello2
15:11:23 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com>
15:11:37 <smooge> congrats relrod!
15:11:41 <nirik> relrod: great news. ;) Might make the cron less chatty when you have a chance...
15:11:47 <cverna> relrod++
15:11:48 <zodbot> cverna: Karma for codeblock changed to 8 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:11:48 <mkonecny> relrod: Great work
15:11:53 <smooge> relrod++
15:11:53 <zodbot> smooge: Karma for codeblock changed to 9 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:12:01 <mkonecny> relrod++
15:12:01 <zodbot> mkonecny: Karma for codeblock changed to 10 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
15:12:10 <cverna> cookies :D
15:12:20 <relrod> :)
15:12:22 <bowlofeggs> congrats to all of the bodhi contributors for getting bodhi 4 out the door ☺ https://bodhi.fedoraproject.org/docs/user/release_notes.html#id2
15:12:22 <relrod> nirik: will look
15:13:35 <smooge> any other announcements?
15:13:44 <smooge> #topic Oncall
15:13:44 <smooge> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:13:44 <smooge> #info relrod is on call from 2019-05-23 -> 2019-05-30
15:13:44 <smooge> #info smooge is on call from 2019-05-30 -> 2019-06-06
15:13:45 <smooge> #info no one is on call from 2019-06-06 -> 2019-06-20
15:13:47 <smooge> #info pingou is on call from 2019-06-20 -> 2019-06-27
15:13:49 <smooge> #info Summary of last week: (from relrod )
15:13:51 <smooge> relrod anything oncall this week?
15:14:13 <relrod> Just a bunch of smaller pings, not anything major...
15:14:51 <relrod> Couple of infra stg tag requests, was asked to add a var to private, things like that
15:15:17 <smooge> zodbot: oncall_take_us
15:15:17 <zodbot> smooge: Kneel before zod!
15:15:30 <smooge> you are off duty
15:15:30 * linuxmodder late
15:15:50 * relrod runs off, muahahahah
15:15:57 <relrod> kidding ;)
15:16:28 <smooge> #topic Monitoring discussion
15:16:28 <smooge> #info https://nagios.fedoraproject.org/nagios
15:16:28 <smooge> #info Go over existing out items and fix
15:17:22 <nirik> storinator down just needs the playbook run?
15:17:33 <smooge> yeah.. doing that
15:17:37 <smooge> I thought I did it last night
15:17:47 <nirik> the busgateway01 bodhi messages also should clean up?
15:17:55 <nirik> ppc8-02 and vh18 have disks out
15:18:06 <smooge> virthost18 needs to be called in. will work on that now that I am on call
15:18:49 <smooge> virthost05 and other Dell systems needs to have newer bios/drac installed and check to see if fan problems being reported are real or crap
15:19:06 <smooge> will put in tickets shortly on those to track
15:19:47 <smooge> I don't know about the busgateway one.. I guess that will need a further track down and find. will do so after meeting
15:20:06 <nirik> I think I fixed it in ansible...
15:20:14 <nirik> bodhi 4 just changed the message it was looking for
15:20:41 <smooge> ok will see when this run finishes
15:22:08 <bowlofeggs> is that the s/masher/compose/ on the message topic?
15:22:59 <nirik> yeah
15:25:49 <smooge> #topic Tickets discussion
15:25:50 <smooge> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:26:02 <smooge> Any tickets that need focus or comments this week?
15:27:01 <nirik> we have been having a pretty steady flow of tickets...
15:27:12 <nirik> but I think we have been handling them well, so kudos to all.
15:27:29 <cverna> we had pbokoc having an issue with pagure earlier today
15:27:38 <cverna> https://pagure.io/fedora-infrastructure/issue/7855
15:27:49 * nirik hasn't looked at any new tickets today yet
15:28:45 <cverna> and I tried to look at the flatpak build failure but there is nothing much I can do on OSBS side ( https://pagure.io/fedora-infrastructure/issue/7846 )
15:29:47 <nirik> it really sounds like it might be not using fast upload there, but that was checked I guess. ;(
15:31:03 <cverna> yes and the stack trace goes into koji fastupload method too
15:31:23 <cverna> so i guess we are using it
15:32:18 <nirik> the thing that changed is that we are using 2 koji frontends now...
15:32:34 <nirik> but the fast upload path fixed mbs and nothing else seemed to have a problem... untiil now. ;)
15:33:29 <cverna> yes not sure how to debug more
15:34:48 <nirik> osbs upload is where it's failing?
15:34:52 <nirik> is that from the master node?
15:36:25 <nirik> anyhow, needs further investigation...
15:36:48 <smooge> ok anything else?
15:36:53 <smooge> #topci Open Floor
15:36:57 <smooge> #topic Open Floor
15:37:39 <nirik> and it's in the upload of the log file... weird
15:38:09 <cverna> nirik: yes it looks like a race condition maybe
15:38:39 <cverna> nirik: it is happening on the OSBS nodes inside the container buildroot
15:41:38 * bowlofeggs looks around and notes how open this floor is
15:42:55 <smooge> ok the floor is now closing
15:43:39 <smooge> thank you all for coming and I hope you have a good week. See you next week
15:43:43 <smooge> #endmeeting