16:00:58 <mboddu> #startmeeting RELENG (2020-03-04)
16:00:58 <zodbot> Meeting started Tue Mar  3 16:00:58 2020 UTC.
16:00:58 <zodbot> This meeting is logged and archived in a public location.
16:00:58 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:58 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:58 <zodbot> The meeting name has been set to 'releng_(2020-03-04)'
16:00:58 <mboddu> #meetingname releng
16:00:58 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec
16:00:59 <mboddu> #topic init process
16:00:59 <zodbot> The meeting name has been set to 'releng'
16:00:59 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz
16:01:05 <nirik> morning
16:01:11 <mboddu> Hello everyone and welcome to new time :)
16:01:25 <nirik> it feels a lot like the old time.
16:01:33 <mboddu> Haha :)
16:01:37 <mboddu> How goes nirik ?
16:01:48 <nirik> not bad. How about with you...
16:02:01 <mboddu> Cant complain
16:02:10 <smooge> morning
16:02:16 * mboddu waves at smooge
16:02:26 <nirik> hey smooge
16:02:27 <mboddu> So, lets get started
16:02:27 <sharkcz> hi all
16:02:41 <mboddu> Hey sharkcz
16:02:46 <mboddu> #topic New Schedule
16:02:51 <decaren> hello all
16:03:54 <nirik> note: our good old friend DST hits the us next week.
16:04:07 <mboddu> So, sharkcz raised a request to change the timezone to EST rather than UTC
16:04:08 <nirik> are we sticking to the same time UTC? or moving with the time change?
16:04:14 <mboddu> Which I would like to do
16:04:15 <nirik> yeah, that
16:04:32 <mboddu> I want to stick with 11 EST, which works great for me
16:04:43 <mboddu> Hello decaren
16:04:56 <nirik> I'm fine either way
16:05:48 <mboddu> I am pretty sure sharkcz is fine with it
16:05:54 <mboddu> smooge: Are you okay with it?
16:05:55 <sharkcz> yup
16:06:04 <smooge> i am ok
16:06:43 <mboddu> #info We will change the meeting time to 11:00 AM EST every Tuesday
16:07:27 * mboddu is jumping with joy and not to worry about DST anymore
16:07:39 <mboddu> I will make the changes and announcement later today
16:07:56 <smooge> if you are in other timezones you will have to move to the US East Coast
16:08:11 <nirik> heh
16:08:14 <smooge> so 11:00 am EST and not 11:00 EDT when that starts next week
16:09:01 <mboddu> smooge: 11:00 AM eastern time
16:09:14 <smooge> EST is Eastern Standard Time (winter) and EDT is Eastern Daylight Savings TIme
16:09:14 * mboddu can never get EST vs EDT
16:09:24 <smooge> so just say Eastern Time
16:09:25 <mboddu> So, its EST
16:09:31 <mboddu> Yes
16:09:49 <smooge> today 11:00 am is EST. Next week 11:00 am on east coast is EDT
16:10:05 <mboddu> Okay, 11:00 AM Eastern Time
16:10:38 <mboddu> ping cverna on a ticket which I would like to discuss later, if he is around
16:10:39 * cverna waives
16:10:44 <mboddu> cverna: What a timing
16:10:47 <cverna> :)
16:10:54 <cverna> I can read your mind :)
16:11:20 <mboddu> #topic quay.io access denied while uploading containers
16:11:35 <mboddu> cverna: Okay, that is scary, I need to find the magneto's helmet
16:11:37 <mboddu> :)
16:11:50 <cverna> mboddu: ha ha :)
16:11:56 <nirik> oh, I have a related item to this once we are done with this.
16:12:17 <mboddu> cverna: So, when I synced the latest containers, I got an error with quay.io authorization
16:12:22 <mboddu> "FATA[0001] Error writing blob: Error initiating layer upload to /v2/fedora/fedora/blobs/uploads/ in quay.io: unauthorized: access to
16:12:22 <mboddu> the requested resource is not authorized"
16:12:48 <mboddu> I looked at the ansible private repo and tried to login with it in quay.io website, but it didn't work
16:13:14 <cverna> yeah I think this is a robot account and token so not sure you can use it on the website
16:13:15 <mboddu> I have no idea how to change password and stuff
16:13:31 <mboddu> Well, there is that then
16:13:41 <cverna> we can look at it together if you want, so that you know where to look at next time
16:14:05 <mboddu> cverna: Sure, how about in 2 hours? I have a meeting after this meeting
16:14:08 <mboddu> Or may be tomorrow
16:14:28 * mboddu not sure if cverna will be around after 2 hours
16:14:37 <cverna> mboddu: yeah tomorrow will be easier
16:14:49 <mboddu> Okay, I will ping you tomorrow then, thanks cverna
16:14:56 <cverna> +1
16:15:04 <mboddu> nirik: You are up, create a new topic if needed
16:15:13 <nirik> well, it's containers. ;)
16:15:23 * cverna escape
16:15:23 <nirik> but there are lots of complaints about our registry...
16:15:29 <nirik> throwing 503's.
16:15:34 * mboddu guessing 503's
16:15:36 <mboddu> Haha :)
16:15:36 <cverna> yeah I saw the ticket
16:15:37 <nirik> and it seems it's the cdn... not us
16:15:54 <nirik> so, I am wondering if we shouldn't move that to cloudfront...
16:16:13 <nirik> or something.
16:16:23 <mboddu> nirik: Isn't the plan to kill registry.fp.o totally at one point?
16:16:33 <mboddu> May be we can move to quay.io totally
16:16:34 <nirik> yeah, but so far we can't yet.
16:16:47 <nirik> we need multiarch and flatpaks
16:16:49 <cverna> yeah quay.io still lacks features
16:16:52 <cverna> :(
16:17:00 <mboddu> Right
16:17:05 <cverna> how much work would that be nirik  to move to cloudfront ?
16:17:15 * cverna has no idea of what that involves
16:17:20 <mboddu> I know the problem with multiarch, but didn't expect flatpaks are also not supported
16:17:56 <nirik> not too much I don't think. Just setting up the amazon end and then adjusting our proxies
16:18:17 <nirik> we could also try and see why the current thing is giving 503's...
16:18:33 <nirik> but I didn't see any issues on their dashboard
16:18:45 <cverna> yes it is curious to have a 503 on a pull request
16:19:25 <cverna> it is still going through the proxies before going in the CDN ?
16:19:35 <nirik> yeah.
16:19:54 <nirik> it basically sends the /v2/ stuff to the cdn,unless its from the cdn
16:20:00 <mboddu> I did hit it once while I tried either skopeo inspect or podman pull
16:20:02 <cverna> ok I remember Patrick doing some magic on the proxies about the auth, might be worth looking at it
16:20:26 <cverna> it looks like it is not all the time so maybe a proxy is not behaving correctly
16:20:29 * puiterwijk reads back.
16:20:31 <nirik> lots of people are hitting it now tho... and it's not good.
16:20:48 <nirik> cverna: one report had the cdn doing it, not our proxies.
16:20:56 <puiterwijk> nirik: yes, I'm wondering whether the CDN-detection is different from cdn77...
16:21:22 <puiterwijk> i.e. whether the header we used to detect cdn77 requests is different in cloudfront
16:21:36 <nirik> we haven't switched anything to cloudfront (yet)
16:21:41 <puiterwijk> Okay
16:21:47 <nirik> I talked about it, but haven't done it. ;)
16:21:59 <nirik> but cdn77 might have changed...
16:22:17 <puiterwijk> Okay. Then my quick-reading failed.
16:23:05 <nirik> anyhow, just wanted to raise it up and perhaps some of us could schedule a time to look more closely and fix it.
16:23:29 <cverna> nirik: we can try to look at it after the Ops standup later today if you want
16:23:46 <nirik> sure
16:24:12 <cverna> +1
16:24:14 <mboddu> And if we cant figure it out, we can try cloudfront since nirik said its an easy change
16:25:48 <cverna> sounds good
16:26:04 * cverna has to go afk, catch you all later
16:26:13 <nirik> sure
16:26:16 <mboddu> #info We will try to debug the 503 error on registry.fp.o later today and probably move it to cloudfront
16:26:19 <mboddu> Bye cverna
16:29:01 <mboddu> #topic #9266 prune EOL content from OSTree repos
16:29:06 <mboddu> #link https://pagure.io/releng/issue/9266
16:29:12 <mboddu> nirik: Can we close that ticket now?
16:29:35 <mboddu> And is there any documentation on how to do it for future needs?
16:29:48 <nirik> I think it's not been done yet because we need to do a bit more adjustment on the permissions.
16:30:29 <mboddu> Okay
16:31:01 <nirik> which we are waiting until after freeze for
16:33:06 <mboddu> Okay, thanks for the update
16:33:13 <mboddu> #topic #9169 Block unretirement commits
16:33:23 <mboddu> #link https://pagure.io/releng/issue/9169
16:33:37 <mboddu> Should we update pdc-updater or just hot fix it?
16:33:59 <mboddu> We decided to change the date to a day in advance, but we haven't done it yet
16:34:20 <nirik> yeah, I guess either way would be fine.
16:35:27 <mboddu> Where is pdc-updater running? pdc-backend01.phx2.fp.o?
16:37:21 <nirik> yes, I think so.
16:37:28 <nirik> or 02/03... but I think its 01
16:38:22 <mboddu> Its on 01
16:38:45 <mboddu> Again, infra freeze, should we change it after the freeze ends?
16:39:47 <nirik> sure, or could request a freeze break. I would think the change is small...
16:40:46 <mboddu> Okay, I will create a diff and send it to infra list for FBR
16:41:07 <nirik> sounds good.
16:42:38 <mboddu> #topic #9246 Please block `fedora-cisco-openh264` from pungi composes
16:42:44 <mboddu> #link https://pagure.io/releng/issue/9246
16:43:07 <mboddu> Since openh264 builds are done in a separate tag, we should not have problem with this change
16:43:28 <nirik> yeah, I would think it would be fine...
16:43:40 <nirik> if we wanted to make sure we could blacklist the packages in pungi config...
16:43:40 <mboddu> I thought of doing a scratch image build with openh264 packages mentioned in ks file, but now I think about it, I dont need to test it as well
16:44:22 <nirik> we have the 'filter' in pungi...
16:44:39 <mboddu> Since the repo that gets generated in composes are coming from fxx tag which should not have the openh264 stuff
16:44:46 <mboddu> Right, thats an option as well
16:45:07 <mboddu> We will filter them out as a precaution
16:45:29 <nirik> yeah, I would think it would be fine. we could run a rawhide compose and set it to not sync out to make sure?
16:46:32 <mboddu> #info As a precaution, we will filter out the packages in pungi configs and test it in rawhide compose by disabling sync.
16:47:22 <mboddu> nirik: Just wondering why cant we ship images with preinstalling openh264 stuff?
16:48:20 <mboddu> s/preinstalling/preinstalled/
16:50:03 <nirik> patents. we don't have a patent to distribute it
16:50:08 * nirik got a phone call
16:50:49 <mboddu> Okay, lets talk about it later
16:50:51 <nirik> or license. or something.
16:51:00 <nirik> but cisco does
16:51:30 <mboddu> I dont know why we cant do it, since we are using the repo as normally as everyone else would
16:51:47 <mboddu> #topic Open Floor
16:52:17 <mboddu> decaren: Are you still around? Wanna introduce yourself?
16:52:44 <decaren> hi everyone, I'm new to the fedora-releng world
16:53:15 <decaren> wanted to try and figure out where to get started, help out where I can
16:53:28 <nirik> mboddu: we do not distribute the rpms. Those are all distributed from cisco's cdn.
16:53:37 <nirik> welcome decaren!
16:54:01 <mboddu> Welcome to Fedora RelEng decaren
16:55:11 <decaren> I guess my main question is, would this be a place to start, or should I go get used to building packages in fedora first?  Day job is developer with previous ops experience
16:55:22 <mboddu> decaren: The best place to start is https://docs.pagure.org/releng/ and understanding fedora packaging and the flow of builds also helps - https://docs.fedoraproject.org/en-US/packaging-guidelines/
16:56:05 <nirik> decaren: if you have time to just hang out in #fedora-releng too and ask questions or offer help thats a good way forward.
16:56:07 <decaren> sounds good.  I will start there.  Thanks!
16:56:13 <mboddu> decaren: I would say building packages is a good start and ^ the docs I mentioned above and following this meeting and #fedora-releng is the way to go
16:57:22 <mboddu> Anything else?
16:57:26 <mboddu> We got 3 more min :)
16:57:48 <decaren> Thanks all!
16:58:23 <decaren> Anyone wanna help me with my borked F31 on my laptop everytime I try to use grubby to switch back to cgroups v1 for docker :)
16:58:41 <nirik> decaren: switch to podman. ;)
16:58:47 <mboddu> There is an issue on bz for it
16:58:57 * mboddu tries to find the ticket
16:59:46 <mboddu> decaren: For such questions, #fedora and #fedora-devel are best channels to get your answers
17:00:11 <decaren> no problem.
17:00:16 <mboddu> With that, I will close this meeting
17:00:20 <mboddu> Thanks everyone for joining
17:00:28 <mboddu> See you all next week
17:00:31 <mboddu> #endmeeting