16:00:18 <mboddu> #startmeeting RELENG (2020-02-19)
16:00:18 <zodbot> Meeting started Wed Feb 19 16:00:18 2020 UTC.
16:00:18 <zodbot> This meeting is logged and archived in a public location.
16:00:18 <zodbot> The chair is mboddu. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:18 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
16:00:18 <zodbot> The meeting name has been set to 'releng_(2020-02-19)'
16:00:19 <mboddu> #meetingname releng
16:00:19 <mboddu> #chair nirik sharkcz pbrobinson pingou mboddu dustymabe ksinny jednorozec
16:00:19 <mboddu> #topic init process
16:00:19 <zodbot> The meeting name has been set to 'releng'
16:00:19 <zodbot> Current chairs: dustymabe jednorozec ksinny mboddu nirik pbrobinson pingou sharkcz
16:00:28 <mboddu> Thanks nirik for reminding it
16:00:32 <dustymabe> .hello2
16:00:33 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
16:00:33 <nirik> hey
16:00:35 * mboddu is still in Tue mode
16:00:44 <nirik> it's still monday to me. ;(
16:01:10 <mboddu> dustymabe: I have a question for you, but lets take it to releng
16:01:16 <mboddu> I mean #fedora-releng
16:01:22 <mboddu> So, lets get started
16:02:06 <dustymabe> mboddu: ok
16:03:17 <mboddu> #topic #9266 prune EOL content from OSTree repos
16:03:23 <mboddu> #link https://pagure.io/releng/issue/9266
16:03:46 <mboddu> dustymabe: So, finally pruning stuff :)
16:04:31 <dustymabe> mboddu: correct :)
16:04:48 <dustymabe> mboddu: we can do this in a few different ways
16:05:05 <dustymabe> we can either prune everything AFTER we do the migration
16:05:09 <dustymabe> or we can do it before
16:05:10 <mboddu> What does "--no-prune" will give us from the first script in that ticket?
16:05:23 <mboddu> Better do it after migration, right?
16:05:40 <dustymabe> mboddu: `--no-prune` is like `--dry-run`
16:05:48 <mboddu> Ah okay
16:05:56 <dustymabe> mboddu: yeah maybe after migration is better
16:06:06 <dustymabe> looks like we have 2T volume so it should all fit
16:07:13 <dustymabe> so maybe we should talk migration strategy
16:08:31 <mboddu> Okay
16:08:43 <dustymabe> what do you folks think about how we should pull off the migration?
16:08:54 <mboddu> dustymabe: 2T in cloudfront or somewhere else?
16:09:59 <dustymabe> mboddu: it's a 2T netapp volume
16:10:05 <nirik> I think like any other data move:
16:10:12 <dustymabe> see https://pagure.io/releng/issue/8811#comment-602395
16:10:48 <nirik> sync data, schedule time for no changes, do final sync, switch web/pointers to the new location
16:11:43 <dustymabe> nirik: so we'd want to do it at a time where no pungi composes are happening, correct?
16:11:49 <nirik> yeah
16:12:11 <mboddu> nirik: If we are only touching 27 and 28, we can do it at anytime
16:12:25 <dustymabe> mboddu: we're migrating the entire repo
16:12:45 <dustymabe> since pungi still makes silverblue, we have to care about pungi
16:12:46 <mboddu> Okay, then dont mind me, you guys are right
16:13:18 * mboddu thought we are only migrating eol'd content
16:13:53 <dustymabe> nirik: so should we enumerate what needs to happen
16:13:59 <dustymabe> 1. make backup of content
16:14:11 <dustymabe> ^^ I guess this can just be "leave the content in place"
16:14:12 <nirik> backup?
16:14:18 <nirik> ah yeah
16:14:39 <dustymabe> ok. so let's do it different
16:14:49 <dustymabe> 1. make sure no writers to ostree repos are running
16:15:01 <dustymabe> ^^ includes pungi and new-updates sync
16:15:19 <dustymabe> 2. copy content from production repos into new netapp volume
16:15:44 <dustymabe> 3. maybe mv prod repos to another directory (minimizes confusion)
16:16:19 <dustymabe> 4. set up /mnt/koji/ostree /mnt/koji/compose/ostree mounts on composer to point to new netapp volume
16:16:37 <dustymabe> 5. do we need to do anything to our webservers/cloudfront?
16:16:55 <nirik> I'd add a '0. make initial copy of content' which we can do anytime and will make step 2 much quicker
16:17:15 <nirik> as long as the new volumes are in the same path for webserver we should be fine for cloudfront
16:19:11 <dustymabe> nirik: i.e. an initial rsync and then a second rsync will take less time?
16:19:20 <nirik> yeah.
16:19:29 <dustymabe> ok sounds good to me
16:19:34 <dustymabe> anything else missing?
16:19:34 <nirik> since it will only need to copy the changes the second time
16:19:52 <nirik> when did we want to schedule it?
16:20:03 <dustymabe> tomorrow?
16:20:03 <mboddu> On that, next week is freeze time
16:20:32 <dustymabe> we can do the initial rsync today
16:20:40 <nirik> I suppose tomorrow might work...
16:22:03 <mboddu> #info We will work on migration first and then pruning, nirik will start working on migration tomorrow and then mboddu will work on pruning later
16:22:38 <dustymabe> mboddu: i should be able to handle pruning myself once we get things over into openshift
16:22:44 <nirik> I can't start anything today until this afternoon. I am in meetings all morning
16:23:11 <nirik> but I can look this afternoon and we can try and do the switch tomorrow after branched/rawhide finish?
16:23:30 <dustymabe> nirik: +1 i'll grab you in 3 or 4 hours
16:23:37 <dustymabe> if that's ok
16:24:11 <nirik> more like 4-5... but feel free to ping me and I'll let you know
16:24:25 <mboddu> #undo
16:24:25 <zodbot> Removing item from minutes: INFO by mboddu at 16:22:03 : We will work on migration first and then pruning, nirik will start working on migration tomorrow and then mboddu will work on pruning later
16:24:40 <mboddu> #info We will work on migration first and then pruning, nirik will start working on migration tomorrow and then dustymabe will work on pruning later
16:24:47 <mboddu> mboddu will help as needed :)
16:26:54 <dustymabe> +1
16:27:00 <dustymabe> ok i gotta go run my other meeting
16:27:03 <dustymabe> thanks mboddu and nirik
16:27:55 <mboddu> Thanks dustymabe
16:27:58 <mboddu> Okay, lets go back to old tickets
16:28:05 <mboddu> #topic #7400 Rethink how we handle networking config with createImage (oz / ImageFactory)
16:28:12 <mboddu> #link https://pagure.io/releng/issue/7400
16:30:26 * mboddu trying to find koji ticket
16:30:34 <nirik> was there one?
16:31:20 <mboddu> I dont think so
16:31:50 <mboddu> #info mboddu will create a upstream koji ticket and will check with mikem
16:31:54 <nirik> so we could file one I guess.. I don't have time to commit to do this one...
16:32:33 <nirik> sounds good
16:32:46 <mboddu> nirik: Understandable
16:34:27 <mboddu> #topic #7555 scratch permission to build livecd for users
16:34:34 <mboddu> #link https://pagure.io/releng/issue/7555
16:35:16 <nirik> so, not sure whats left here...
16:35:25 <nirik> are our compose images still scratch?
16:35:31 <mboddu> nirik: Yes
16:35:49 <mboddu> But can we give the users 'livemedia' permissions?
16:36:19 <nirik> hum, I wonder how that interacts with GC
16:37:24 <bcotton> fwiw (i haven't read through the entire ticket), but that change proposal was never submitted
16:38:34 <nirik> so, it looks like our cloud images at least are normal... but we never move them to updates tags so they get GCed
16:38:36 <mboddu> Sorry, they are not running using scratch
16:39:31 <nirik> and even our GA ones get GCed. ;(
16:40:02 <mboddu> But we import them into our compose, so..., thats still fine
16:40:23 <nirik> Wed Oct 23 17:08:41 2019: Fedora-Xfce-31-1.9 tagged into f31-updates-candidate by releng
16:40:23 <nirik> Sat Oct 26 12:15:33 2019: Fedora-Xfce-31-1.9 untagged from f31-updates-candidate by oscar
16:40:23 <nirik> Fri Jan 10 12:24:35 2020: Fedora-Xfce-31-1.9 tagged into trashcan by oscar
16:40:23 <nirik> Sat Feb  8 01:39:33 2020: Fedora-Xfce-31-1.9 untagged from trashcan by oscar
16:40:55 <nirik> so, we might want to start tagging some of them to keep em... but I guess we still have them on ftp / mirror
16:41:07 <mboddu> Exactly
16:41:30 <nirik> anyhow, I'd be open to granting some people perms to spin media...
16:41:33 <mboddu> bcotton: Yeah, I haven't seen the change as well
16:41:48 <nirik> this might go hand in hand with the idea of having a acutal list of maintainers.
16:41:53 * kalev is interested to spin Workstation test composes.
16:42:22 * Southern_Gentlem interested
16:42:32 <mboddu> nirik: Its good to give them to spins/labs maintainers, but what about people who want add new media?
16:42:47 <mboddu> s/who want/who want to/
16:42:51 <nirik> sure, we could just also do it on a case by case basis?
16:44:16 <nirik> I can't recall... can you use a local ks for a scratch livemedia build?
16:44:56 <mboddu> I am not sure either, but I remember giving a ks file from my fork, so that should help
16:45:59 <nirik> yeah, looks like no, but I think it will accept any pagure.io git for them
16:46:26 <mboddu> So, how should we handle this?
16:46:30 <nirik> so, whats next steps here?
16:46:31 <nirik> ha.
16:47:02 <mboddu> Lets give it couple of folks who need it and see how they use it and get feedback and then announce it out for people who want it?
16:47:09 <nirik> how about we grant Southern_Gentlem and kalev perm and ask them to try it out and report back to us if it's useful/working?
16:47:21 <bcotton> +!
16:47:24 <nirik> nice. great minds think alike. ;)
16:47:27 <mboddu> Lol, exactly the same thing twice :)
16:47:27 <bcotton> +1 too
16:47:43 <mboddu> Southern_Gentlem and kalev : Are you okay with this?
16:47:47 <Southern_Gentlem> go with kalev first because i will need to work with someone to learn what i am doing
16:48:26 <kalev> mboddu: sure, works for me
16:48:53 <mboddu> Southern_Gentlem: Okay, but its simple
16:49:11 <mboddu> .fasinfo kalev
16:49:12 <zodbot> mboddu: User: kalev, Name: Kalev Lember, email: klember@redhat.com, Creation: 2009-02-25, IRC Nick: kalev, Timezone: Europe/Stockholm, Locale: en, GPG key ID: 8F62AF74, Status: active
16:49:13 <nirik> I am wondering if the limitations will make it not so useful, but we can see.
16:49:14 <zodbot> mboddu: Approved Groups: cla_fedora cla_done fedorabugs provenpackager cla_fpca gitspin-kickstarts qa +packager hosted-content @gnome-sig
16:49:25 <Southern_Gentlem> mboddu,  i have been doing respins for years on my own equipment
16:49:35 <nirik> you do have to pass it a install tree... so it has to use a existing one of those...
16:49:48 <mboddu> nirik: install tree and the repo as well
16:50:03 <mboddu> Southern_Gentlem: Right, it should definitely help you a lot
16:50:38 <mboddu> kalev: You should have the permissions now
16:50:44 <kalev> nice, thanks!
16:50:45 <nirik> repo for ks yeah
16:51:24 <nirik> koji spin-livemedia --help
16:51:37 <nirik> make sure to use --scratch :)
16:51:50 <mboddu> kalev: Right
16:51:55 <kalev> ok, let me give it a try :)
16:53:32 <nirik> mboddu: 7min left... anything else?
16:53:52 <bcotton> o/
16:55:24 <mboddu> #topic Open Floor
16:55:38 * bcotton has a thing
16:55:38 <nirik> freeze and bodhi enable is next tuesday?
16:55:45 <mboddu> nirik: So, I found the issue with stuck 32 compose, I will work with dustymabe on that
16:55:49 <bcotton> nirik: correct
16:55:50 <mboddu> nirik: yes
16:56:20 <nirik> I have something for that. can we change the time as we discussed a bunch last cycle? pretty please?
16:58:09 <mboddu> nirik: ?
16:58:15 <mboddu> Which one are you talking about?
16:58:43 <nirik> the freeze starting at 00:00 tuesday... this is confusing to everyone
16:59:06 <kalev> ah yes, can you just move it one minute earlier? no real changes, just makes it much more clear when it actually starts
16:59:37 <kalev> first test scratch build running: https://koji.fedoraproject.org/koji/taskinfo?taskID=41658761
16:59:41 <nirik> I was hoping to move to to something like tuesday at 14UTC (or 15, whatever we normally release on)
17:00:06 <mboddu> nirik: Ah, right, 14UTC seems okay
17:00:09 <mboddu> bcotton: ^
17:00:20 <nirik> or later. or whatever. just not 00:00
17:00:43 <kalev> agreed, just as long as it's not 00:00 :)
17:00:44 <bcotton> WFM, so long as everyone has a shared understanding
17:00:55 <bcotton> i can update the schedule text to say 1400 UTC
17:01:18 <nirik> mboddu: can you be around at 14utc to enable bodhi?
17:01:26 <Southern_Gentlem> it would be more consistant across the board, people know we release around 1400
17:01:48 <mboddu> nirik: Yes, I would be around
17:02:02 <nirik> if everyone is ok with that, perhaps a email this week with a reminder/note about it to devel-announce would be good.
17:02:12 <nirik> and we should do the same for final, etc.
17:02:36 <bcotton> nirik++
17:02:42 <nirik> so for final that would mean a updates push at 14 and then freezing.
17:03:24 <bcotton> #link https://pagure.io/fedora-project-schedule/issue/175
17:03:25 <mboddu> Right
17:03:33 <mboddu> Thanks bcotton
17:03:45 <mboddu> So, lets close this for today
17:03:49 * mboddu got to run
17:03:54 <mboddu> Thanks everyone for joining
17:03:55 <bcotton> i'll try to get the schedule updated by EOB tomorrow so that you can point to it when sending the remidner
17:04:12 <mboddu> bcotton: Okay, please ping me once its done
17:04:12 <nirik> bcotton++
17:04:12 <zodbot> nirik: Karma for bcotton changed to 12 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:04:19 <bcotton> mboddu: ack
17:04:20 <nirik> thanks mboddu
17:04:25 <mboddu> bcotton++ nirik++ kalev++ Southern_Gentlem++
17:04:26 <zodbot> mboddu: Karma for bcotton changed to 13 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:04:29 <zodbot> mboddu: Karma for jbwillia changed to 2 (for the current release cycle):  https://badges.fedoraproject.org/tags/cookie/any
17:04:39 <mboddu> #endmeeting