15:00:03 <smooge> #startmeeting Infrastructure (2019-06-06)
15:00:03 <zodbot> Meeting started Thu Jun  6 15:00:03 2019 UTC.
15:00:03 <zodbot> This meeting is logged and archived in a public location.
15:00:03 <zodbot> The chair is smooge. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:03 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:00:03 <zodbot> The meeting name has been set to 'infrastructure_(2019-06-06)'
15:00:03 <smooge> #meetingname infrastructure
15:00:03 <zodbot> The meeting name has been set to 'infrastructure'
15:00:03 <smooge> #topic aloha
15:00:04 <smooge> #chair nirik pingou puiterwijk relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
15:00:04 <zodbot> Current chairs: abompard bowlofeggs cverna mizdebsk mkonecny nirik pingou puiterwijk relrod smooge tflink
15:00:16 <smooge> hello for the people who are here this week
15:00:28 <cverna> hello
15:00:54 <abompard> hi!
15:01:33 <smooge> will wait a couple of minutes for anyone else to be able to show up
15:02:12 <tflink> hello
15:02:14 <smooge> #topic announcements and information
15:02:14 <smooge> #info Most of infrastructure will be unavailable from 2019-06-09 -> 2019-06-14
15:02:14 <smooge> #info -- Put in tickets for any items and expect a multiple day lag
15:02:14 <smooge> #info There will be NO meeting next week
15:02:14 <smooge> #info bowlofeggs will be going on extended leave in 2019-06
15:02:15 <smooge> #info cverna will be going on extended leave in 2019-06
15:02:17 <smooge> #info nirik is on leave from 2019-06-03 to 2019-06-07
15:02:19 <smooge> #info Flock2Fedora 2019-08-08 -> 2019-08-11
15:02:23 <smooge> #info Site trip to PHX2 will be 2019-07
15:02:25 <smooge> #info buildvm_armv7 Koji builders were upgraded from F27 to F29 with 4.19 kernel - mizdebsk
15:02:27 <smooge> #info updating mock/createrepo in Koji is blocked by PPC64 removal from EPEL - mizdebsk
15:02:34 * jlanda was going to introduce himself since i lurk around and i realized that never have introduced myself, but gtg so see you tomorrow ^^
15:03:14 <cverna> jlanda: o/
15:03:24 <smooge> I am not sure why mock/createrepo is blocked by PPC64 removal
15:03:36 <smooge> hello jlanda
15:04:59 <dustymabe> .hello2
15:05:00 <zodbot> dustymabe: dustymabe 'Dusty Mabe' <dusty@dustymabe.com>
15:05:12 <bowlofeggs> .hello2
15:05:13 <zodbot> bowlofeggs: bowlofeggs 'Randy Barlow' <rbarlow@redhat.com>
15:05:48 <mkonecny> .hello zlopez
15:05:49 <zodbot> mkonecny: zlopez 'Michal Konečný' <michal.konecny@packetseekers.eu>
15:06:15 <smooge> ok #topic Oncall
15:06:15 <smooge> #info https://fedoraproject.org/wiki/Infrastructure/Oncall
15:06:15 <smooge> #info smooge is on call from 2019-05-30 -> 2019-06-06
15:06:15 <smooge> #info no one is on call from 2019-06-06 -> 2019-06-20
15:06:16 <smooge> #info pingou is on call from 2019-06-20 -> 2019-06-27
15:06:17 <smooge> #info ?????? is on call from 2019-06-27 -> 2019-07-04
15:06:18 <smooge> #info ?????? is on call from 2019-07-04 -> 2019-07-11
15:06:22 <smooge> #info Summary of last week: (from smooge )
15:06:41 <smooge> currently on call is 'off' until after the infrastructure f2f
15:07:19 <smooge> this week has been mainly running jobs, dealing with nagios alerts, and trying to triage tickets
15:07:22 <cverna> I ll be online during the f2f so I could take the on call
15:07:25 <bowlofeggs> smooge: i could probably take on call during the f2f
15:07:27 <bowlofeggs> heh
15:07:32 <bowlofeggs> or cverna beat me to it ☺
15:07:36 <cverna> ha ha :)
15:08:03 <cverna> yes it is just that I cannot guarantee to be online all the time :P
15:08:06 <bowlofeggs> cverna could do today to next thursday and i could do the following thurs -> thurs?
15:08:06 <smooge> so I would like you guys to watch and triage tickets but a lot of oncall usually needs root access or a person who will not be around
15:08:18 <smooge> so I would like people to put in the tickets since you will be saying
15:08:18 <bowlofeggs> yeah
15:08:34 <cverna> works for me
15:08:37 <bowlofeggs> yeah we could just have the oncall alias say to file a ticket
15:08:41 <bowlofeggs> i'll watch the tickets
15:08:51 <bowlofeggs> not sure what i'll do if a crazy one comes in
15:09:01 <bowlofeggs> send some international SMS's? ☺
15:09:02 <smooge> {nirik,smooge,pingou,puiterwijk,relrod} is not around this week. please put in or update a ticket and they will see it when they get back
15:09:48 <smooge> I expect that contacting us via work 'chat' may be the only way as we will be laptops closed and off
15:09:49 <cverna> most things can usually wait a fews days anyway :)
15:10:31 <smooge> #topic Monitoring discussion
15:10:31 <smooge> #info https://nagios.fedoraproject.org/nagios
15:10:31 <smooge> #info Go over existing out items and fix
15:11:07 <smooge> ok the arch64 boxes need to be power cycled via the ups. I need to figure out how to do that
15:12:06 <smooge> the busgateway items look like things which have been moved into openshift and no longer talk on the bus the same way
15:12:40 <puiterwijk> Right.
15:12:55 <puiterwijk> I forgot that FAS sends fedmsg's. I'll go fix that one up later
15:13:44 <smooge> i had greenwave look at what was going on for them, and nothing has emitted a message for them to pick up in N days so they aren't emitting messages
15:14:30 * cverna has to step away
15:14:43 <smooge> ok tickets next
15:14:50 <smooge> #topic Tickets discussion
15:14:51 <smooge> #info https://pagure.io/fedora-infrastructure/report/Meetings%20ticket
15:15:30 <smooge> OK we have a growing pile of tickets. I am hoping that after the f2f we can move some of the 'well we want this open in case we ever can prioritize it' into taiga
15:15:40 <smooge> or whereever it is supposed to go
15:16:00 <smooge> but that is a discussion for the f2f and I am getting ahead of myself :)
15:16:15 <smooge> #topic options for release artifact signing for Fedora CoreOS - dustymabe
15:16:25 * dustymabe waves
15:16:38 <puiterwijk> wat. Didn't ... we have a meeting about this like a few months ago?
15:16:49 <dustymabe> smooge: do you want to copy the text from the ticket ?
15:17:01 <dustymabe> err s/ticket/agenda
15:17:34 <smooge> #info We are exploring options for signing our release artifacts for Fedora CoreOS.
15:17:34 <smooge> #info We have an open ticket where we have defined various options and narrowed down
15:17:34 <smooge> #info the options to ones we'd like to explore with Fedora Infrastructure. The ticket
15:17:34 <smooge> #info and all context can be found here: https://github.com/coreos/fedora-coreos-tracker/issues/187
15:17:35 <smooge> #info The TL;DR is that we'd like to deliver an artifact and a detached signature:
15:17:36 <smooge> #info i.e. `fcos.iso` and `fcos.iso.sig`. We'd like to discuss with infra to see what
15:17:38 <smooge> #info the limitations are in the infra for achieving this goal and if we can make it happen.
15:18:06 <dustymabe> puiterwijk: we did have a meeting, where we decided to ditch existing CL signing for Fedora's. this is us circling back on details
15:18:17 * bgilbert waves
15:18:19 <smooge> so thank you for bringing it up, but we will have to really talk about this after the f2f
15:18:21 * bgilbert also works on Fedora CoreOS
15:18:34 <bgilbert> it'll be many artifacts, actually, not just the .iso
15:18:37 <puiterwijk> "Does releng need to receive the entire blob to sign, or can we just send its hash?" -> we need the full blob
15:18:50 <puiterwijk> We don't sign things we can't see and verify
15:19:38 <dustymabe> puiterwijk: thanks for answering that question
15:20:16 <dustymabe> I'll let everyone finish reading/digesting before we move forward with discussion
15:20:26 <dustymabe> just let me know when ready
15:22:43 <dustymabe> anyone ready for me to continue ?
15:22:56 * puiterwijk reads some comments that are just really infuriating because they're false.
15:23:41 <dustymabe> ouch - I assume they weren't intentionally false, maybe you can help us correct them in a constructive way?
15:24:14 <puiterwijk> Basically, the part where people are saying straight up that big artifacts are a big problem. They're not. *many* artifacts are usually the problem.
15:24:49 <dustymabe> ahh - well that's why we're here - to discuss the assumed problems with the people who know
15:24:50 <puiterwijk> So basically, just sending us the full artifact should work
15:25:09 <dustymabe> puiterwijk: i.e. option `1.` should be fine?
15:25:24 <puiterwijk> "I think the pivotal thing really becomes the pain of using the Fedora GPG signer - and it's reasonable to consider that a bug to be fixed, but like I said I doubt it will be." -> I don't think it's such a problem, and if it is, I'll be glad to fix it. Just funny people assume I won't
15:25:33 <bgilbert> how many artifacts is "many"?
15:25:54 <bgilbert> we're looking at N images (on the order of 10), for each of three streams, every two weeks.
15:26:02 <puiterwijk> bgilbert: https://koji.fedoraproject.org/koji/buildinfo?buildID=1271623  - all of the individual subpackages from that
15:26:14 <bgilbert> !!
15:26:19 <bgilbert> it won't be _that_ many :-)
15:26:36 <puiterwijk> That's what I figured :)
15:26:59 <puiterwijk> bgilbert: for another fun one: https://koji.fedoraproject.org/koji/buildinfo?buildID=1206943
15:27:04 <dustymabe> puiterwijk: are there any other concerns with artifacts being large ?
15:27:07 <bgilbert> and each artifact will be a disk image, several GB
15:27:21 <dustymabe> i.e. the signing server is good with them, but any concerns with transfer of files, etc ?
15:27:23 * tflink was expecting texlive as the example there :)
15:27:32 <puiterwijk> tflink:  :)
15:27:41 * dustymabe waves at tflink
15:28:26 <puiterwijk> dustymabe: so, the network traffic is all local gbit, so that's not a big deal. at this moment, we have a limitation in Sigul (0.x, fixed in 1.x which I hope to release RSN) of a 32-bit (unsigned) size field.
15:29:08 <dustymabe> where the size is in bytes?
15:29:11 <puiterwijk> So.. 4294967296 bytes is the limit at this moment, but hopefully it'll be 2^64 in about two weeks or so. So I don't think that's a real issue
15:29:12 <bgilbert> actually, I'll retract the several-GB part
15:29:12 <puiterwijk> Yeah.
15:29:22 <bgilbert> if we're signing the compressed blobs, several hundred MB
15:29:43 <dustymabe> ok cool
15:29:48 <puiterwijk> So right now, the limit would be about 4GB.
15:30:01 <dustymabe> puiterwijk: any other concerns with the proposal as it stands for option `1.` ?
15:30:12 <puiterwijk> That I'm only halfway through the page? :)
15:30:18 <puiterwijk> But no, I don't think so
15:30:33 <dustymabe> :) just worry about the first comment with the listed options :)
15:30:41 <puiterwijk> So, the one requirement we'd want is to get a message sent on the fedora messaging bus when it's composed and up for signing
15:30:51 <puiterwijk> (to trigger the auto-signer)
15:30:55 <dustymabe> puiterwijk: yeah
15:31:02 <dustymabe> so that was going to be my next question
15:31:09 <dustymabe> i.e. how do we wire things up
15:31:24 <puiterwijk> We have the broker on the internet. So as long as you have anywhere you can call fedora-messaging, you can send the message.
15:31:25 <dustymabe> and also, do we need to make any changes to sigul or robosig
15:31:33 <puiterwijk> Yes, we will need at least robosig changes
15:31:44 <puiterwijk> Sigul can handle detached signatures already
15:32:02 <dustymabe> ok - i'll get with you after to get a short description of needed robosig changes
15:32:13 <dustymabe> one more question regarding "wiring things up"
15:32:40 <dustymabe> i assume that content that gets signed needs to be on fedora's NFS share
15:32:54 <puiterwijk> No, we don't need to do that.
15:32:58 <dustymabe> oh nice?
15:33:04 <puiterwijk> I can make robosignatory download it from somewhere else.
15:33:06 <dustymabe> so we can do a direct upload to the signer
15:33:24 <dustymabe> s/upload/download/
15:33:29 <puiterwijk> But in that case, I would like a url and expected checksum in the message
15:33:38 <dustymabe> I think we can do that
15:33:43 <puiterwijk> (so I can verify that what I'm about to sign did not get manipulated in between)
15:33:51 <dustymabe> ok cool - i'll get with you after the meeting and work out some details
15:33:55 <bgilbert> we were planning to stage the artifacts into an S3 bucket without public-read access
15:33:56 <smooge> I would like a project request and plan that we can hand to our managers to approve
15:34:02 <bgilbert> I assume authenticated S3 fetches are off the table?
15:34:14 <puiterwijk> bgilbert: nah, I think that's simple enough to do
15:34:19 <bgilbert> puiterwijk: +1
15:34:22 <dustymabe> note that with some guidance I can help work on this
15:34:35 <puiterwijk> oh, right. We need a project request and everything now :/
15:34:52 <smooge> in any case, this is more than a 40 minute work request and has multiple tie-ins
15:35:03 <dustymabe> puiterwijk: that's fine. i can write something up
15:35:46 <pingou> +1 on document the demand and scope, but we're still working through the processes here, so I don't think we should block on this if we would have done the work w/o it before
15:35:49 <dustymabe> #action dustymabe to get with puiterwijk after the meeting to hammer out some details for signing fcos artifacts
15:36:04 <dustymabe> #action dustymabe to write project request for fcos signing
15:36:11 <pingou> thanks dustymabe :)
15:36:12 <dustymabe> will I open that request against the infra repo ?
15:36:18 <puiterwijk> No clue
15:36:26 <jlanda> Is dusty chaired?
15:36:27 <puiterwijk> For now, we send it to mgmt and pray.
15:36:33 <pingou> dustymabe: put it on the wiki and send it to the infra list for now I'd say
15:36:37 <puiterwijk> jlanda: he isn't.
15:36:40 <jlanda> You're going ti loose all that #actions
15:36:44 <smooge> #action dustymabe to get with puiterwijk after the meeting to hammer out some details for signing fcos artifacts
15:36:49 <smooge> #action dustymabe to write project request for fcos signing
15:36:51 <walters> sorry if I was wrong about the size
15:36:55 <jlanda> I tought :)
15:37:00 <dustymabe> jlanda: it's fine - i was just wanting to make everything clear to the people who were attending
15:37:10 <smooge> dustymabe, open it as a ticket. after the f2f we may have some changes and places you need to add more data to it
15:37:12 <dustymabe> walters: I also had the same mis-conception on size
15:37:21 <pingou> lol, 3 answers, 3 options :D
15:37:27 <jlanda> np, just wanna clarify that some else need to action to go to minutes ;)
15:37:27 <pingou> I guess: take your pick :)
15:37:48 <walters> but I'm still a bit skeptical since this will be the first time we'll be needing signatures *often* on large data
15:38:04 <walters> there are a few large RPM packages but I don't think they're built with any frequency, nor are they really critical path
15:38:26 <dustymabe> walters: bgilbert above mentioned we'll only need signatures when we do a release, so maybe not too "often"
15:38:47 <walters> if we do that it's going to introduce all the development problems that existed when rawhide wasn't signed
15:38:49 <smooge> once every 2 weeks is not too often
15:38:51 <puiterwijk> walters: I use sigul for myself for all of my GPG stuff :). And worst case, we have two sign vaults, and we could look at active/active, which should double capacity
15:38:57 <bowlofeggs> 4 GB is not large data
15:39:17 <bgilbert> smooge, bowlofeggs: cool :-)
15:39:18 <puiterwijk> (right now, it's active/passive)
15:39:49 <dustymabe> walters: I guess I'm not familiar with those exact problems - maybe we can follow up in #fedora-coreos after the meeting?
15:39:56 <smooge> so have we covered all the items?
15:40:18 <walters> packagekit's GPG stuff broke several times because it wasn't tested in a production way until release
15:40:45 <dustymabe> smooge: i think so
15:40:48 <puiterwijk> Well, we have a staging environment where we can test everything with non-production keys
15:40:59 <puiterwijk> So let's make sure to not repeat that one :)
15:41:44 <smooge> ok I am moving to our last item
15:41:47 <bgilbert> thanks puiterwijk, thanks all!
15:41:51 <smooge> #topic Open Floor
15:42:22 <smooge> #info There will be no meeting next week. The next scheduled meeting will be 2019-06-20
15:43:22 <smooge> #info Final Reminder: There is no one on call til 2019-06-20. We will look at tickets when we can but do not expect any 'OMG I need it done today' to happen until then
15:44:09 <smooge> Thank you all for coming. Please tip the wait staff before you leave. They have to deal with me all day while you get to leave
15:44:15 <smooge> #endmeeting