15:34:13 <dgilmore> #startmeeting RELENG (2015-10-05)
15:34:13 <zodbot> Meeting started Mon Oct  5 15:34:13 2015 UTC.  The chair is dgilmore. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:34:13 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:34:24 <dgilmore> #meetingname releng
15:34:24 <zodbot> The meeting name has been set to 'releng'
15:34:25 <dgilmore> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson pingou maxamillion
15:34:25 <zodbot> Current chairs: bochecha dgilmore masta maxamillion nirik pbrobinson pingou sharkcz tyll
15:34:27 <dgilmore> #topic init process
15:34:35 <tyll> Hi there
15:34:37 <maxamillion> .hello maxamillion
15:34:38 <zodbot> maxamillion: maxamillion 'Adam Miller' <maxamillion@gmail.com>
15:34:43 <nirik> morning
15:36:35 <dgilmore> #topic #6262 drop rawhide-stable tag and consider master branch to be always stable
15:36:43 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6262
15:36:52 <dgilmore> so maxamillion started a thread on the releng list
15:36:56 * masta looks in
15:37:26 <dgilmore> I am all for changing up things, but we need to set clear guildelines and expectations on how we are going to work
15:37:41 <dgilmore> I do not think that just removing the tagging is sufficient
15:38:11 <maxamillion> agreed
15:38:15 <nirik> I'm fine with the git flow maxamillion suggested...
15:38:41 <tyll> do we actually have a setup to do with commits that are not yet in the master branch?
15:39:18 <tyll> afaics nowadays all changes go to forks and are then merged into the master branch after a review - and then the tag is reset to match master
15:39:19 <dgilmore> tyll: not sure what you mean
15:39:39 <nirik> we don't have a devel/develop branch yet... but could make one
15:40:06 * nirik wonders if we shouldn't wait and do this after f23 is out the door.
15:40:11 <dgilmore> tyll: mostly that is nhow it works, but at least one person last week was mad at me for the change to pagure and did not want to make a fork and send in a pull request
15:40:15 <tyll> but what would be do with the devel branch? afaik we do not have a staging setup to run it first
15:40:20 <dgilmore> nirik: I think we should wait
15:40:30 <maxamillion> nirik: +1 - I'd rather not rock the boat until post f23
15:40:38 <dgilmore> and we need to do better about documenting and communicating the workflow
15:40:58 <dgilmore> set clear expectations for all
15:41:56 <nirik> it may be we could get a setup in staging that pulls from develop and does composes...
15:42:08 <nirik> would take some more work, but might be possible.
15:42:30 <dgilmore> nirik: indeed.
15:42:41 <tyll> dgilmore: but I guess that person did not directly commit to our repo but sent a patch, therefore there is already a review - but afaics there is now further review/testing after a commit to master and then re-setting the tag
15:43:00 <dgilmore> tyll: no
15:43:13 <dgilmore> they had commit access to the fedorahosted repo
15:43:20 <dgilmore> and in the past had commited and pushed
15:43:32 <dgilmore> and were upset that they could not just do that
15:43:52 <dgilmore> I am sure it is entirely about bad communication
15:43:59 <dgilmore> they wanted to add a new script
15:44:10 <dgilmore> which I think in the end is not public anywhere
15:44:23 <nirik> ... but should be. ;)
15:44:27 <dgilmore> because they thought it was too much overhead
15:44:33 <dgilmore> nirik: indeed it should be
15:46:05 <dgilmore> anyway I Think at this point we need better communication and will wait until F23 is out the door to make changes
15:46:10 * nirik nods.
15:46:23 <dgilmore> #action dgilmore to communicate workflows better
15:46:38 <dgilmore> #info no changes to be made until after f23 is final
15:47:08 <dgilmore> #topic #6244 package blocking not working in koji for EPEL
15:47:16 <dgilmore> https://fedorahosted.org/rel-eng/ticket/6244
15:47:33 <dgilmore> I believe this is all working
15:47:59 <nirik> yeah,no errors from it.
15:48:02 <tyll> I am not sure if we had any retirement in EPEL recently
15:48:11 <nirik> there was one.
15:48:13 * nirik looks
15:49:10 <nirik> pidgin... and it looks right
15:49:20 <tyll> on autosign it does not work right now
15:49:58 <tyll> is the system were it runs in the cron job not a RHEL 7 system?
15:50:29 <nirik> yes, should be rhel7
15:50:58 <nirik> oh wait...
15:51:01 * nirik digs up errors
15:51:41 <nirik> http://paste.fedoraproject.org/274963/60289144/
15:52:14 <tyll> yes, this is what I am seeing as well
15:52:55 <maxamillion> did koji get upgraded recently?
15:53:03 <tyll> iirc dgilmore planned to prepare a new koji package - is it already available?
15:53:09 <dgilmore> not recently
15:53:13 <nirik> it did
15:53:20 <nirik> kalev updated it.
15:53:21 <dgilmore> I did not do it though
15:53:42 * nirik checks versions
15:53:43 <tyll> at least from the upstream commit msgs it should fix the bug, unless there are also other bugs
15:53:54 <dgilmore> there is one other patch that I think is needed, I am getting itreviewed by others and was planning to update with it
15:54:04 <nirik> ok, thats only in testing
15:54:46 <tyll> after re-reading my comment - it seems that the patch from https://lists.fedoraproject.org/pipermail/buildsys/2015-January/004484.html is also required, not sure if it ever landed upstream
15:55:12 <dgilmore> tyll: that is the one I am getting reviewed by others
15:55:27 <tyll> dgilmore: perfect
15:55:51 <kalev> yup, koji-1.10.0-2.el7 in testing should fix the blocking error as I understand it
15:56:17 <dgilmore> I was planning to do an update with all the patches but kalev went and did an update without my okay
15:56:53 <kalev> ...
15:57:06 <dgilmore> kalev: I told you to wait
15:57:19 <dgilmore> you didn't
15:57:25 <dgilmore> but it is done now
15:57:59 <kalev> anyway, blocking is broken with -1 that's in stable; -2 in testing should fix it
15:58:20 <kalev> might be worth updating the maching that runs the cron job and see if it makes it work
15:59:07 <kalev> I backported the fixes that had gone upstream and those are all in -2
15:59:33 <kalev> but I am sure the area needs more fixes as dgilmore is saying, but -2 should already be an improvement
16:00:16 <dgilmore> anyway
16:00:54 <dgilmore> #info there was a ca cert missing from compose box, blocking appears to be working now
16:01:11 <dgilmore> #topic Secondary Architectures updates
16:01:16 <nirik> right, this will get fixed soon either way
16:01:22 <dgilmore> any generic secondary arch topics?
16:01:41 <nirik> whats the status of the s390 koji? tyll was seeing some issue with apache...
16:01:56 <dgilmore> I am hoping to work on getting arm, the ppc moved to ansible in the next few weeks
16:02:32 <tyll> the s390 koji apache config seems to be changed between broken and working for singing, currently it is broken
16:02:33 <nirik> I was thinking of making arm hub/db later this week... before freeze and we can test and move after release?
16:02:44 <dgilmore> we may need to setup some vhosts on it in apache, one for s390.koji.fedoraproject.org and one for s390pkgs.fedoraproject.org
16:03:12 <dgilmore> other option is to setup a squid box we can use for all secondaries for <arch>pkgs.fp.o
16:03:31 <dgilmore> nirik: that would be good
16:03:47 <dgilmore> nirik: we want to setup a shared box for running koji-shadow on also
16:04:00 <nirik> ok.
16:04:15 <dgilmore> with one box for all secondaries
16:05:06 <dgilmore> #topic Secondary Architectures update - ppc
16:05:19 <dgilmore> not sure we have pbrobinson or sharkcz around
16:05:52 <dgilmore> #topic Secondary Architectures update - s390
16:05:52 <maxamillion> womp womp
16:05:59 <dgilmore> #topic Secondary Architectures update - arm
16:06:04 <dgilmore> hrrm
16:06:11 <dgilmore> anyway
16:06:18 <dgilmore> #topic open floor
16:06:26 <dgilmore> does anyone hav anything?
16:06:31 <tyll> just wondering, shouldn't the pkgs hosts be named  $arch.kojipkgs.fedoraproject.org to match the  $arch.koji.fpo naming scheme?
16:07:09 <nirik> Did we come to any conclusion on https://bugzilla.redhat.com/show_bug.cgi?id=1246701 ? (including gpg keys for previous releases in fedora-repos)?
16:07:10 <dgilmore> tyll: no
16:07:31 <dgilmore> tyll: well I guess we could
16:07:53 <dgilmore> but we chose <arch>pkgs.fp.o years ago
16:08:14 <dgilmore> nirik: not going to do it
16:08:27 <dgilmore> nirik: well I do not think we should
16:08:38 <dgilmore> I think it makes things needlesly messy
16:08:48 <nirik> right, we already will be including the non eol ones right? for upgrades?
16:09:26 <dgilmore> f21 has f22 and f23 keys
16:09:37 <dgilmore> f22 will have f23 and f24
16:09:48 <dgilmore> f23 will have f24 and f25
16:09:49 <tyll> dgilmore: hm, if it is too much trouble, it might not be worth it - not sure, if the $archpkgs hostnames are used in scripts, it just might make things complicated if a script works both for primary and the secondaries
16:09:55 <dgilmore> each in addition to their own
16:10:32 <dgilmore> tyll: not sure I buy that argument.
16:10:46 <dgilmore> they have to do something different for primary to secondary
16:11:05 <dgilmore> and what they do for secondary is consistent across all secondaries
16:11:23 <nirik> dgilmore: sure, then I guess close the bug with that we will have non eol ones and thats the way it goes I guess. ;)
16:12:22 <dgilmore> nirik: I just thik we should remove EOL key which means extra updates,  then confusion for users that something that used to work on a os release no longer works
16:12:30 <tyll> dgilmore: the difference between primary and secondary koji is to prepend $arch. to it, for the pkgs it is to replace pkgs by $arch pkgs, there they are two different operations - but I did not have any need to access the secondary pkgs hosts, so not sure, if it is really an issue
16:12:38 <dgilmore> tyll: no
16:13:21 <tyll> dgilmore: sorry, I meant replace kojipkgs with ${arch}pkgs
16:13:28 <dgilmore> tyll: $arch.koji.fp.o vs koji.fp.o / $archpkgs.fp.o vs kojipkgs.fp.o
16:13:54 <dgilmore> tyll: and in the future world I am trying to get to all the secondary arch kojis go away entirely
16:14:24 <dgilmore> I do not see it as any different
16:14:29 <tyll> dgilmore: this would be even better :-)
16:14:47 <dgilmore> I just do not see enough of a reason to go changing it
16:15:05 <dgilmore> I do not see any benefit to doing so.  but maybe I am missing something
16:15:46 <tyll> I am also fine with keeping it, now that I know about it :-)
16:16:32 <dgilmore> nirik: I will close the bug
16:16:34 <tyll> btw. I would think it would be good to include older GPG keys in fedora-repos
16:16:53 <dgilmore> tyll: for what reason?
16:17:05 <dgilmore> and by older what do you mean?
16:17:15 <nirik> I think we should keep what we do now, or all of them... not churn.
16:17:19 <tyll> dgilmore: to make it possible to create secure chroots as described in the bug reports
16:17:31 <nirik> but eol ones contributes to people installing eol stuff, which I think is bad
16:17:52 <dgilmore> tyll: I think we need to do everything we can to not enable people to run eol code
16:18:11 <dgilmore> and I think that shipping older ones would help with that
16:18:13 <tyll> but how would we enable people doing so with this?
16:18:31 <tyll> afaik most systems already contain all the GPG keys that they ever used in the RPM database
16:18:43 <dgilmore> I also think that people should use mock and not dnf to make chroots
16:18:44 <tyll> at least I do not know anyone else who is removing them after an upgrade
16:19:18 <nirik> tyll: good point. /me thinks of filing a RFE for the dnf upgrade plugin. ;)
16:20:12 <dgilmore> lets step back a sec
16:20:16 * nirik runs to get more coffe.
16:20:27 <dgilmore> way way back in the past. before the incident
16:20:32 <dgilmore> we had 2 keys
16:20:47 <dgilmore> one for testing builds and one for stable builds
16:21:23 <tyll> also if people want to install something, they will do it, e.g. if a guide says to run "dnf install --releasever $eolversion $package" it will just say "dnf install --releasever $eolversion --nogpg $package" and people who want to install eol packages will do it
16:21:36 <dgilmore> we would sign all updates for all releases going to testing with one key,  then resign them all when they went stable with the stable key
16:22:09 <dgilmore> after the incident we moved to making a new key for every release
16:22:55 <dgilmore> the theory being that the key is short lived and if need be it can be replaced, or we can just say f it and still sign with the comprimised key because its not going to be used soon
16:23:13 <dgilmore> not sure that the second point of view is necessarily great
16:23:19 <nirik> tyll: true.
16:23:43 <nirik> I guess if we really didn't want people to use eol releases we would remove them all (after 3 years or whatever for the gpl)
16:24:09 <dgilmore> perhaps we want to move to a different way of doing key management
16:25:30 <nirik> I think the current setup is pretty ok... but happy to hear a better one.
16:25:38 <maxamillion> I actually really like how docker notary does signing and key management but I'm not sure how applicable that is at all and if it is, it would require a lot of work https://github.com/docker/notary
16:26:15 <nirik> I guess I am fine with either just doing what we are doing now in fedora-repos, or just including all the old keys (well, perhaps not the bad f9 one or whatever)
16:27:03 <dgilmore> the key files are not big. but I bet that the clud WG would bitch about it
16:27:17 <nirik> they are trivial sized. ;)
16:27:21 <dgilmore> maybe make a fedora-keys-old subpackage with the older keys
16:27:38 <dgilmore> something people could install but is not there by default
16:27:50 <nirik> that seems like more work than it's worth...
16:27:57 <dgilmore> perhaps
16:27:57 <tyll> switching to new keys regularly also allows to adjust to new crypto algorithms btw - which is not needed every 6 months, but at least sometimes
16:28:07 <dgilmore> f21 would always have f20 and older in it
16:28:26 <dgilmore> f22 f21 and older
16:28:36 <tyll> and the difference between the rawhide and stable signing key allows us to make rawhide signing a little less secure with the autosigner without reducing the protection of the stable signing key
16:28:40 <nirik> if we just always include them all there wouldn't be more churn... just keep them around and add new ones.
16:29:09 <dgilmore> nirik: there would not be more churn with the subpackage idea
16:29:16 <nirik> true.
16:29:29 <nirik> if you want to do that, thats cool with me too... I just think it's more work.
16:30:08 <dgilmore> the 21 22 and 23 keys are all under 4K each
16:30:25 <nirik> yeah, it's noise. I can't see how anyone would care about the size.
16:30:35 <dgilmore> the cloud WG will
16:30:42 <maxamillion> heh
16:31:05 <nirik> that's pretty boggiling. If they do, I'd be happy to tell them how crazy they are.
16:31:24 <dgilmore> nirik: :) they have tried to get smaller things out
16:31:44 <nirik> or better: we add them all to fedora-repos, if they complain, require them to submit the PR to split it out and deal with that work. ;)
16:32:10 <tyll> we could also make two alternate pkgs, one with only the current key and one with all - cloud could use the minimal pkg and everyone else the normal pkg
16:32:28 <nirik> thats even _more_ work. ;)
16:32:31 <tyll> then it would be even an improvement for the cloud
16:32:33 <dgilmore> tyll: I think that would be a lot more work
16:33:07 <tyll> dgilmore: it is just like adding a subpackage but only with the current key afaics
16:33:23 <kalev> what happens if they are in the main package? does rpm start trusting old keys out of the box then?
16:33:42 <tyll> and both need a common Provides off course
16:33:57 <tyll> kalev: rpm only trusts after importing them
16:34:21 <dgilmore> kalev: they have to be imported into the rpmdb
16:34:31 <dgilmore> kalev: but once imported they will be trusted
16:34:54 <kalev> that's not so bad then :)
16:34:57 <dgilmore> kalev: someone could put in %post or %pre to import it
16:35:04 <dgilmore> would be ugly as but could be done
16:35:18 <nirik> but they could do that now also. ;)
16:35:39 <tyll> afaik there is also a second storage location for GPG keys that is file based, maybe a GPG keyring file somewhere in /var/lib/rpm or so
16:35:41 <nirik> the dnf chroot thing would ask them if they wanted to install it I think
16:37:02 <nirik> so, we want to just add them all now? or
16:37:14 * nirik is sorry for a lengthy topic, thought it might be quick. ;)
16:38:06 <dgilmore> nirik: :) it is okay
16:38:26 <nirik> I can try and submit a PR on it if no one else wants to
16:38:57 <dgilmore> nirik: if you want. the archmap file will need updating quite a bit
16:39:09 <nirik> I can try... or someone else can, I don't mind. ;)
16:39:10 <tyll> I guess I would prefer just all keys that are not eol at the time the pkgs needs to be touched anyhow
16:39:11 <dgilmore> and will need to add all the keys
16:39:34 <nirik> tyll: right, we wouldn't go back and re-update eol stuff...
16:39:34 <dgilmore> tyll: removing keys will catch people off guard
16:39:55 <nirik> so f23 would have f24 and older, f24 would have f25 and older, etc.
16:40:14 <dgilmore> fedora-repos does exist for f21 up
16:40:26 <tyll> dgilmore: ok, let me rephrase, I only thought about the Rawhide pkgs and forgot the later updates to stable pkgs
16:40:30 <dgilmore> nirik: not quite right
16:40:40 <nirik> I guess we need +1 more.
16:40:42 <dgilmore> nirik: f23 will have f24 and f25
16:40:44 <nirik> when it exists.
16:40:45 <nirik> yeah
16:40:55 <dgilmore> but not newer than f25
16:41:10 <tyll> so make the Rawhide pkgs contain only non-eol pkgs and only add newer keys to older non-eol release compared to Rawhide
16:41:24 <nirik> right.
16:43:24 <nirik> anything else on this? oh, shall I update the bug?
16:43:32 <dgilmore> nirik: if you want to send in a PR i will gladly review
16:43:45 <nirik> can try :)
16:44:55 <dgilmore> #action nirik to send in PR for fedora-repos to include older keys
16:45:00 <dgilmore> anything else?
16:46:07 <tyll> I do not have anything more
16:46:59 <dgilmore> #endmeeting