14:30:01 <nirik> #startmeeting RELENG (2014-11-10)
14:30:02 <zodbot> Meeting started Mon Nov 10 14:30:01 2014 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:30:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
14:30:02 <nirik> #meetingname releng
14:30:02 <zodbot> The meeting name has been set to 'releng'
14:30:02 <nirik> #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson
14:30:02 <nirik> #topic init process
14:30:02 <zodbot> Current chairs: bochecha dgilmore masta nirik pbrobinson sharkcz tyll
14:30:10 <nirik> who all is around for a releng meeting?
14:30:15 <bochecha> I am
14:30:42 * pbabinca here
14:32:44 * sharkcz is here
14:33:06 * nirik will wait another minute or two for folks to wander in
14:35:13 <nirik> ok, I guess lets go ahead...
14:35:15 <nirik> #topic Secondary Architectures update - s390
14:35:23 <nirik> sharkcz: any news to report?
14:36:24 <sharkcz> not much, I created a compose a week ago, roughly like beta in primary, works fine, but there is a problem with writing to console in 3.17 kernel, bug is reported and IBM is looking on it
14:36:52 <nirik> ah, ok.
14:37:01 <nirik> so no official beta until thats squashed?
14:37:07 <sharkcz> bug #1158848 for the record, network login works
14:37:20 <sharkcz> yes
14:38:00 <nirik> ok, makes sense.
14:38:19 <nirik> no pbrobinson or masta, I guess we skip ppc and arm for now.
14:38:26 <nirik> #topic Tickets
14:38:36 <sharkcz> I can comment ppc a bit
14:38:41 <nirik> ok, coo.
14:38:47 <nirik> #topic Secondary Architectures update - ppc
14:39:32 <sharkcz> there is beta rc1, but is blocked by 2 issues - cryptsetup fails on arches with page size >=64k (ppc* and aarch64), fix exists upstream so I need to ask jwb to include it
14:40:18 <nirik> ok.
14:40:47 <sharkcz> another issue is multipath install, seems to work on x86 in minimal setup (kvm + 1 multipathed scsi disk), we are reproducing this setup on ppc and will see later today hopefully
14:41:35 <nirik> alright, sounds good.
14:41:42 <nirik> so hopefully soon for a beta
14:41:53 <sharkcz> yep
14:42:07 <nirik> excellent. thanks.
14:42:14 <nirik> #topic Tickets
14:42:23 <nirik> https://fedorahosted.org/rel-eng/report/10?sort=created&asc=1&page=1
14:42:30 <nirik> we have 14 tickets open with meeting...
14:42:39 <nirik> not sure which we can/should go over today
14:42:51 <nirik> (with many folks not here)
14:43:48 <nirik> sharkcz: did https://fedorahosted.org/rel-eng/ticket/6014 get a final list? or do we even have a way to find out?
14:45:01 <sharkcz> I don't have the list at hand, but should be doable by a direct query to koji db
14:45:25 <sharkcz> question is whether it is worth the effort
14:45:39 <nirik> yeah, no idea on that part. perhaps it's just history now.
14:46:11 <sharkcz> yes, I'd close the ticket, very likely there are new builds for the affected packages
14:46:24 <nirik> ok.
14:46:46 <nirik> also https://fedorahosted.org/rel-eng/ticket/6024 can probibly be closed, we discussed it last week and I think pingou already implemented what we talked about.
14:48:55 <nirik> ok, those closed. ;) then there were 12. ;)
14:50:06 <bochecha> 6039 seems like something we could solve quickly?
14:50:42 <nirik> bochecha: off hand I would say we could publish dumps, but I am not fully sure there's no sensitive info in there.
14:50:49 <nirik> I would want to check with dgilmore on that...
14:51:47 <nirik> we have a framework already in place for sharing db's... but not sure koji would be ok to share that way.
14:52:09 <sharkcz> but won't be the dumps too large? I have 7GB+ on s390, the dump, compressed
14:52:39 <sharkcz> and primary has much more info in it I guess
14:53:12 <nirik> huh... primary ones are only 1.3gbish
14:53:14 <bochecha> looking at the db for another koji, I can't think of anything that could be sensitive
14:53:25 <nirik> -rw-r--r--. 1 postgres postgres 1.3G Nov 10 04:55 koji-2014-11-10.dump.xz
14:53:48 <sharkcz> let me check it, I hope I read the scale correctly :-)
14:54:18 <sharkcz> ah, it's 700M ;-)
14:54:24 <nirik> cool.
14:55:13 <nirik> bochecha: I don't know if we setup initial admins when installing with passwords instead of certs? otherwise, yeah, not sure what could be sensitive.
14:55:30 <bochecha> nirik, ah, that maybe
14:56:11 <nirik> anyhow I can check with dgilmore and see what he thinks... it would be pretty trivial to add it to the db's we already make available.
14:56:40 <nirik> http://infrastructure.fedoraproject.org/infra/db-dumps/ (btw)
14:57:34 <nirik> any other tickets we think we can make progress on? or should we call it a short meeting?
14:58:13 <bochecha> about 6016, didn't we say we'd like some data about savings?
14:58:40 <sharkcz> #6023 - should we express our opinions the ticket?
14:59:19 <nirik> bochecha: yeah, you want to add a comment about that there?
14:59:39 <bochecha> pbabinca, did you have any time to look at savings from using fedpkg-minimal? :)
14:59:59 <pbabinca> bochecha, I haven't made any progress there.
15:00:00 <nirik> sharkcz: sure I guess. Again, I would want to check with dgilmore before adding any folks there... it's pretty sensitive...
15:00:12 <nirik> but we should/could add some more folks, especially in .eu
15:03:26 <nirik> #topic Open Floor
15:03:31 <nirik> anyone have items for open floor?
15:04:18 <tyll> can we decide something about https://fedorahosted.org/rel-eng/ticket/6040 - e.g. can an existing systems be used for it?
15:05:09 <nirik> tyll: possibly releng04... but it's rhel6... does that present problems?
15:05:51 <tyll> nirik: uh, if I can login there, I can check - I only ran it on Fedora and RHEL7
15:06:23 <nirik> tyll: I can get you access to it if you don't have it already.
15:06:45 <nirik> thats the machine fedora updates pushes happen on, so it sends already bodhi emails and such.
15:06:52 <tyll> yes, I can access it
15:06:58 <nirik> or we could just make a new releng-backend or something.
15:07:26 <tyll> Can I add an ansible task for the cronjob for it or does it need to go into puppet?
15:08:33 <nirik> sadly, it's in puppet.
15:09:06 <nirik> let me think on it a bit more perhaps...
15:09:07 <tyll> the script does not work out-of-the box on it :-(
15:09:23 * nirik only saw the request this morning before coffee... and have had no time to ponder on it.
15:09:30 <tyll> ok
15:10:02 <nirik> ok, we can make a releng-backend01 perhaps. that makes more sense off hand anyhow.
15:12:09 <nirik> tyll: oh, where are we on rawhide autosigner?
15:13:51 <tyll> I just started looking into it again, it seems that secondary sigul is more stable than a few months ago (or it is because I am monitoring it now)
15:14:44 <nirik> I saw your report of an outage on it... no idea on that. I was asleep... nothing should have been changing on it.
15:14:59 <tyll> It seems to work quite good (still without rawhide gating), except that arm koji shows some issues sometimes
15:15:15 <tyll> ah, there is also something I need to investigate
15:15:37 <tyll> sometimes builds are not properly tagged on secondary kojis, even though a tagging fedmsg was sent earlier
15:15:42 <tyll> not sure if the packages are untagged again
15:16:15 <tyll> or if there is a race condition that makes koji send the fedmsg message before tagging was finished so a query for it fails
15:16:26 <nirik> ah...dunno.
15:16:47 <nirik> one side effect of the rawhide signing is we get the weird metadata problems with pungi making boot.img
15:17:01 <nirik> I think we should put a 'yum clean metadata' in there to work around it.
15:17:26 <tyll> does it fail if RPM checksums fail?
15:17:30 <tyll> checksums change?
15:17:38 <nirik> yeah...
15:18:04 <tyll> I see, this should not happen anymore when there is some gating in koji
15:18:21 <nirik> Error downloading packages:
15:18:21 <nirik> Delta RPM of 1:openssl-libs-1.0.1j-1.fc22.x86_64: Checksum of the delta-rebuilt RPM failed
15:18:35 <nirik> also, probibly it shouldn't be using deltas. ;)
15:18:42 <tyll> hehe
15:19:02 <nirik> anyhow, will see about adding a clean metadata in there.
15:19:32 <tyll> btw. would the releng cron mailing list be the right target to let autosigner report errors via e-mail?
15:20:21 <nirik> Oh, one other tidbit: I got all the builders re-isntalled/updated. they are all runinning 3.17.2-200.fc20 now (aside the ppc and bkernel ones)
15:20:30 <nirik> tyll: sounds good to me. +1
15:21:18 <nirik> ok, if nothing else will close out in a minute.
15:21:35 <tyll> btw. I also added scripts to releng-git a while ago to check whether packages are not signed when mashing them
15:21:50 <tyll> and one script to monitor sigul by checking whether login works every 10 minutes
15:22:01 <tyll> is the last one something that could be added to nagios?
15:22:05 <nirik> mash can check/reject them no?
15:22:10 <nirik> yes, definitely.
15:22:21 <nirik> if it was, then we could more quickly restart them if they got stuck.
15:22:24 <tyll> it looks into the mash logs, because there are only awrnings currently
15:22:53 <nirik> mash has a config option to treat that as a warn or a fail
15:22:58 <tyll> for the check script it would make sense to create a test key that is not used for anything to make it not that critical to use the credentials for it
15:23:12 <tyll> for Rawhide and secondary Branched it is a warning currently
15:23:39 <tyll> the scripts looks into the logs for the warning (I use it to see if autosigner signs all builds)
15:23:44 <nirik> yeah, for bodhi/updates it's strict fail
15:24:02 <nirik> we could make branched that way now too...
15:24:13 <nirik> yeah, test key sounds good.
15:24:17 <tyll> I believe it is for primary Branched
15:24:28 <tyll> ah, this reminds me of one more thing
15:24:56 <tyll> buildbranched and buildrawhide should be merged, but there is at least one diff where I am not sure whether it is intended that they differ
15:25:11 <nirik> yeah, we have wanted to merge them for a long time. ;)
15:25:28 <nirik> whats the diff?
15:25:30 <tyll> I synced most of them
15:25:49 <tyll> http://paste.fedoraproject.org/149368/56331421
15:26:01 <tyll> there is a call for wait
15:26:03 * masta wonders in late
15:28:24 <nirik> huh, no idea why thats there?
15:28:31 <nirik> oh, unless it needs all arches to finish?
15:28:39 <nirik> no, no idea
15:29:56 <tyll> so can it just be synced to buildrawhide?
15:30:34 <nirik> as far as I can tell.
15:30:45 <nirik> there's actually nothing being backgrounded there is there?
15:31:41 <tyll> pungify is
15:31:59 <tyll> and spam-o-matic
15:32:19 <nirik> ok, wonder why rawhide doesn't wait for those and branched does?
15:32:44 <nirik> I would think both should
15:33:23 <tyll> uh, actually both wait before pungify
15:33:47 <nirik> ok... lets continue this after meeting?
15:33:53 <tyll> but rawhide has an extra wait (So I confused both)
15:33:59 <tyll> ok
15:34:12 <nirik> thanks for coming everyone!
15:34:14 <nirik> #endmeeting