14:30:01 #startmeeting RELENG (2014-11-10) 14:30:02 Meeting started Mon Nov 10 14:30:01 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:30:02 Useful Commands: #action #agreed #halp #info #idea #link #topic. 14:30:02 #meetingname releng 14:30:02 The meeting name has been set to 'releng' 14:30:02 #chair dgilmore nirik tyll sharkcz bochecha masta pbrobinson 14:30:02 #topic init process 14:30:02 Current chairs: bochecha dgilmore masta nirik pbrobinson sharkcz tyll 14:30:10 who all is around for a releng meeting? 14:30:15 I am 14:30:42 * pbabinca here 14:32:44 * sharkcz is here 14:33:06 * nirik will wait another minute or two for folks to wander in 14:35:13 ok, I guess lets go ahead... 14:35:15 #topic Secondary Architectures update - s390 14:35:23 sharkcz: any news to report? 14:36:24 not much, I created a compose a week ago, roughly like beta in primary, works fine, but there is a problem with writing to console in 3.17 kernel, bug is reported and IBM is looking on it 14:36:52 ah, ok. 14:37:01 so no official beta until thats squashed? 14:37:07 bug #1158848 for the record, network login works 14:37:20 yes 14:38:00 ok, makes sense. 14:38:19 no pbrobinson or masta, I guess we skip ppc and arm for now. 14:38:26 #topic Tickets 14:38:36 I can comment ppc a bit 14:38:41 ok, coo. 14:38:47 #topic Secondary Architectures update - ppc 14:39:32 there is beta rc1, but is blocked by 2 issues - cryptsetup fails on arches with page size >=64k (ppc* and aarch64), fix exists upstream so I need to ask jwb to include it 14:40:18 ok. 14:40:47 another issue is multipath install, seems to work on x86 in minimal setup (kvm + 1 multipathed scsi disk), we are reproducing this setup on ppc and will see later today hopefully 14:41:35 alright, sounds good. 14:41:42 so hopefully soon for a beta 14:41:53 yep 14:42:07 excellent. thanks. 14:42:14 #topic Tickets 14:42:23 https://fedorahosted.org/rel-eng/report/10?sort=created&asc=1&page=1 14:42:30 we have 14 tickets open with meeting... 14:42:39 not sure which we can/should go over today 14:42:51 (with many folks not here) 14:43:48 sharkcz: did https://fedorahosted.org/rel-eng/ticket/6014 get a final list? or do we even have a way to find out? 14:45:01 I don't have the list at hand, but should be doable by a direct query to koji db 14:45:25 question is whether it is worth the effort 14:45:39 yeah, no idea on that part. perhaps it's just history now. 14:46:11 yes, I'd close the ticket, very likely there are new builds for the affected packages 14:46:24 ok. 14:46:46 also https://fedorahosted.org/rel-eng/ticket/6024 can probibly be closed, we discussed it last week and I think pingou already implemented what we talked about. 14:48:55 ok, those closed. ;) then there were 12. ;) 14:50:06 6039 seems like something we could solve quickly? 14:50:42 bochecha: off hand I would say we could publish dumps, but I am not fully sure there's no sensitive info in there. 14:50:49 I would want to check with dgilmore on that... 14:51:47 we have a framework already in place for sharing db's... but not sure koji would be ok to share that way. 14:52:09 but won't be the dumps too large? I have 7GB+ on s390, the dump, compressed 14:52:39 and primary has much more info in it I guess 14:53:12 huh... primary ones are only 1.3gbish 14:53:14 looking at the db for another koji, I can't think of anything that could be sensitive 14:53:25 -rw-r--r--. 1 postgres postgres 1.3G Nov 10 04:55 koji-2014-11-10.dump.xz 14:53:48 let me check it, I hope I read the scale correctly :-) 14:54:18 ah, it's 700M ;-) 14:54:24 cool. 14:55:13 bochecha: I don't know if we setup initial admins when installing with passwords instead of certs? otherwise, yeah, not sure what could be sensitive. 14:55:30 nirik, ah, that maybe 14:56:11 anyhow I can check with dgilmore and see what he thinks... it would be pretty trivial to add it to the db's we already make available. 14:56:40 http://infrastructure.fedoraproject.org/infra/db-dumps/ (btw) 14:57:34 any other tickets we think we can make progress on? or should we call it a short meeting? 14:58:13 about 6016, didn't we say we'd like some data about savings? 14:58:40 #6023 - should we express our opinions the ticket? 14:59:19 bochecha: yeah, you want to add a comment about that there? 14:59:39 pbabinca, did you have any time to look at savings from using fedpkg-minimal? :) 14:59:59 bochecha, I haven't made any progress there. 15:00:00 sharkcz: sure I guess. Again, I would want to check with dgilmore before adding any folks there... it's pretty sensitive... 15:00:12 but we should/could add some more folks, especially in .eu 15:03:26 #topic Open Floor 15:03:31 anyone have items for open floor? 15:04:18 can we decide something about https://fedorahosted.org/rel-eng/ticket/6040 - e.g. can an existing systems be used for it? 15:05:09 tyll: possibly releng04... but it's rhel6... does that present problems? 15:05:51 nirik: uh, if I can login there, I can check - I only ran it on Fedora and RHEL7 15:06:23 tyll: I can get you access to it if you don't have it already. 15:06:45 thats the machine fedora updates pushes happen on, so it sends already bodhi emails and such. 15:06:52 yes, I can access it 15:06:58 or we could just make a new releng-backend or something. 15:07:26 Can I add an ansible task for the cronjob for it or does it need to go into puppet? 15:08:33 sadly, it's in puppet. 15:09:06 let me think on it a bit more perhaps... 15:09:07 the script does not work out-of-the box on it :-( 15:09:23 * nirik only saw the request this morning before coffee... and have had no time to ponder on it. 15:09:30 ok 15:10:02 ok, we can make a releng-backend01 perhaps. that makes more sense off hand anyhow. 15:12:09 tyll: oh, where are we on rawhide autosigner? 15:13:51 I just started looking into it again, it seems that secondary sigul is more stable than a few months ago (or it is because I am monitoring it now) 15:14:44 I saw your report of an outage on it... no idea on that. I was asleep... nothing should have been changing on it. 15:14:59 It seems to work quite good (still without rawhide gating), except that arm koji shows some issues sometimes 15:15:15 ah, there is also something I need to investigate 15:15:37 sometimes builds are not properly tagged on secondary kojis, even though a tagging fedmsg was sent earlier 15:15:42 not sure if the packages are untagged again 15:16:15 or if there is a race condition that makes koji send the fedmsg message before tagging was finished so a query for it fails 15:16:26 ah...dunno. 15:16:47 one side effect of the rawhide signing is we get the weird metadata problems with pungi making boot.img 15:17:01 I think we should put a 'yum clean metadata' in there to work around it. 15:17:26 does it fail if RPM checksums fail? 15:17:30 checksums change? 15:17:38 yeah... 15:18:04 I see, this should not happen anymore when there is some gating in koji 15:18:21 Error downloading packages: 15:18:21 Delta RPM of 1:openssl-libs-1.0.1j-1.fc22.x86_64: Checksum of the delta-rebuilt RPM failed 15:18:35 also, probibly it shouldn't be using deltas. ;) 15:18:42 hehe 15:19:02 anyhow, will see about adding a clean metadata in there. 15:19:32 btw. would the releng cron mailing list be the right target to let autosigner report errors via e-mail? 15:20:21 Oh, one other tidbit: I got all the builders re-isntalled/updated. they are all runinning 3.17.2-200.fc20 now (aside the ppc and bkernel ones) 15:20:30 tyll: sounds good to me. +1 15:21:18 ok, if nothing else will close out in a minute. 15:21:35 btw. I also added scripts to releng-git a while ago to check whether packages are not signed when mashing them 15:21:50 and one script to monitor sigul by checking whether login works every 10 minutes 15:22:01 is the last one something that could be added to nagios? 15:22:05 mash can check/reject them no? 15:22:10 yes, definitely. 15:22:21 if it was, then we could more quickly restart them if they got stuck. 15:22:24 it looks into the mash logs, because there are only awrnings currently 15:22:53 mash has a config option to treat that as a warn or a fail 15:22:58 for the check script it would make sense to create a test key that is not used for anything to make it not that critical to use the credentials for it 15:23:12 for Rawhide and secondary Branched it is a warning currently 15:23:39 the scripts looks into the logs for the warning (I use it to see if autosigner signs all builds) 15:23:44 yeah, for bodhi/updates it's strict fail 15:24:02 we could make branched that way now too... 15:24:13 yeah, test key sounds good. 15:24:17 I believe it is for primary Branched 15:24:28 ah, this reminds me of one more thing 15:24:56 buildbranched and buildrawhide should be merged, but there is at least one diff where I am not sure whether it is intended that they differ 15:25:11 yeah, we have wanted to merge them for a long time. ;) 15:25:28 whats the diff? 15:25:30 I synced most of them 15:25:49 http://paste.fedoraproject.org/149368/56331421 15:26:01 there is a call for wait 15:26:03 * masta wonders in late 15:28:24 huh, no idea why thats there? 15:28:31 oh, unless it needs all arches to finish? 15:28:39 no, no idea 15:29:56 so can it just be synced to buildrawhide? 15:30:34 as far as I can tell. 15:30:45 there's actually nothing being backgrounded there is there? 15:31:41 pungify is 15:31:59 and spam-o-matic 15:32:19 ok, wonder why rawhide doesn't wait for those and branched does? 15:32:44 I would think both should 15:33:23 uh, actually both wait before pungify 15:33:47 ok... lets continue this after meeting? 15:33:53 but rawhide has an extra wait (So I confused both) 15:33:59 ok 15:34:12 thanks for coming everyone! 15:34:14 #endmeeting