15:01:05 #startmeeting fedoraqadevel 15:01:05 Meeting started Tue Nov 4 15:01:05 2014 UTC. The chair is tflink. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:05 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:01:10 #meetingname fedoraqadevel 15:01:10 The meeting name has been set to 'fedoraqadevel' 15:01:17 #topic Welcome 15:01:28 #chair kparal mkrizek 15:01:28 Current chairs: kparal mkrizek tflink 15:01:43 Who all do we have? 15:01:46 * mkrizek is here 15:01:50 * kparal 15:01:51 \me is here 15:01:56 hah 15:01:58 * garretraziel is here 15:02:06 garretraziel: a windows user! 15:02:07 * lbrabec is here 15:02:20 or latex 15:02:40 * jskladan is here 15:03:01 * tflink doesn't quite understand the connection between specific people and latex/windows, but ok :) 15:03:26 it should be LaTeX :-) 15:03:41 sure sure 15:04:18 before this descends into flamewars of nitpicking, let's get started since there's another meeting right after us 15:04:30 #topic Status Updates 15:04:35 see? it's motivating :) 15:04:51 kparal: the latex/LaTeX flamewar or the time limit? 15:04:59 time limit 15:05:27 the current "big thing" is supporting disposable clients 15:05:49 we have seen some progress, so wanted to go over that 15:06:11 * kparal listening 15:06:21 I've not set up any test systems yet for remote logging - keep getting distracted by other stuff 15:06:51 but one potential complication that came up when I was chatting with the beaker devs is this: what do we do about non-line-based artifacts? 15:07:08 what is that? 15:07:09 rsyslog can only handle stuff line-by-line - no tarballs, no images etc. 15:07:32 we don't produce anything like that 15:07:41 not at the moment, no 15:07:54 why would anyone want to transfer a tarball through rsyslog? 15:08:18 true, but that's not what I was getting at. I'll rephrase 15:09:00 what do we do if we have tasks which don't output line-based artifacts? stuff like tarballs, generated files, pictures, disk images etc. 15:09:28 if we're using something like rsyslog for remote logging - that effectively prevents us from supporting those kind of outputting tasks 15:09:54 we will need to mark them somehow for buildbot to transfer them over to the buildbot master 15:10:18 so my plan was to poke at some more generic distributed filesystems to see if it would be reasonable to use them in place of or in addition to something for tracking taskotron.log 15:10:49 but at the moment, I think that rsyslog is probably the best solution for us since the primary goal was to collect taskotron and system logs in one place 15:10:55 even if we log remotely, I think we can't get rid of a generic "result output" directory where arbitrary files can be stored 15:11:13 yeah, but we might be able to put it off 15:11:43 eventually, I suspect that we might have something similar to what autotest did: directory for a job 15:12:06 #info remote logging hasn't progressed much, still need to set up demo systems and make a final decision 15:12:43 #info a more generic distributed filesystem may be an option for logging and/or storing output artifacts but this can likely wait for later 15:12:56 I think that's it on the logging 15:13:06 jskladan: can you give a quick update on execdb? 15:13:51 well, there is not much to update on, I'm in the state of waiting for the "taskotron in cloud" 15:14:03 so we can deploy the initial code and see how it works 15:14:23 * tflink was tempted to say "status with the exception of the bit about pretty much being blocked on me" :) 15:14:26 what is taskotron in cloud? 15:14:35 since the initial code, I briefly explored the buildbot more 15:14:49 kparal: a method for deploying a taskotron system as a local dev/demo setup 15:14:50 but the data we currently collect seems to be about the limits of it at the moment 15:14:59 there will need to be changes trigger-wise 15:15:01 in the execdb 15:15:10 everything in one machine/vm (trigger, resultsdb, master, clients) 15:15:13 since we want to store the fedmsg data (IIRC) 15:15:28 and the place where I put the code for requesting new UUID 15:15:40 is (IIUIC) far behind that 15:16:07 #info execdb has an initial implementation, waiting for a place to deploy it for demo/testing 15:17:11 jskladan: thanks for the update 15:17:18 np :0 15:17:19 :) 15:17:42 mkrizek: can you give an update on how the openstack+buildbot stuff has been going? 15:18:13 yeah, the next step is to create an image, which I need to discuss with infra 15:18:14 have you asked the guys I sent you contact for? 15:18:20 the image that contains a buildslave that is 15:18:34 those with buildbot and working openstack deployment 15:19:08 kparal: I haven't responded yet, no. it's on my todo list but keeps falling off :-/ 15:19:38 ok, just reminding there are some people who maybe solved it already 15:19:41 do they use openstack or libvirt? 15:19:44 libvirt 15:19:54 ok 15:19:55 mkrizek: that's the all-in-one solution that we were talking about last week 15:20:06 yeah, I thought so 15:20:09 ah, right, I'm starting to remember it more clearly 15:20:27 strike openstack, my error 15:20:37 I have some concerns about that implementation but I'm planning to ask them some questions 15:21:14 mkrizek: what about starting with a generic image and updating, even though that'd take a huge amount of time/build 15:21:35 or am I misunderstanding where you are with the investigation/demo? 15:21:48 the image needs to have buildslave installed 15:22:02 oh 15:22:03 which I didn't realized before 15:22:06 * mkrizek feels stupid 15:22:08 me neither 15:22:17 but yeah, obvious when someone says it :) 15:22:27 :) 15:23:23 #info openstack+buildbot is somewhat working, need to get a new image with buildslave installed before more progress can be made 15:23:33 when the image is done I *think* we *should* be all set 15:23:43 mkrizek: would it be possible to use the post-install scripts in openstack to install/configure buildslave? 15:24:22 mkrizek: you might want to explore the fedora-cloud bits, I think they have a method for creating a custom qcow image 15:24:36 I don't think the buildmaster has any access to the slave, but I'll check 15:24:40 #info hopefully, the image stuff is the last bit that needs to be done 15:25:06 yeah, I don't remember what the latent buildslave lets you access in terms of openstack functionality 15:25:49 mkrizek: anything else on disposable clients? 15:26:06 nope, that's it 15:26:09 cool, thanks 15:26:14 which brings us to depcheck 15:26:45 we had a bug report about depcheck failing x86_64 when it shouldn't 15:27:02 it took a bit, but we figured out that it was a pre/ordering problem 15:27:14 do we have a ticket number? 15:27:16 the bug appeared to be happening every couple of days 15:27:49 https://phab.qadevel.cloud.fedoraproject.org/T366 15:28:06 a fix has been coded and reviewed, currently running in dev 15:28:52 but while I was fixing that, I started formalizing the situations where depcheck doesn't work like we want it to 15:29:16 https://bitbucket.org/fedoraqa/depcheck_scenarios/src/3ad160ad3145dc4595286291e5e43e2d91737ea9/scenarios/known_broken/?at=master 15:29:38 I read the blogpost, I'd like to respond to it soon 15:29:38 and did a rather long writeup on the issue along with two potential solutions 15:29:47 http://tirfa.com/current-state-of-depcheck-and-paths-forward.html 15:30:16 so, not a whole lot to say about it 15:30:30 after a minor issue fix yesterday, dev has been running fine 15:30:44 great 15:30:52 I'll be moving the updated depcheck to stg today, probably to prod when that gets updated post-freeze 15:31:37 there's some other metrics stuff I've been working on but that'll come up later in the agenda 15:31:54 if there's nothing I've forgotten about, let's move on 15:32:28 #topic Potentially Upcoming Stuff 15:32:48 * tflink wanted to spend a bit of time updating folks on stuff that's (potentially) on the horizon 15:33:14 there has been some renewed talk about using automated results for build/update gating 15:33:34 bodhi2 will probably have support for this when it's deployed after f21 is released 15:33:46 https://github.com/fedora-infra/bodhi 15:34:01 s/will probably/may 15:35:01 so it would likely be wise to start looking more closely at what exactly we're failing/passing to get an idea of any holes in the results 15:35:43 the depcheck hole I wrote about is going to be relevant here - is that enough of an issue to spend time on fixing sooner than later? 15:36:15 adamw also had a question about starting to run rpmguard again 15:36:28 to detect changes in provides, reuqires etc. 15:37:30 another thing that I wanted to mention is that we're going to have to look at blockerbugs again in the not-so-distant future 15:37:43 people are using it, noticing non-trivial bugs and asking for enhancements 15:38:21 I was hoping to put that off until we have fedmsgs coming out of bugzilla, since that'll require some major refactoring anyways 15:38:42 are those non-trivial bugs filed? 15:38:46 it might be wise not to increase the number of maintained projects for the time being (rpmguard etc) 15:38:51 but it might be worth poking at before then if we can't get a "guess date" on when that'd actually happen 15:38:59 mkrizek: not sure, I should check 15:39:28 #action tflink to verify that the feature requests and bugs he's been told about have been filed 15:39:49 kparal: yeah, just noting that it was requested 15:40:18 I suspect that automation will be a topic if the governance/direction vfad happens next year 15:40:48 regarding depcheck, I'd rather keep smaller functionality, but 100% working, than expanding coverage. I think spending the time on taskotron itself is more useful 15:41:14 kparal: for the moment, I agree as long as we aren't failing too often 15:41:29 especially after reading your blogpost, tflink, it seemed scarily complicated :) 15:41:53 yeah, I think it would take a week at least 15:42:04 x 3.5 15:42:19 that's my magic multiplication constant 15:42:23 but given that I haven't done any more than a PoC, it could easily expand a lot more 15:42:35 kparal: I did say _at least_ :) but you do have a point 15:43:03 but this is a good segue into metrics 15:43:34 I wanted to answer the question of "how often is depcheck failing due to T366" and came up with two half-assed scripts for gathering data 15:43:37 https://bitbucket.org/fedoraqa/taskmetrics 15:44:02 how do you measure whether we failed or not? 15:44:08 one of them gets all stable push requests over a time period T and correlates pass/fail 15:44:52 it's looking for several situations: push with failed checks, negative karma after push with passed checks 15:45:03 s/several/2 15:45:03 oh, due to T366, got it know 15:45:06 *now 15:45:34 it's not perfect, but it's a decent start 15:45:41 the other script is what ended up being more useful 15:45:52 not documented yet, just pushed it earlier today 15:46:13 it goes through all the depcheck results in a time period and extracts all the failure strings 15:46:18 and sorts them by update 15:46:36 I'm trying to run that daily, but I don't always remember to do it 15:47:34 example output: http://paste.fedoraproject.org/147786/11603214/ 15:48:41 if we get folks looking for less time-sensitive projects, it would be interesting to have something like that run on a regular basis and generate html logs for viewing 15:49:56 another option I thought of for less time-sensitive projects is to start asking folks to contribute upstream to buildbot 15:50:12 their new release is starting to pick up steam and would help us immensely 15:50:33 but it still seems to be quite a ways out 15:50:43 figured I would mention it 15:50:54 since we have 10 minutes left, let's move on to iteration 7 15:51:01 #topic Taskotron Iteration 7 15:51:05 bah 15:51:12 #topic Taskotron Iteration 7 15:51:39 I suspect that kparal and jskladan are going to get pulled right back into the validation treadmill 15:51:53 yay for validations! :) 15:52:00 I have taken the double-bodhi-comment issue, since I expect I can solve it fast 15:52:21 https://phab.qadevel.cloud.fedoraproject.org/w/taskotron-7/ 15:52:21 as for Final validation, we voted to move it one week earlier, i.e. TC1 should be tomorrow 15:52:51 kparal: yeah, I think TC1 might be done (saw someone mention it on IRC, anyways) 15:53:09 kparal: don't spend too much time on it, we won't be using bodhi1 for much longer and nobody's complained yet 15:53:21 ok 15:53:26 that's something I happened to notice while digging into the depcheck issue 15:54:00 * tflink is going to get a logging test setup done 15:54:08 and the other misc. sysadmin tasks, I suppose 15:54:54 unless someone else feels like deploying new builds :) 15:55:30 mkrizek: you up for handling the mass reboot tomorrow? it's a bit late for you but might be good to have someone other than me familiar with the entire process 15:55:49 tflink: for the record i wasn't interesting in running rpmguard per se, i was interested in a package-level depcheck based on changes in package provides: . 15:56:07 adamw: true but I'm not really keen on adding that to depcheck 15:56:13 tflink: sure 15:56:24 the output, sure but depcheck is complicated enough as it is :) 15:56:45 mkrizek: ok, we can touch base after the meeting if you still have time? 15:56:54 I think that's about it, though 15:57:03 if you take a ticket, please add it to iteration 7 15:57:18 * tflink will be pestering folks for status updates either friday or monday 15:57:40 with that, I think we're on to ... 15:57:44 #topic open floor 15:57:54 tflink: I need to go after the meeting, can we postpone that until tomorrow? 15:58:18 mkrizek: sure, the reboots aren't until 22:00 UTC or something tomorrow 15:58:35 I had a question about ppisar requesting we disable subscription requirement in qa-devel. but we only have 3 minutes left, let's discuss in #fedora-qa 15:58:37 * tflink forgot to mention this but we're making progress on new infrastructure 15:58:56 2 new boxes have been delivered to PHX2, hopefully they'll be usable in a few weeks 15:59:33 we're looking at using some of the old qa0* boxes as beaker clients, not sure if that's going to work yet, waiting on some RHIT folks 16:00:06 but we may have a new version of beaker that can work in our infra in a month or so, so that's another bit which may be coming up 16:00:26 bah, lots of stuff to mention :-/ 16:00:32 anyhow, we're out of time 16:00:40 * tflink will send out minutes shortly 16:00:46 #endmeeting