15:01:05 <tflink> #startmeeting fedoraqadevel
15:01:05 <zodbot> Meeting started Tue Nov  4 15:01:05 2014 UTC.  The chair is tflink. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:05 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
15:01:10 <tflink> #meetingname fedoraqadevel
15:01:10 <zodbot> The meeting name has been set to 'fedoraqadevel'
15:01:17 <tflink> #topic Welcome
15:01:28 <tflink> #chair kparal mkrizek
15:01:28 <zodbot> Current chairs: kparal mkrizek tflink
15:01:43 <tflink> Who all do we have?
15:01:46 * mkrizek is here
15:01:50 * kparal 
15:01:51 <garretraziel> \me is here
15:01:56 <garretraziel> hah
15:01:58 * garretraziel is here
15:02:06 <kparal> garretraziel: a windows user!
15:02:07 * lbrabec is here
15:02:20 <lbrabec> or latex
15:02:40 * jskladan is here
15:03:01 * tflink doesn't quite understand the connection between specific people and latex/windows, but ok :)
15:03:26 <garretraziel> it should be LaTeX :-)
15:03:41 <lbrabec> sure sure
15:04:18 <tflink> before this descends into flamewars of nitpicking, let's get started since there's another meeting right after us
15:04:30 <tflink> #topic Status Updates
15:04:35 <kparal> see? it's motivating :)
15:04:51 <tflink> kparal: the latex/LaTeX flamewar or the time limit?
15:04:59 <kparal> time limit
15:05:27 <tflink> the current "big thing" is supporting disposable clients
15:05:49 <tflink> we have seen some progress, so wanted to go over that
15:06:11 * kparal listening
15:06:21 <tflink> I've not set up any test systems yet for remote logging - keep getting distracted by other stuff
15:06:51 <tflink> but one potential complication that came up when I was chatting with the beaker devs is this: what do we do about non-line-based artifacts?
15:07:08 <kparal> what is that?
15:07:09 <tflink> rsyslog can only handle stuff line-by-line - no tarballs, no images etc.
15:07:32 <kparal> we don't produce anything like that
15:07:41 <tflink> not at the moment, no
15:07:54 <kparal> why would anyone want to transfer a tarball through rsyslog?
15:08:18 <tflink> true, but that's not what I was getting at. I'll rephrase
15:09:00 <tflink> what do we do if we have tasks which don't output line-based artifacts? stuff like tarballs, generated files, pictures, disk images etc.
15:09:28 <tflink> if we're using something like rsyslog for remote logging - that effectively prevents us from supporting those kind of outputting tasks
15:09:54 <kparal> we will need to mark them somehow for buildbot to transfer them over to the buildbot master
15:10:18 <tflink> so my plan was to poke at some more generic distributed filesystems to see if it would be reasonable to use them in place of or in addition to something for tracking taskotron.log
15:10:49 <tflink> but at the moment, I think that rsyslog is probably the best solution for us since the primary goal was to collect taskotron and system logs in one place
15:10:55 <kparal> even if we log remotely, I think we can't get rid of a generic "result output" directory where arbitrary files can be stored
15:11:13 <tflink> yeah, but we might be able to put it off
15:11:43 <tflink> eventually, I suspect that we might have something similar to what autotest did: directory for a job
15:12:06 <tflink> #info remote logging hasn't progressed much, still need to set up demo systems and make a final decision
15:12:43 <tflink> #info a more generic distributed filesystem may be an option for logging and/or storing output artifacts but this can likely wait for later
15:12:56 <tflink> I think that's it on the logging
15:13:06 <tflink> jskladan: can you give a quick update on execdb?
15:13:51 <jskladan> well, there is not much to update on, I'm in the state of waiting for the "taskotron in cloud"
15:14:03 <jskladan> so we can deploy the initial code and see how it works
15:14:23 * tflink was tempted to say "status with the exception of the bit about pretty much being blocked on me" :)
15:14:26 <kparal> what is taskotron in cloud?
15:14:35 <jskladan> since the initial code, I briefly explored the buildbot more
15:14:49 <tflink> kparal: a method for deploying a taskotron system as a local dev/demo setup
15:14:50 <jskladan> but the data we currently collect seems to be about the limits of it at the moment
15:14:59 <jskladan> there will need to be changes trigger-wise
15:15:01 <jskladan> in the execdb
15:15:10 <tflink> everything in one machine/vm (trigger, resultsdb, master, clients)
15:15:13 <jskladan> since we want to store the fedmsg data (IIRC)
15:15:28 <jskladan> and the place where I put the code for requesting new UUID
15:15:40 <jskladan> is (IIUIC) far behind that
15:16:07 <tflink> #info execdb has an initial implementation, waiting for a place to deploy it for demo/testing
15:17:11 <tflink> jskladan: thanks for the update
15:17:18 <jskladan> np :0
15:17:19 <jskladan> :)
15:17:42 <tflink> mkrizek: can you give an update on how the openstack+buildbot stuff has been going?
15:18:13 <mkrizek> yeah, the next step is to create an image, which I need to discuss with infra
15:18:14 <kparal> have you asked the guys I sent you contact for?
15:18:20 <mkrizek> the image that contains a buildslave that is
15:18:34 <kparal> those with buildbot and working openstack deployment
15:19:08 <tflink> kparal: I haven't responded yet, no. it's on my todo list but keeps falling off :-/
15:19:38 <kparal> ok, just reminding there are some people who maybe solved it already
15:19:41 <mkrizek> do they use openstack or libvirt?
15:19:44 <tflink> libvirt
15:19:54 <mkrizek> ok
15:19:55 <tflink> mkrizek: that's the all-in-one solution that we were talking about last week
15:20:06 <mkrizek> yeah, I thought so
15:20:09 <kparal> ah, right, I'm starting to remember it more clearly
15:20:27 <kparal> strike openstack, my error
15:20:37 <tflink> I have some concerns about that implementation but I'm planning to ask them some questions
15:21:14 <tflink> mkrizek: what about starting with a generic image and updating, even though that'd take a huge amount of time/build
15:21:35 <tflink> or am I misunderstanding where you are with the investigation/demo?
15:21:48 <mkrizek> the image needs to have buildslave installed
15:22:02 <tflink> oh
15:22:03 <mkrizek> which I didn't realized before
15:22:06 * mkrizek feels stupid
15:22:08 <tflink> me neither
15:22:17 <tflink> but yeah, obvious when someone says it :)
15:22:27 <mkrizek> :)
15:23:23 <tflink> #info openstack+buildbot is somewhat working, need to get a new image with buildslave installed before more progress can be made
15:23:33 <mkrizek> when the image is done I *think* we *should* be all set
15:23:43 <tflink> mkrizek: would it be possible to use the post-install scripts in openstack to install/configure buildslave?
15:24:22 <tflink> mkrizek: you might want to explore the fedora-cloud bits, I think they have a method for creating a custom qcow image
15:24:36 <mkrizek> I don't think the buildmaster has any access to the slave, but I'll check
15:24:40 <tflink> #info hopefully, the image stuff is the last bit that needs to be done
15:25:06 <tflink> yeah, I don't remember what the latent buildslave lets you access in terms of openstack functionality
15:25:49 <tflink> mkrizek: anything else on disposable clients?
15:26:06 <mkrizek> nope, that's it
15:26:09 <tflink> cool, thanks
15:26:14 <tflink> which brings us to depcheck
15:26:45 <tflink> we had a bug report about depcheck failing x86_64 when it shouldn't
15:27:02 <tflink> it took a bit, but we figured out that it was a pre/ordering problem
15:27:14 <kparal> do we have a ticket number?
15:27:16 <tflink> the bug appeared to be happening every couple of days
15:27:49 <tflink> https://phab.qadevel.cloud.fedoraproject.org/T366
15:28:06 <tflink> a fix has been coded and reviewed, currently running in dev
15:28:52 <tflink> but while I was fixing that, I started formalizing the situations where depcheck doesn't work like we want it to
15:29:16 <tflink> https://bitbucket.org/fedoraqa/depcheck_scenarios/src/3ad160ad3145dc4595286291e5e43e2d91737ea9/scenarios/known_broken/?at=master
15:29:38 <kparal> I read the blogpost, I'd like to respond to it soon
15:29:38 <tflink> and did a rather long writeup on the issue along with two potential solutions
15:29:47 <tflink> http://tirfa.com/current-state-of-depcheck-and-paths-forward.html
15:30:16 <tflink> so, not a whole lot to say about it
15:30:30 <tflink> after a minor issue fix yesterday, dev has been running fine
15:30:44 <kparal> great
15:30:52 <tflink> I'll be moving the updated depcheck to stg today, probably to prod when that gets updated post-freeze
15:31:37 <tflink> there's some other metrics stuff I've been working on but that'll come up later in the agenda
15:31:54 <tflink> if there's nothing I've forgotten about, let's move on
15:32:28 <tflink> #topic Potentially Upcoming Stuff
15:32:48 * tflink wanted to spend a bit of time updating folks on stuff that's (potentially) on the horizon
15:33:14 <tflink> there has been some renewed talk about using automated results for build/update gating
15:33:34 <tflink> bodhi2 will probably have support for this when it's deployed after f21 is released
15:33:46 <tflink> https://github.com/fedora-infra/bodhi
15:34:01 <tflink> s/will probably/may
15:35:01 <tflink> so it would likely be wise to start looking more closely at what exactly we're failing/passing to get an idea of any holes in the results
15:35:43 <tflink> the depcheck hole I wrote about is going to be relevant here - is that enough of an issue to spend time on fixing sooner than later?
15:36:15 <tflink> adamw also had a question about starting to run rpmguard again
15:36:28 <tflink> to detect changes in provides, reuqires etc.
15:37:30 <tflink> another thing that I wanted to mention is that we're going to have to look at blockerbugs again in the not-so-distant future
15:37:43 <tflink> people are using it, noticing non-trivial bugs and asking for enhancements
15:38:21 <tflink> I was hoping to put that off until we have fedmsgs coming out of bugzilla, since that'll require some major refactoring anyways
15:38:42 <mkrizek> are those non-trivial bugs filed?
15:38:46 <kparal> it might be wise not to increase the number of maintained projects for the time being (rpmguard etc)
15:38:51 <tflink> but it might be worth poking at before then if we can't get a "guess date" on when that'd actually happen
15:38:59 <tflink> mkrizek: not sure, I should check
15:39:28 <tflink> #action tflink to verify that the feature requests and bugs he's been told about have been filed
15:39:49 <tflink> kparal: yeah, just noting that it was requested
15:40:18 <tflink> I suspect that automation will be a topic if the governance/direction vfad happens next year
15:40:48 <kparal> regarding depcheck, I'd rather keep smaller functionality, but 100% working, than expanding coverage. I think spending the time on taskotron itself is more useful
15:41:14 <tflink> kparal: for the moment, I agree as long as we aren't failing too often
15:41:29 <kparal> especially after reading your blogpost, tflink, it seemed scarily complicated :)
15:41:53 <tflink> yeah, I think it would take a week at least
15:42:04 <kparal> x 3.5
15:42:19 <kparal> that's my magic multiplication constant
15:42:23 <tflink> but given that I haven't done any more than a PoC, it could easily expand a lot more
15:42:35 <tflink> kparal: I did say _at least_ :) but you do have a point
15:43:03 <tflink> but this is a good segue into metrics
15:43:34 <tflink> I wanted to answer the question of "how often is depcheck failing due to T366" and came up with two half-assed scripts for gathering data
15:43:37 <tflink> https://bitbucket.org/fedoraqa/taskmetrics
15:44:02 <kparal> how do you measure whether we failed or not?
15:44:08 <tflink> one of them gets all stable push requests over a time period T and correlates pass/fail
15:44:52 <tflink> it's looking for several situations: push with failed checks, negative karma after push with passed checks
15:45:03 <tflink> s/several/2
15:45:03 <kparal> oh, due to T366, got it know
15:45:06 <kparal> *now
15:45:34 <tflink> it's not perfect, but it's a decent start
15:45:41 <tflink> the other script is what ended up being more useful
15:45:52 <tflink> not documented yet, just pushed it earlier today
15:46:13 <tflink> it goes through all the depcheck results in a time period and extracts all the failure strings
15:46:18 <tflink> and sorts them by update
15:46:36 <tflink> I'm trying to run that daily, but I don't always remember to do it
15:47:34 <tflink> example output: http://paste.fedoraproject.org/147786/11603214/
15:48:41 <tflink> if we get folks looking for less time-sensitive projects, it would be interesting to have something like that run on a regular basis and generate html logs for viewing
15:49:56 <tflink> another option I thought of for less time-sensitive projects is to start asking folks to contribute upstream to buildbot
15:50:12 <tflink> their new release is starting to pick up steam and would help us immensely
15:50:33 <tflink> but it still seems to be quite a ways out
15:50:43 <tflink> figured I would mention it
15:50:54 <tflink> since we have 10 minutes left, let's move on to iteration 7
15:51:01 <tflink> #topic Taskotron Iteration 7
15:51:05 <tflink> bah
15:51:12 <tflink> #topic Taskotron Iteration 7
15:51:39 <tflink> I suspect that kparal and jskladan are going to get pulled right back into the validation treadmill
15:51:53 <jskladan> yay for validations! :)
15:52:00 <kparal> I have taken the double-bodhi-comment issue, since I expect I can solve it fast
15:52:21 <tflink> https://phab.qadevel.cloud.fedoraproject.org/w/taskotron-7/
15:52:21 <kparal> as for Final validation, we voted to move it one week earlier, i.e. TC1 should be tomorrow
15:52:51 <tflink> kparal: yeah, I think TC1 might be done (saw someone mention it on IRC, anyways)
15:53:09 <tflink> kparal: don't spend too much time on it, we won't be using bodhi1 for much longer and nobody's complained yet
15:53:21 <kparal> ok
15:53:26 <tflink> that's something I happened to notice while digging into the depcheck issue
15:54:00 * tflink is going to get a logging test setup done
15:54:08 <tflink> and the other misc. sysadmin tasks, I suppose
15:54:54 <tflink> unless someone else feels like deploying new builds :)
15:55:30 <tflink> mkrizek: you up for handling the mass reboot tomorrow? it's a bit late for you but might be good to have someone other than me familiar with the entire process
15:55:49 <adamw> tflink: for the record i wasn't interesting in running rpmguard per se, i was interested in a package-level depcheck based on changes in package provides: .
15:56:07 <tflink> adamw: true but I'm not really keen on adding that to depcheck
15:56:13 <mkrizek> tflink: sure
15:56:24 <tflink> the output, sure but depcheck is complicated enough as it is :)
15:56:45 <tflink> mkrizek: ok, we can touch base after the meeting if you still have time?
15:56:54 <tflink> I think that's about it, though
15:57:03 <tflink> if you take a ticket, please add it to iteration 7
15:57:18 * tflink will be pestering folks for status updates either friday or monday
15:57:40 <tflink> with that, I think we're on to ...
15:57:44 <tflink> #topic open floor
15:57:54 <mkrizek> tflink: I need to go after the meeting, can we postpone that until tomorrow?
15:58:18 <tflink> mkrizek: sure, the reboots aren't until 22:00 UTC or something tomorrow
15:58:35 <kparal> I had a question about ppisar requesting we disable subscription requirement in qa-devel. but we only have 3 minutes left, let's discuss in #fedora-qa
15:58:37 * tflink forgot to mention this but we're making progress on new infrastructure
15:58:56 <tflink> 2 new boxes have been delivered to PHX2, hopefully they'll be usable in a few weeks
15:59:33 <tflink> we're looking at using some of the old qa0* boxes as beaker clients, not sure if that's going to work yet, waiting on some RHIT folks
16:00:06 <tflink> but we may have a new version of beaker that can work in our infra in a month or so, so that's another bit which may be coming up
16:00:26 <tflink> bah, lots of stuff to mention :-/
16:00:32 <tflink> anyhow, we're out of time
16:00:40 * tflink will send out minutes shortly
16:00:46 <tflink> #endmeeting