13:01:26 <mvollmer> #startmeeting
13:01:26 <zodbot> Meeting started Mon Aug 17 13:01:26 2015 UTC.  The chair is mvollmer. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:01:26 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
13:01:31 <mvollmer> .hello mvo
13:01:32 <zodbot> mvollmer: mvo 'Marius Vollmer' <marius.vollmer@gmail.com>
13:01:40 <mvollmer> #topic Agenda
13:02:00 <mvollmer> * Multipath
13:02:12 <stefw> * journalctl regression
13:02:24 <stefw> * Further testing fires
13:03:15 <mvollmer> * hubbot disk usage
13:03:33 <dperpeet> .hello dperpeet
13:03:36 <zodbot> dperpeet: dperpeet 'Dominik Perpeet' <dperpeet@redhat.com>
13:03:46 <stefw> .hello stefw
13:03:47 <zodbot> stefw: stefw 'Stef Walter' <stefw@redhat.com>
13:03:59 <dperpeet> * Makefile conventions
13:04:07 <petervo> dperpeet, rpmquery is available
13:04:10 <stefw> * Internet Explorer 11
13:04:12 <dperpeet> petervo, thanks!
13:04:43 <mvollmer> * Moving things out of the shell
13:04:55 <stefw> * Moving things out of cockpit-wrapper
13:06:38 <mvollmer> ok, let's start
13:06:45 <mvollmer> #topic Multipath
13:06:52 <mvollmer> So I have started playing with this
13:07:09 <mvollmer> some notes here: https://github.com/cockpit-project/cockpit/wiki/Storage---Multipath
13:07:17 <github> [cockpit] stefwalter opened pull request #2590: Update cockpit.spec for changes from RHEL (master...rhel-spec-updates) http://git.io/v3pmW
13:07:46 <mvollmer> should be almost straightforward to get Cockpit "multipath safe"
13:07:59 <mvollmer> more work to make it "multipath nice"
13:08:07 <stefw> nice research
13:08:14 <stefw> how much of the work should go into storaged?
13:08:17 <github> [cockpit] dperpeet pushed 1 new commit to master: http://git.io/v3pmd
13:08:17 <github> cockpit/master 967318a Marius Vollmer: test: Only skip check-journal with systemd 219-21....
13:08:23 <mvollmer> i think we should offer UI for setting multipath up and configuring it
13:08:38 <mvollmer> storaged should expose the mpath configuration
13:08:45 <mvollmer> basically the multipathd state
13:08:46 <stefw> makes sense
13:09:08 <mvollmer> that is, storaged would need to wrap a d-bus api around multipathd
13:09:37 <mvollmer> the first bit of info we need is to identitfy which of the multiple block devices is the 'leader'
13:10:12 <stefw> phatina, have you seen the research mvollmer has done on multipath?
13:10:13 <mvollmer> initially, I'll do some regexp matching to find the device mapper device, but storaged should do that more directly
13:11:02 <mvollmer> I also found a bug in Cockpit re a misunderstanding of the storaged Drive property.
13:11:21 <phatina> stefw: yes
13:12:32 <mvollmer> phatina, I just learned about the sysfs slaves/ and holders/ construct
13:12:46 <mvollmer> maybe storaged could expose that?
13:13:06 <mvollmer> but I tink storaged should also natively know about multipath
13:13:20 <mvollmer> and then we don't need that generic slaves/ holders/ thing
13:13:52 <mvollmer> phatina, https://www.redhat.com/archives/dm-devel/2006-February/msg00091.html
13:15:06 <mvollmer> so anyway, I continue playing with this to learn more about which pieces can be relied upon.
13:16:14 <mvollmer> next topic?
13:16:29 <stefw> yip
13:16:45 <mvollmer> #topic journalctl regression
13:17:27 <stefw> A regression was committed up stream
13:17:28 <stefw> to systemd
13:17:33 <stefw> where journalctl -f -t unmatched
13:17:34 <stefw> no longer works
13:17:39 <dperpeet> 219-21
13:17:47 <stefw> and that patch was backported to 219-21
13:17:51 <stefw> and then pushed through f22 updates
13:18:00 <stefw> systemd git master now contains my fix to patch this
13:18:05 <stefw> but it's not yet fixed in F22
13:18:21 <stefw> https://bugzilla.redhat.com/show_bug.cgi?id=1253649
13:18:32 <stefw> Tests should be failing with this
13:18:41 <stefw> but we've shut off journal tests
13:18:46 <stefw> when that systemd version is present
13:19:02 <stefw> we need to remove the workaround once that bug is fixed
13:19:35 <stefw> that's it
13:19:37 <mvollmer> http://files.cockpit-project.org/hubbot/bug_status.html
13:19:44 <mvollmer> should be listed there ^^
13:19:59 <mvollmer> cockpit itself is also broken by the regression, of course.
13:20:06 <stefw> yup
13:20:23 <mvollmer> but not completely: as soon as the display is not empty, it will work
13:20:33 <mvollmer> i hope that is a rare thing
13:20:52 <dperpeet> should we extend the tests?
13:20:55 <dperpeet> and catch the issue
13:21:03 <dperpeet> / handle it
13:21:04 <mvollmer> well, we do... :-)
13:21:28 <mvollmer> ahh, you mean workaround in the cockpit code?
13:21:37 <dperpeet> yeah
13:21:41 <mvollmer> detect a non-blocking journalctl?
13:21:45 <dperpeet> I don't think it's that necessary
13:21:47 <mvollmer> I'd say no.
13:22:04 <mvollmer> what would we do, poll?
13:22:19 <stefw> we certainly shouldn't fix our journal code to do that
13:22:29 <dperpeet> there's no practical way
13:22:33 <mvollmer> we should only do it when f22 drags their feet too long
13:22:43 <stefw> we should raise it with FESCO if they do
13:22:47 <stefw> i'll prepare a patch to the rpm build
13:22:51 <mvollmer> yeah
13:22:57 <dperpeet> that makes more sense
13:23:08 <mvollmer> why did they include the patch in the first place?
13:23:25 <mvollmer> probably to fix the bug that I have reported way back... :-)
13:24:21 <mvollmer> https://bugs.freedesktop.org/show_bug.cgi?id=71548
13:25:09 <mvollmer> next topic?
13:25:39 <mvollmer> #topic Further testing fires
13:26:02 <mvollmer> phantomjs is quite often crashing with check-login, no?
13:26:11 <stefw> yeah it's the first time i've s tarted seeing that
13:26:15 <stefw> was it due to disk space issues?
13:26:26 <mvollmer> right, might be
13:26:42 <dperpeet> we can start a blanket retest of failed tests tonight
13:26:46 <mvollmer> /home was 100% full.
13:26:49 <dperpeet> when it doesn't block that much
13:27:14 <dperpeet> this was due to a reluctance to automatically delete images
13:27:23 <dperpeet> mvollmer and I fixed this today
13:27:24 <mvollmer> yeah, my fault
13:27:38 <dperpeet> hubbot now cleans up instead of pushing things into the basement
13:28:02 <mvollmer> we still use about 120 GiB
13:28:09 <dperpeet> how much space do we have left?
13:28:23 <mvollmer> let me check
13:28:26 <dperpeet> daily master checks are a pretty significant drain
13:28:34 <mvollmer> /dev/mapper/vg_dhcpclient06-lv_home  204G  137G   57G  71% /home
13:28:39 <dperpeet> we could automatically delete those as soon as a test finishes
13:28:45 <mvollmer> heh, not so useful
13:28:50 <dperpeet> indeed
13:28:52 <mvollmer> 57G available
13:29:34 <dperpeet> we haven't changed the wildcard regarding images to keep yet
13:29:44 <dperpeet> right now I believe all of a pull requests images are kept
13:29:55 <dperpeet> while we only really need the newest one
13:29:58 <mvollmer> 36G	cockpit-data
13:30:05 <mvollmer> 46G	hubbot
13:30:12 <mvollmer> not as bad as I thought
13:30:37 <mvollmer> 9.5G	/srv/testdata/
13:30:37 <dperpeet> we can tweak this a bit
13:30:59 <dperpeet> or even change behavior to more aggressively prune when disk space is running low
13:31:18 <mvollmer> one idea is to only keep the last run for a given commit
13:31:31 <mvollmer> now we keep them all, mostly because lazy
13:31:55 <mvollmer> anyway
13:31:57 <dperpeet> we probably need to do both
13:31:58 <mvollmer> other fires?
13:32:02 <stefw> i'll also try to look at more fires, and fix them
13:32:04 <stefw> there's this one:
13:32:12 <stefw> https://github.com/cockpit-project/cockpit/issues/2591
13:32:22 <stefw> the thing that keeps me going is that so many of these are real bugs
13:32:25 <stefw> both in cockpit and elsewhere
13:33:16 <mvollmer> yep
13:33:42 <stefw> mvollmer, can i assign that one to you?
13:33:48 <mvollmer> yes
13:34:21 <stefw> dperpeet, have you discovered anything about Travis?
13:34:30 <stefw> is it time to move away from Travis if it keeps falling over?
13:34:33 <stefw> or should we disable our root tests
13:34:35 <dperpeet> the tests seem to work again
13:34:51 <dperpeet> the thing is, I have the feeling they want to move away from the type of test we use
13:34:52 <stefw> but i still see broken output:
13:34:52 <stefw> https://travis-ci.org/cockpit-project/cockpit/jobs/75573396
13:34:54 <dperpeet> towards non-root containers
13:35:00 <stefw> i can move it to non-root
13:35:03 <stefw> but it means disabling certain tests
13:35:06 <stefw> that never get run anywhere
13:35:21 <dperpeet> all the tests I started after last friday afternoon seemed fine
13:35:30 <stefw> what about this one?
13:35:30 <stefw> https://travis-ci.org/cockpit-project/cockpit/jobs/75573396
13:35:39 <stefw> i can restart it
13:36:07 <dperpeet> that's three days ago
13:36:09 <dperpeet> I think
13:36:12 <dperpeet> from the date
13:36:31 <dperpeet> mvollmer, what was the other ci system you linked?
13:36:41 <mvollmer> semaphoreci.com
13:37:09 <dperpeet> they have caching
13:37:12 <stefw> from one non-open-source to another
13:37:13 <stefw> wheeee
13:37:51 <dperpeet> I can test how much effort it would be to move to travis' new structure
13:37:57 <mvollmer> why do we use travis in the first place?
13:38:03 <stefw> because it was so easy
13:38:11 <mvollmer> could we extend hubbot to do the job now?
13:38:27 <dperpeet> it's just make check et cetera
13:38:30 <dperpeet> we could have a dedicated vm
13:38:40 <stefw> and you've invented jenkins
13:38:45 <stefw> well already done that with hubbot
13:38:46 <dperpeet> adds load to our server
13:39:04 <dperpeet> a distinct service adds variety to the testing
13:39:25 <dperpeet> how about I see how much effort it would be to get our tests to run in the new travis?
13:39:32 <dperpeet> with containers
13:39:54 <dperpeet> they require non-root but supposedly cache better
13:40:06 <stefw> it's just one line change
13:40:18 <stefw> i can make that change
13:40:20 <stefw> not a problem
13:40:24 <stefw> well two lines
13:40:32 <dperpeet> unless something doesn't work :)
13:40:40 <dperpeet> well, go for it
13:40:48 <dperpeet> if something crops up, we can start putting out the fires
13:42:30 <stefw> hmmm, no it's not thath easy
13:42:37 <stefw> "If you require sudo, for instance to install Ubuntu packages, a workaround is to use precompiled binaries, uploading them to S3 and downloading them as part of your build, installing them into a non-root directory."
13:42:56 <dperpeet> like I said, non-root
13:43:27 <mvollmer> is running the unit tests in mock good enough?
13:43:37 <mvollmer> the output is hidden, but we can polish that up
13:44:02 <stefw> 'make distcheck'
13:44:05 <stefw> 'make check-memory'
13:44:09 <dperpeet> in mock on travis or with hubbot?
13:44:13 <stefw> and a full autogen.sh
13:44:14 <mvollmer> hubbot
13:44:17 <stefw> all those things need to be tested
13:44:23 <stefw> to have a similar feature set
13:44:33 <dperpeet> we could have a vm and do what travis does
13:44:45 <dperpeet> except we cache the updated base system
13:44:49 <mvollmer> also clang builds
13:45:15 <mvollmer> but I'd say we didn't get much value from those, or did we?
13:45:16 <stefw> yup
13:45:25 <stefw> well often one would fail and not the other
13:45:30 <dperpeet> we'd be putting more eggs into the hubbot basket
13:45:45 <dperpeet> and load on the server
13:46:18 <mvollmer> ok, time for the next topic?
13:46:29 <dperpeet> one last question
13:46:38 <mvollmer> should we arrange more discussion about this?
13:46:39 <dperpeet> stefw, do you want to look into travis changes?
13:46:45 <dperpeet> just to be clear on the action
13:46:47 <stefw> yes doing that now
13:46:50 <dperpeet> ok
13:46:55 <dperpeet> /eot
13:47:16 <mvollmer> #topic hubbot disk usage
13:47:26 <mvollmer> we have covered that
13:47:37 <mvollmer> #topic Makefile conventions
13:47:54 <mvollmer> dperpeet?
13:47:55 <dperpeet> background on this is that we have quite a few variables in the Makefile.am files
13:48:00 <dperpeet> for the individual packages
13:48:06 <dperpeet> and it's not that easy to write or review
13:48:15 <dperpeet> to see if files are missing, and which file goes into what
13:48:33 <dperpeet> do we want to simplify this or somehow write up a mini-guideline?
13:48:40 <dperpeet> or do we even have one and I missed it
13:48:51 <dperpeet> with debugdata, gz, min, data, ...
13:48:57 <dperpeet> bundle
13:49:13 <stefw> we need a guideline
13:49:20 <stefw> unfortunately it seems hard to make it simpler than it is
13:49:22 * stefw cries
13:50:02 <dperpeet> maybe we can add a sanity check
13:50:13 <dperpeet> compare with files present
13:50:18 <dperpeet> or even write one if necessary
13:50:27 <dperpeet> and those files need to be in the variables somehow
13:50:45 <dperpeet> we can agree on writing it into the wiki or hacking.md
13:50:56 <github> [cockpit] stefwalter opened pull request #2592: Use Travis container based infrastructure (master...travis-container-based) http://git.io/v3puJ
13:51:10 <mvollmer> I guess I got some details wrong/forgot some file and that is what broke Travis
13:51:48 <stefw> well because travis runs 'make distcheck'
13:52:01 <dperpeet> that passed for me locally
13:52:05 <mvollmer> for me too
13:52:39 <stefw> also difference between --enable-debug
13:52:40 <stefw> and not
13:52:43 <stefw> so it's good to have these checks
13:52:56 <dperpeet> enable-debug is probably the culprit
13:53:21 <dperpeet> of course it is - I can start writing the guidelines
13:53:40 <dperpeet> #action dperpeet to start wiki page on Makefile.am conventions for packages
13:54:20 <mvollmer> next?
13:54:47 <mvollmer> #topic Internet Explorer 11
13:54:56 * mvollmer rushes a bit, sorry
13:55:15 <dperpeet> I was able to reproduce the error message we saw in https://github.com/cockpit-project/cockpit/pull/2566
13:55:29 <dperpeet> and I couldn't see it locally after a patch, waiting on tests to run
13:56:05 <dperpeet> functionality seems fine
13:56:18 <dperpeet> so after merging current cockpit should work fine with IE11
13:56:21 <dperpeet> /eot
13:57:03 <mvollmer> # Moving things out of the shell / out of the cockpit wrapper
13:57:13 <mvollmer> i did some mechanical work there
13:57:19 <mvollmer> lots more still to do
13:57:38 <mvollmer> one goal could be to first get the shell into a clean state
13:57:48 <mvollmer> and move the legacy stuff somewhere else
13:58:01 <mvollmer> and then continue to clean this up, as we feel like it
13:58:33 <dperpeet> I think refactoring is more important than actual cleanup at this point
13:58:53 <dperpeet> a clean structure makes it easier to clean the smaller parts as we touch them
13:59:26 <stefw> moving stuff out of cockpit-wrapper is important
13:59:29 <stefw> as it lets us run as a container
14:00:43 <mvollmer> yep
14:01:01 <dperpeet> I like the level of change you put into it so far
14:01:02 <mvollmer> ok, my plan is to return to this after multipath
14:03:17 <stefw> sounds good
14:03:37 <mvollmer> with priority on cockpit-wrapper
14:04:04 <dperpeet> yeah
14:04:12 <mvollmer> #topic Any other biz
14:05:10 <stefw> we need to do a release today
14:05:19 <stefw> everything marked priority is targetted to go in
14:05:55 <mvollmer> ok, I'll do https://github.com/cockpit-project/cockpit/pull/2589
14:06:13 <stefw> subin, when #2572 is ready, i can review again
14:06:23 <stefw> and i'll try to merge #2566 when it's done testing
14:06:25 <petervo> mvollmer, i'm in the process of doing that
14:06:36 <subin> stefw , working on it
14:06:38 <mvollmer> oh, sorry.
14:06:41 <stefw> great
14:07:59 <dperpeet> stefw, the fixup goes into the second commit (fix race condition)
14:08:02 <dperpeet> in #2566
14:08:09 <stefw> ok
14:08:15 <dperpeet> if it works :)
14:08:28 <mvollmer> ok
14:08:32 <mvollmer> #endmeeting