15:02:40 #startmeeting [fedora-ci] UX for rawhide package gating 15:02:40 Meeting started Wed Mar 20 15:02:40 2019 UTC. 15:02:40 This meeting is logged and archived in a public location. 15:02:40 The chair is pingou. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:02:40 Useful Commands: #action #agreed #halp #info #idea #link #topic. 15:02:40 The meeting name has been set to '[fedora-ci]_ux_for_rawhide_package_gating' 15:03:51 ttomecek: bookwar: adamw: Son_Goku: tstellar_: bowlofeggs dperpeet Evolution :) 15:04:04 .hello2 15:04:05 dperpeet: dperpeet 'None' 15:04:06 .hello ngompa 15:04:09 Son_Goku: ngompa 'Neal Gompa' 15:04:10 coremodule: ping 15:04:12 adamw: Ping with data, please: https://fedoraproject.org/wiki/No_naked_pings 15:04:24 * adamw is sorta around 15:04:30 .hello2 15:04:31 bookwar: bookwar 'Aleksandra Fedorova' 15:04:35 .hello adamwill 15:04:36 adamw: adamwill 'Adam Williamson' 15:04:41 * nirik is also sort of here 15:04:53 .hello2 15:04:55 pingou: pingou 'Pierre-YvesChibon' 15:04:59 .hello tstellar 15:05:00 tstellar_: tstellar 'Tom Stellard' 15:05:12 * Son_Goku is here for a while 15:05:53 .hello jperrin 15:05:54 Evolution: jperrin 'None' 15:06:22 Evolution, you should fix that :) 15:07:05 alright, let's wait a couple of minutes more 15:07:16 why did jim get renamed after an email client 15:07:31 adamw: I've been wondering that for a while, but was too afraid to ask 15:07:33 adamw: I had this name before the email client. they're ruining my rep 15:07:37 :-P 15:07:51 (yes, I'm old) 15:07:53 hi mhroncok :) 15:08:02 Evolution: i mean, you can ask which of the two I prefer, but I wouldn't recommend it ;) 15:08:05 15:08:10 pingou: hi, sorry for being late, was typing some code and not checking time 15:08:34 adamw: huh, I always picked you for a gmail web interface kinda guy 15:08:36 mhroncok: no worries, we haven't started yet 15:08:37 * Evolution flees from adamw 15:08:43 * adamw waves white flag 15:08:50 uncle! 15:09:21 alright, let's get started and if more people show up they'll join then :) 15:09:33 so first of all, thanks for being here 15:10:19 we've managed to kick off the discussion over emails in the last two days and I was hoping that a more dedicated time (such as a meeting) could help improving the current proposal 15:11:17 my concerns are mostly around what the user experience should be 15:11:36 what we would deem acceptable changes to the current workflow and what we really do not want 15:13:04 does someone want to start poking holes into the current proposal? 15:13:46 * pingou re-reading the last email from tibbs 15:13:54 pingou: why is the single package / multi package thing separated? 15:14:02 pingou: for those who aren't on the email thread, maybe a quick elevator pitch? 15:14:13 since we're logging this meeting, might help for posterity 15:14:16 so we support some kind of easy fedpkg build --nogate when i want to do chained builds? 15:14:22 Evolution: good point, I'll start with this then go to your question mhroncok :) 15:14:49 so, during the discussion about https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates on the fedora-devel list 15:14:57 there were some concerns raised 15:15:14 some about the CI system and its capabilities (or in some cases lack thereof) 15:15:26 some about the opt-in/opt-out possibilit 15:15:28 +ies 15:15:47 some about the distinction between single-build updates and multi-builds updates that this proposal introduces 15:16:01 I believe we were able to answer some of the concerns raised, but not all 15:16:33 and the idea was to gather us to see if we could adjust the proposal in such a way that these concerns are no longer true or considered acceptable 15:16:53 so mhroncok regarding your question about the distinction b/w single and multi-build updates 15:17:05 we did this split for two reasons: 15:17:39 - we have experience with doing single-build gating in koji and know how to do it 15:18:10 - it's technically quicker to implement, thus given us a quick turn around for testing, validations and improvements 15:18:20 both makes sense 15:18:47 pingou: can we promise that once koji part required for multi-package changes is done, the single package workflow will be merged into the multi-package one 15:18:59 so in the end there will be just one multi-package workflow 15:19:14 single-package gate is just a milestone to achieve it 15:19:24 bookwar: I'd question is this is something we want or not? 15:19:24 09:37 < pingou> mvadkert: ok, so in your opinion, should bodhi check greenwave even for updates that passed greenwave? 15:19:27 no 15:19:41 pingou: i think it is 15:19:47 or ... in our case, we notify the user that somethin spooky is going on 15:19:52 pingou: but we do not change the gate 15:20:02 pingou: decision 15:20:07 bookwar: that means: creating a side-tag for every update, regardless of the number of builds in it 15:20:12 yes 15:20:25 and providing sidetag for testing for CI pipelines 15:20:41 this way we unify also how CI pipelines work with the inputs 15:20:41 I'm fine with this, but it does impact, potentially more, the packager workflow, unless we have a way to automate it 15:20:51 which at the moment, we do not 15:21:05 I like tibbs' suggestion in email to have fedpkg chain-build do the side-tag creation, so two steps in one command 15:21:15 we don't have any automation around packager workflows, which is why this gets really hairy 15:21:21 but I'm not sure we want fedpkg build to do side-tag creation 15:21:39 pingou, where would it go, then? 15:22:13 Son_Goku: the way I saw it, for multi-builds, was a separate command (or as tibbs suggested, in chain-build) 15:22:24 there are two classes of multi-builds 15:22:35 one class is where you know all the stuff is going to work, so you chain-build it 15:22:37 Having fedpkg chain-build do side-tag creation would be a generally useful feature in my opinion, even without gating. 15:22:45 tstellar_: +1 15:22:48 another class is when you don't know, and you're doing it one by one 15:23:10 my understanding is that the latter happens more often in rawhide, and the former happens more often in stable releases 15:23:11 pingou: i dont' think automation is the problem here, we can have various subcommands for different use-cases - chain-build, one by one build, ... 15:23:25 the idea is that the sidetag is the unit of work for a gate 15:23:46 ^ this makes sense 15:23:49 gate won't see things as builds, it will see them as tags with repos 15:23:51 I do like the idea 15:23:52 it's functionally equivalent to a PR 15:24:33 my question is more: how to make this as easy as possible to get in the packager's workflow 15:24:42 and that gets us miles closer to doing stuff like this: https://build.opensuse.org/project/staging_projects/openSUSE:Factory & https://build.opensuse.org/project/staging_projects/openSUSE:Leap:15.1 15:25:29 pingou, for now, if we're talking about minimal disruption right now, make it opt-in for everybody 15:25:30 but that's a step two, which requires koji changes, while in step one we can do the bodhi part with providing gating interface and operations on a single build level, and then transfer this to sidetags when they are ready 15:25:47 Son_Goku: the question is, what does opt in means here? 15:25:56 i think the 'it needs to do reverse dependency builds' response is interesting because of what it implies 15:26:24 bookwar, that means the rawhide-gating target must be specified to use it 15:26:33 i think what those responses are really telling us is "i am probably not going to bother with this because it won't test the thing that breaks most often" 15:26:45 adamw has it right on the money 15:27:00 Son_Goku: so you'll opt-in at the package (gating.yml in dist-git) + at the build (fedpkg build --target=rawhide-gated)? 15:27:05 Son_Goku: why? what are you opting in for? ihaving pre-tag doesn't affect your workflow unless you have test resuired 15:27:06 if that's true, it doesn't matter a lot how easy it is to put this into your workflow as a packager, if you think it's just not going to help a lot anyway 15:27:09 required* 15:27:22 * cverna joins 15:27:28 pingou, there are packages where other people have put in the gating.yml file 15:27:39 that doesn't mean everyone is equipped to deal with the consequences of it 15:27:48 bookwar: ^ I think this was for you :) 15:27:48 Another benefit as having sidetag as the unit of work is it's easier to integrate modules, since modules come with their own tag. 15:27:50 if you have no tests configured - gating doesn't affect you, so there is no need to opt-in for gating, you opt-in for tests 15:28:00 Atomic WG likes putting CI in stuff that no one can deal with, for example 15:28:24 and when you're trying to fix the main Fedora deliverables, it sucks that the Atomic-only CI stops you 15:28:24 adamw: this is an interesting point 15:28:36 and you can't even fix atomic stuff anyway 15:29:02 bookwar: dperpeet: is the reverse dependency testing something that's in the roadmap for the CI system? 15:29:05 bookwar, and lets not forget that taskotron and bodhi can enforce all kinds of checks irrespective of gating.yml for failing a build for CI 15:29:23 Son_Goku: that's not correct :) 15:29:39 pingou, they don't right now, but they *can* 15:29:44 taskotron doesn't enforce anything 15:29:52 Son_Goku: no really, taskotron cannot :) 15:30:00 well, yes, strictly speaking, taskotron can't do much except run what it's told to run :) 15:30:21 only bodhi can enforce either based on its logic or relying on greenwave's decision engine 15:30:24 pingou, bookwar I'd say reverse dependency testing is something that's proven well for other established systems, so yes it should be there 15:30:45 dperpeet: there being on the roadmap? 15:30:50 can we go back to workflow? 15:30:51 yes 15:30:57 dperpeet: ok thanks 15:31:00 Son_Goku: that's why we need a gating: to let various systems which run something to make use of those runs 15:31:02 mhroncok: please :) 15:31:25 bookwar, also, lets not forget how little of these tests can be run locally :) 15:31:26 let's assume I've opted in for gating for my package 15:31:38 so I do: fedpkg build 15:32:01 the package is built, tagged into some gating tag or whatever, all the different kinds of tests are started 15:32:40 1st, I want to get an instant feedback about the bodhi update - but the build is async, so teh feedback is async as well 15:33:00 correct 15:33:24 2nd, in bodhi, I need to see what kinds of tests will run (or are already running), being able to click on them and see the details 15:33:37 well the bodhi update will be automatically created right after the build, so it'll not be instant but still quite quick 15:33:56 pingou: but how do I know about the bodhi update? 15:34:08 do i get an e-mail? is it linked from koji build? 15:34:11 mhroncok: how would you like to know? 15:34:23 just going to the UI would work or would you like something else? 15:34:37 what UI - that's where I'm heading 15:34:50 bodhi.fp.o 15:34:52 so i do fedpkg build --nowait 15:34:53 alright folks, I gotta go, I'll be back in a bit 15:34:58 and i geta koji link to probe 15:35:21 well, we already *have* koschei doing reverse build tests, right? have we thought about just having koschei file results to resultsdb? 15:35:21 I shouldn't need to rememeber to go to bodhi and see my updates if it is there yet 15:35:36 adamw: kicked an email just on that question this morning :) 15:35:53 great 15:36:05 mhroncok: you shouldn't have to go there at all if the tests pass 15:36:35 pingou: the tests will take days to finsih, especially if we do the koschei thing 15:36:42 unless you want to monitor them indeed 15:36:46 I want to be able to monitor the progress, as I'm with the build 15:36:47 mhroncok: if you do --nowait, you need to remember that you are not done yet anyway, how would you like to get reminded about it? 15:37:09 bookwar: I have the link for koji right away - that's one place I check 15:37:20 now I need to figure out the second link 15:37:38 mhroncok: I believe bowlofeggs' idea was to have comment be added for messages from the CI system about running, passed/failed 15:37:58 so the bodhi notifications about new comment would kick in there and let you know about this 15:38:15 I don't get bodhi notification for updates soembody else created 15:38:19 mhroncok: how does it work for builds to stable branches currently? where you need karma? 15:38:38 bookwar: for stable, i create the update - that's the current workflow 15:39:05 for rawhide, i don't - and that's what everybody is used to - so we need to remind them -> bah, here's your bodhi update 15:39:14 if you get email when bodhi update is created, will it be sufficient? 15:39:20 mhroncok: I'll follow up on this with bowlofeggs but one option would be to have the update be created under the user who made the build 15:39:35 see this tab? it says 5 tests are running, 3 are already OK and 15 more will yet to strat 15:39:53 click on this test to see what's going on 15:40:39 oh, it errored because whatever? click this "restart test" button to make it run again or check this "ignore result" checkbox if you know what you are doing 15:41:25 mhroncok: this are points to bodhi interface, before you were saying you need a way to _get_to_ bodhi interface in the first place 15:41:54 bookwar: yes, i'm still saying that 15:41:57 (good ones nonetheless) 15:42:04 I need both 15:42:37 ok 15:42:54 the updates ebing created under my user probably solve the "get the update link" problem 15:43:00 so we need to tell the the packager which update was automatically created 15:43:15 so he can go there and check the status of the tests 15:43:20 to summarize 15:43:22 yes 15:43:23 .hello2 15:43:24 or debug 15:43:25 bowlofeggs: bowlofeggs 'Randy Barlow' 15:43:42 Hi bowlofeggs, we were just thinking about you :) 15:43:54 depending on your FMN settings, you could get an email about the update create even if you don't create it... 15:43:54 cverna: yes 15:43:56 mhroncok, pingou: actually, sebwoj wrote a patch for bodhi that comments when the test gating status changes, so it's not just an idea, it's implemented (not released yet, will be in 4.0.0) 15:44:00 if you just own the package 15:44:43 mhroncok, pingou: yeah i think we would show the user who build the package as the one who created the update 15:44:46 nirik: you can. that's good, but we want this to be easy, so unless this is the default, it's not very helpful 15:44:48 so you should get an e-mail 15:45:22 and than you get 85 more e-mails at the tests pass/fail? 15:45:25 sure, I am just also saying some people could opt out if they deliberately do so... 15:45:43 bowlofeggs: do you remember if the comment on gating status changes include a link to the corresponding jenkins' job? 15:45:50 I don't think the test results should comment, you should just be able to see it 15:46:07 mhroncok: one doesn't go against the other 15:46:09 mhroncok: well if it's one update you should only get one e-mail when the tests pass/fail 15:46:14 a mass build should go into one update 15:46:28 bowlofeggs: so it's always just one test? 15:46:31 but the commit gives you this notification/states of thing while not making you go to bodhi's UI 15:46:42 I assumed the idea is that you can have plenty different tests in gating.yaml 15:46:46 incl. reverse deps 15:46:50 pingou: no link to a jenkins job is in the e-mail - it's just a comment telling you that a gating decision was changed 15:46:56 you'd have to click to the update to see more 15:47:18 roger 15:47:20 mhroncok: it's not a comment per test, it's a comment per gating decision 15:47:27 so it's just one absolute gating decision you'll get via comment? that's good 15:47:28 so ideally, you'd just see it once 15:47:42 in reality, the gating system is flaky and that could generate multiple e-mails 15:47:53 heh 15:47:57 but I get it 15:48:09 so I go to bodhi and I see that there are tests that will run? 15:48:18 or i only see the tests that already started? 15:48:35 mhroncok: currently you can only see the tests that finished 15:48:37 aka do I get instant feedback on: what is this update waiting for? 15:48:45 so you can't see planned ones or currently running ones, unfortunately 15:48:48 i would like that though 15:49:04 that's IMHO a must for good UX of this 15:49:04 the CI pipelines announces when it starts a new run 15:49:18 mhroncok: yeah i've long wanted that too - otherwise you can't know if the system is stuck or not 15:49:23 taskotron does not, at present 15:49:25 if I see an automatic update stuck in bodhi for 3 days I want to be able to see what's going on 15:49:36 there is an interesting problem there too: for some taskotron tests it's difficult or impossible to do 15:49:55 well we know what the package is being gated on 15:49:56 because the test job runs on an entire tag, and you can only figure out what packages it ran on *when it's done* 15:50:03 adamw: it kinda does via execdb 15:50:11 but yeah, listing tests expected for the gating would be easy 15:50:15 so at least we can say: this is gated on taskotron.whatnot.lint and it is yet to be started/finsihed 15:50:20 mhroncok: yeah that's a good point, we could at least say we are waiting on the ones we gate on 15:50:46 and while the update is being gated 15:50:47 I believe greenwave does give us that info 15:50:55 we may just not be showing it right now 15:50:57 is it, or is it not available in the buildroot? 15:50:57 pingou: ok, i meant it does not do it via the 'standardized' fedmsgs :) 15:51:06 adamw: fair :) 15:51:46 mhroncok: for me, as long as the update isn't stable it's not in the buildroot 15:52:05 (just like side-tag currently are) 15:52:14 and i can do buildroot override if I need to? so the experience is similar to stable releases 15:52:44 with side-tags you can't quite buildroot override, you could rebuild the package in your own side-tag though 15:53:05 then there may be a conflict when merging the two side-tags (because one will have a newer build) 15:53:14 so each update gets it's own side tag by magic? 15:53:34 at this point, the second side-tag being merged will need to rebuild that package again 15:53:43 (I'm lost) 15:53:46 mhroncok: that's the idea for multi-build updates 15:53:59 and what's the idea with single package updates? 15:54:06 which is what you're describing (if you need a buildroot override, you're in a multi-build update) 15:54:06 so I do a build, it gets and update 15:54:16 gating fails - oh, i need to rebuild this as well - what do i do? 15:54:36 create a side tag, move your build to it, rebuild this in that side-tag 15:54:40 unpush and start over with a different approach? 15:54:57 err 15:55:12 bowlofeggs: would we be able to edit an existing update to replace a build (or list of builds) by a side-tag? 15:55:37 or should we open a new update from this side-tag that will deprecate the existing update? 15:56:16 pingou: hmm, i hadn't considered this problem, but yeah i agree we need to do something 15:56:37 it might be nice to allow converting an update to a side tag 15:56:48 and it'd be nice if bodhi could move that build for you in doing so? 15:57:16 should bodhi then create the side-tag as well? 15:57:26 so you build, it creates an update, tests fail 15:57:35 you go to the ui and click on "convert to side-tag" ? 15:57:39 our original plan had bodhi creating side tags, but we had decided that fedpkg would do it via a koji plugin more recently 15:58:03 one isn't against the other 15:58:08 if we make it so you can create it either way, i guess that's ok (but more work for bodhi ☺) 15:58:14 esp in the case where you know in advance you'll need a side-tag 15:58:32 I like the "convert to side-tag" idea 15:58:42 and when I build in the side tag, it gets added to the update 15:58:56 so i don't have to do anything 15:59:10 other than click "ready/submit" in the update 15:59:21 yeah we were going to have a "create update from side tag" which was going to work like this 15:59:48 but it wasn't going to create the side tag itself. i guess we could make it do that, but then we diverge from what bookwar is working on a bit 16:00:06 I don't think so 16:00:15 because you still need fedpkg side-tag create 16:00:39 what the bodhi "convert to side-tag" gives us is a shorter way to do: fedpkg side-tag create 16:00:48 koji move tag1 tag2 16:00:59 yeah 16:01:19 Are normal users allowed to run koji move? 16:01:32 nirik: ^? 16:01:41 so convert-to-side-tag sounds like a good improvement, but likely something for a 1.1 or 1.2 16:01:57 assuming the answer to tibbs' question is yes :) 16:03:27 what's the UX of seeing the test run, being able to restart them and being able to waive them? 16:03:33 I might have got it wrong but if a single build update fails gating and you need to build a second package to fix the tests, then it becomes a multi build update right ? 16:03:37 I assume if they have perms to both tags 16:03:41 cverna: yes 16:04:01 ok so we need multi-build for the first release 16:04:24 cverna: or you waive that build and opt-out for the other ones 16:04:50 then better not opt-in at all if you can fix the tests 16:05:23 that frustrating to be able to fix something 16:05:26 not* 16:05:31 fair 16:05:49 mhroncok: waiving is centralized, it is integrated with greenwave, there is a cli to waive, but i guess bodhi button to issue a waiver is also possible. For retrigger CI system must provide a "rebuild" link in the message when it sends a result 16:06:06 messages, clis, uaaa 16:06:14 cverna: you can retrigger failed test once the other fix has landed 16:06:48 mhroncok: bodhi has a button to waive tests, but i've never witnessed that working correctly and haven't heard of anyone else who has so… 16:06:56 mhroncok: remember that the other half of the community reacts the same way on "bodhi, click, interface, AAA" 16:06:57 bookwar: we're in the case where there is no side-tag, so all builds are considered independently 16:07:00 I see a fialure in bodhi, I want a [restart] button and an [ignore] checkbox - of course being able to do this via command line is a good thing, but I don't want (as a packager) to think about builds, messages, greenware, etc.) 16:07:33 mhroncok: +1 16:08:09 mhroncok: anything else on your list? 16:08:18 otherwise I'd like to try to summarize what I wrote down 16:08:26 mhroncok: i was providing the background, but yes, all actions are available, it shouldn't be a big deal to add them in the ui 16:08:38 there was no answer for "how do i see the test run, its progress" 16:09:03 but that I guess is more of a question for individual tests systems 16:09:15 currently we don't have requirement on CI systems to send a test running message 16:09:25 one thing we could do it comment on the update with links to the tests 16:09:27 we have this message defined, bt it is considered optional 16:09:44 (for the systems that do support these messages ^) 16:10:14 generally we need to work on a CI system requirements doc, i think 16:10:17 (getting a linkt ot he test is one thing, seeing some output form the tast is a second thing) 16:10:55 anyway, pingou, I'll let you summarize :) thanks 16:11:05 so; 16:11:39 - I want to be informed about the bodhi update that's created for my build (let's agree to inform only at the point there is something of interest for the packager: tests started, tests completed) 16:12:02 - I want to see what is blocking the update from going through (pending test results) 16:12:12 - I want to retest/waive from the UI (and the CLI) 16:12:38 - I would like to be able to convert a single build update into a side-tag-based update 16:13:07 - I would like to be able to follow the progress of test system 16:13:46 mhroncok: did I miss anything ? 16:14:12 I don't think so 16:14:38 based on this, I think we may want to merge back the two proposals into one 16:15:09 so we have multi-builds update available to fix unexpected but valid issues with single-build 16:15:12 mhroncok: would you be able to give some priorities to these ( must have, can be done in a second phase) 16:15:42 single build update into a side-tag-based update is probably a good candidate to "nice to have" 16:15:54 for me the last two are nice to have 16:16:07 (thus would like), while the top three are must (thus want) 16:16:08 I want to retest/waive from the UI (and the CLI) - having at least UI or CLI is a must, having both is nice 16:16:56 I would like to be able to follow the progress of test system - that is a very very nice to have - it should probobly not block the thing but it is very bad UX to wait in the dark 16:17:01 mhroncok: would one be in the CLI and the other in UI, be good? 16:17:11 pingou: sufficient 16:17:28 mhroncok: well, you would know what you're waiting on, but not what it's doing 16:17:31 mhroncok: ok 16:18:46 I think this is the type of thing you want to do at the same time anyway ( adding support to the UI and CLI) so that when you develop the change you have the context 16:19:06 would be weird to come back after few weeks to add support to the CLI or UI 16:20:12 cverna: there are sometime challenges around authenticated to other applications via a website :) 16:20:34 for some reason people do not like that we send requests pretending to be them to other websites :] 16:20:48 anyway, technical details to be sorted out 16:21:08 do we want to add anything? 16:21:22 otherwise, we've been at this for over a hour and I think we deserve a break :) 16:21:49 I'll be trying to send a summary of all this tomorrow and see how to adjust the proposal based on these feedback :) 16:21:53 I still have concerns about the actual CI 16:22:23 there was no reply on devel 16:22:27 mhroncok: do you think you could gather them all in an email? 16:22:39 I have 16:22:49 your email to the thread about gating? 16:22:54 yes 16:22:59 * mhroncok looks for a link 16:23:03 I've forwarded it already, I'll ping again asking for their inputs 16:23:22 https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QUDIXHS4EWJFILODUL6JZJOAWCQ5M5L7/ 16:23:29 thanks :) 16:23:35 I think these concerns should be directed to the council, since the CI is a council objective 16:23:52 we should seek council support to address these 16:24:17 let's me try a more diplomatic way first and if there is no response we could consider this :) 16:24:43 pingou: what's the diplomatic aperoach? 16:24:47 *approach 16:24:55 works for me, I saw the council as diplomatic :) 16:25:04 mhroncok: I'm going to email them again asking them to reply :) 16:25:24 I think the problem is that we need more people to work on this (and be responsible for it) - should we e-mail Red Hat instead? 16:25:32 cverna: about as diplomatic as going to your boss asking for your attention :) 16:25:54 pingou: not really because the CI lead is part of the council 16:26:36 mhroncok: I'll try to get more eyes on this email 16:26:38 * cverna considers the council as helper rather than bosses 16:26:39 anyway it would be nice to hear what the "CI people" actually need, if it is people, money, machines, time... 16:26:51 pingou: thnaks 16:27:13 I wonder if time machines would help with the money angle :-p 16:27:23 sorry, disgressing :) 16:27:31 mhroncok: we need to build CI people community to start hearing from "them" i think 16:27:32 should we end here ? 16:27:46 pingou: good for me 16:27:50 bookwar: but how? 16:27:51 thanks pingou 16:27:58 pingou++ 16:28:02 #endmeeting