13:00:44 #startmeeting meeting 13:00:44 Meeting started Mon May 8 13:00:44 2017 UTC. The chair is mvollmer. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:00:44 Useful Commands: #action #agreed #halp #info #idea #link #topic. 13:00:44 The meeting name has been set to 'meeting' 13:00:49 .hello mvo 13:00:50 mvollmer: mvo 'Marius Vollmer' 13:01:14 .hello dperpeet 13:01:15 dperpeet: dperpeet 'None' 13:01:19 .hello martinpitt 13:01:20 pitti: martinpitt 'Martin Pitt' 13:01:43 .hello andreasn 13:01:44 andreasn: andreasn 'Andreas Nilsson' 13:02:14 .hello garrett 13:02:16 garrett: garrett 'Garrett LeSage' 13:02:42 #topic Agenda 13:03:29 * image pruning on servers / distributed storage 13:04:16 * application manager, next steps 13:05:33 okay, let's start 13:05:40 #topic image pruning on servers / distributed storage 13:05:52 ok, so we've run out of space on fedorapeople (again) 13:06:08 since we carry more than one branch now, our pruning has become a bit more complicated 13:06:21 by pruning I mean the process of cleaning up unused images 13:06:27 for our integration testing 13:06:56 we've recently patched our vm-download process to be more careful when pruning (by considering branches other than master) 13:07:08 right now I'm working on a pr to do the same for our vm-upload pruning 13:07:31 I'm implementing a staggered approach, whereas right now we delete images that aren't used and are older than 30 days 13:07:51 i've been working on distributing storage 13:07:56 once we look at all pull requests and branches, we can be a lot more aggressive and don't need to wait for 30 days 13:07:57 today i've been working on bittorrent for that reason 13:08:06 in addition i've provisioned 300G of storage on verifymachine5 13:08:06 over to stef 13:08:17 and added an http server to the standard cockpit/test stuff 13:08:23 so any cockpit-test instance can share its images 13:08:35 is that storage persistent? 13:08:38 so hopefully between all these things we'll solve the problem 13:08:40 it is persistent 13:08:43 nice 13:08:51 but i think it could have an outage and be gone at some point 13:08:57 hence my work on bittorrent 13:09:07 where basically all the cockpit/tests containers talk to each other 13:09:10 and just plain share images 13:09:19 a bit longer term, but i think that's the end goal 13:09:23 yes 13:09:30 the images have been one painful part where we haven't been properly distributed and scaleable 13:09:32 and that would change that 13:10:03 FWIW, when you look at the effects of the tasks that are done by these automated bots 13:10:09 we can iron out the technical details (tracker et cetera), but I like the direction 13:10:18 dperpeet, ++ 13:10:27 DHT vs tracker ... and behind firewall nodes 13:10:31 trending toward zero configuration 13:10:45 maybe just one or two more well know addresses of nodes in order to bootstrap things 13:10:52 lets talk about that more 13:10:53 we may also want to scale logs 13:11:02 at the same time 13:11:06 if possible 13:11:08 i worked a bit on that 13:11:14 and adding a cockpit/sink container 13:11:20 which can be run in multiple places 13:11:29 technically we have scaling going on there 13:11:34 it's just not in use 13:11:39 everything gravitated towards fedorapeople.org 13:11:44 but yes, i think we can find a solution there too 13:12:20 on a meta level: So one of the things that's interesting here 13:12:26 even if we do have backups, the one place that's a bit "static" is github 13:12:35 yeah, but that mirrors our project 13:12:39 yup 13:12:47 we really do depend on these bots as team members 13:12:53 and the fact that they're organic and self-organizing is important 13:13:23 when you get away from the implementation details 13:13:27 of how the bots work 13:13:34 and look at the effect of what they pull off it's pretty amazing 13:13:44 yes, I agree 13:13:48 from the massive coordination of bots around testing a pull request, to the way they submit changes for review, and then other bots come in and check it out. 13:13:54 to the way they do releases 13:14:16 anyway ... a bit of a side track there ... but it's pretty amazing 13:14:31 cockpituous is our #7 contributor on github :) 13:14:38 yeah, i'm not surprised 13:15:25 is there any part you think we should discuss now? 13:15:32 i don't think so 13:15:43 I think we can keep chipping away at this piece by piece 13:15:44 as we have been 13:15:47 i'd like to talk about this with whoever is interested 13:15:55 but yeah, agree on the direction, and then chip away on it bit by bit 13:15:55 I am obviously 13:16:15 i also think i know why others can't just take our bots and duplicate what they do, use them elsewhere 13:16:32 one thing to take away for contributors: if a PR becomes too old, it may take less time in the future for images to get deleted 13:16:34 i think we want to make them easier, but i've figured out why that's really hard at a meta level 13:16:46 do tell! 13:18:05 who should tell? 13:18:26 stefw, you said " but i've figured out why that's really hard at a meta level" 13:18:36 ok ... but don't laugh 13:18:40 kind of a cliff-hanger, I'm waiting here myself 13:18:41 take a step away from the implementation for a second 13:18:48 we've essentially built simple bot team members 13:18:58 mvollmer built parts of that when doing the image creation PR+images stuff 13:19:03 we built them a bit self-organizing 13:19:08 and helped them to grow over time 13:19:12 even if someone copies them as is 13:19:18 they're sorta copying something that's alive and part of the team 13:19:25 even if they manage to copy it as is, the two will diverge 13:19:33 more likely they can copy parts of the components of such a bot 13:19:37 but copying it wholesale is really hard 13:19:43 sorta like cloning life forms is hard 13:20:01 well, that's because what we call the "bot" is actually a bunch of stuff spread all over 13:20:10 we've distilled a lot into cockpituous 13:20:12 yeah, most of the testing framework feels like that to me 13:20:16 but some core logic still remains in cockpit 13:20:28 if you step away from the implementation ... and try and explain this even to technical people not familiar with the details 13:20:33 how we rely on these bots, and what they do 13:20:37 you end up coming to this conclusion 13:20:46 without them the team stops, we rely on them to do work 13:20:47 etc. 13:21:03 so i am cleaning up parts of their implementation 13:21:18 it's the distilled wisdom and best practice of many man-years, no wonder it's hard to explain in an elevator pitch :) 13:21:21 cleaning up round #X :) 13:21:42 I think we got a cleanup every time the team grew 13:21:52 one of their big amazing aspects is that each of us humans on the team can modify the bots 13:21:54 try some change out 13:22:03 and then if it works, that change goes to them all 13:22:10 i wouldn't say our bots are super sophisticated, they are just super specific 13:22:18 but what they do is sophiscticated 13:22:20 the implementation is not 13:22:21 "devops" 13:22:28 but that fundamental principle allows them to grow properly 13:22:33 the Atomic guys have similar bots, with different sophistication 13:22:39 and at the same time makes them really hard for someone else to take and copy and bring over somewhere else 13:22:41 mvollmer, exactly 13:22:57 and i'm coming to grips with the fact that i don't think that's a flaw necessarily 13:23:08 we could say our bots are ..... proprietary 13:23:14 so are you 13:23:18 you are proprietory 13:23:19 ary 13:23:35 this discussion has become a bit metaphysical 13:23:35 their componenst are not, but the bots as composed are really just a part of the project 13:23:48 yup 13:24:50 anyway, when people say automation is even taking over software engineering ... this is exactly what they mean ... it may look tedious from the details ... but the effect of what's happening in our project is exactly that 13:24:59 essentially, we can't generalize them too much for others, because the specialization/generalization is something that didn't happen by accident 13:25:04 yeah 13:25:19 anyway ... that was fun :) 13:25:30 cup of robots! 13:25:58 * mvollmer pokes the robot to change the topic 13:26:12 #topic application manager, next steps 13:26:38 so, I think I have the basic pieces in place 13:26:44 and have to decide what to work on next 13:27:01 let me dig out the video link... 13:27:10 #link https://www.youtube.com/watch?v=Pmi6AJanRtk 13:27:15 thanks 13:27:24 those are the baosc pieces 13:27:40 really nice demo! 13:27:43 i think we could release a package with metainfo to Fedora and have it all work 13:27:54 but there is lots to optimize 13:28:07 and the spec needs to be officially updated 13:28:11 the appstream spec 13:28:29 and we need to make dynamic manifests work 13:28:38 I have some ideas on how the design can be improved in some places, but this is a great start! 13:28:42 I think I'll try to do all this in parallel 13:28:56 mvollmer: so that "Demo application" is just a normal RPM with the extra appstream metadata? 13:29:05 pitti, yes 13:29:15 nice! (I haven't watched it to the end yet) 13:29:37 i need to figure out how Debian handles appstream metadata 13:30:13 and talk to richard and kalev and maybe get a server version of the fedora appstream meta data 13:30:29 otherwise we'll have to install all of the GNOME and KDE meta data on the server 13:30:50 but if we do that, things should just work 13:31:43 one more track to work on: the actual FreeIPA application 13:31:49 mvollmer: so there's an appstream metadata "app type" thing that says "cockpit"? 13:32:08 (or "gnome", "kde", or other categories) 13:32:09 I think once we have that in shape, we can consider actually releasing the Application Manager 13:32:30 pitti, right now I am using the new "launchable" meta data 13:32:52 any package that has a "systemd-unit" or "cockpit-package" launchable will be shown by Cockpit 13:33:14 the system-unit launchable needs a new component type "service" 13:33:36 cockpit-package launchable goes with a "addon" component 13:33:44 that cockpit 13:34:05 needs to be discussed and formally specced with the appstream people 13:34:21 but the basic ideas have been agreed upon 13:35:43 pitti, on fedora, the appstream metadata comes in a RPM itself: "appstream-data" 13:35:51 pitti, do you know how that works on Debian? 13:36:07 i think apt-get might download it as part of the repo metadata, actually 13:36:13 anyway, I'll figure that out 13:36:33 mvollmer: not precisely; I thought it was encoded in the Packages.gz metadata somehow 13:36:34 fedora needs to be improved, likely. Not just for us, but in general 13:36:49 http://dep.debian.net/deps/dep11/ 13:36:58 pitti, they have a yaml file next to packages.gz, I think 13:37:06 question is, who downloads it? 13:37:14 but, I'll get there 13:37:31 I thought e. g. gnome-software would 13:37:36 not apt itself 13:37:40 maybe 13:37:55 maybe apt helps to construct the urls 13:38:05 but let's not guess 13:38:38 http://appstream.ubuntu.com/ too 13:39:24 but these are just some pointers I know about, TBH I've never looked into the details 13:40:56 Thanks! 13:41:16 okay,done? 13:41:45 think so 13:42:30 okay! 13:42:33 #endmeeting