18:00:02 <nirik> #startmeeting Infrastructure (2015-07-16)
18:00:02 <zodbot> Meeting started Thu Jul 16 18:00:02 2015 UTC.  The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:02 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:02 <nirik> #meetingname infrastructure
18:00:02 <zodbot> The meeting name has been set to 'infrastructure'
18:00:02 <nirik> #topic aloha
18:00:02 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk pbrobinson
18:00:02 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pbrobinson pingou puiterwijk relrod smooge threebean
18:00:03 <nirik> #topic New folks introductions / Apprentice feedback
18:00:08 <nirik> morning all.
18:00:10 <smooge> here
18:00:15 <pcreech|work> morning
18:00:16 * puiterwijk is here
18:00:25 <nirik> any apprentices with questions or new folks who would like to introduce themselves?
18:00:47 <nerdsville> Well I can introduce myself
18:00:54 <nirik> please do. :)
18:01:16 * tflink is here
18:01:30 <nerdsville> My name is Joshua and I am a developer, the past year or so I have been doing PHP development for ecommerce sites and maintaining a web and DB server
18:01:45 <nerdsville> I am 19 years old and am just looking to increase my exposure :) and contribute
18:02:10 <nerdsville> Been programming in many different paradigms since I was young, it's just fun :P
18:02:17 * pingou here
18:02:18 <nirik> nerdsville: welcome!
18:02:27 <nerdsville> nirik: thanks! :)
18:02:32 <pingou> hey nerdsville :)
18:02:34 <nirik> I take it you are interested more in application development side of things?
18:02:39 <nerdsville> pingou: heyyy
18:02:46 * rahulrrixe_ is here
18:02:48 <nerdsville> I am interested in both infra and dev
18:02:49 <nerdsville> :)
18:02:57 <pingou> nerdsville has already made some contributions to election
18:03:04 <dgilmore> hey all
18:03:04 <nirik> excellent!
18:03:08 <pingou> (and I heard might be working on more :-p)
18:03:09 <nirik> morning dgilmore
18:03:22 <nerdsville> Yup I finished part of it, just finishing up the other parts of the revoting
18:03:22 <AadityaN1ir> hey nerdsville.
18:03:23 <nerdsville> :)
18:03:27 <nerdsville> Should be finished soon
18:03:33 <nerdsville> Hey Aaditya :)
18:03:37 <nirik> nerdsville: welcome again and do ask questions as you run into things. :)
18:03:47 <nerdsville> Most definitely
18:03:47 <dgilmore> afternoon nirik
18:03:58 <nirik> any other questions or new folks? if not, moving on to GSoC checkins...
18:04:06 <pcreech|work> just a quick one
18:04:12 <nirik> pcreech|work: fire away
18:04:16 <pcreech|work> when messing with ansible scripts, what do you test on?
18:04:43 <nirik> we don't have any good way to completely test them... you can use --check and --diff
18:04:47 <nirik> and --syntax-check
18:05:05 <pcreech|work> k.  I'll hopefully be starting on the darkserver one soon... been learning ansible
18:05:07 <nirik> typically stuff is commited for staging and run there, then production is added.
18:05:31 <nirik> cool. :) Do ask if you run into any questions... we will try and answer
18:05:37 <nerdsville> If I may chime in
18:05:40 <nerdsville> Ansible is amazing
18:05:41 <nerdsville> :P
18:05:54 <pcreech|work> nirik: will do!
18:06:19 <nirik> we are getting really close to finishing our puppet -> ansible migration. :) Of course the last handfull of hosts are always the hard ones.
18:06:26 <nirik> #topic GSoC student update - kushal
18:06:40 <nirik> kushal: you around? any GSoC folks around who would like to check in on progress?
18:06:46 <pingou> prth: want to give a status update to everyone?
18:07:07 <prth> sure pingou, i've been working on cropping the wallpaper on server side
18:07:30 <pingou> (for nuancier)
18:08:25 <nirik> nice. anyone else have updates?
18:08:31 <sonalkr132> yup
18:08:40 <AnuradhaW> Hi, I have been working on integrating the styles with my askbot instance set up in Openshift. I have also written a blog post about how to integrate them here in my blog: http://anuradhanotes.blogspot.com/2015/07/gsoc-weekly-update-how-to-override.html
18:09:05 <rahulrrixe_> This week I have worked on integerating comments mdoule in the review process. There are some bugs in it and will update soon when it get fixed.
18:09:47 <sonalkr132> I has set up ssh server and gitish shell on vps (thanks for that btw). It worked fine, I will be moving on to implementing api for glittergallery
18:10:11 <sonalkr132> shell will use api for authorization
18:10:59 <nirik> sonalkr132: glad to hear it. ;)
18:11:16 <pingou> rahulrrixe_: do you have a blog? with screenshots? :)
18:12:45 <nirik> I wonder about that... would it be worth seperating out the simple comments? we could reuse them for some other things like copr...
18:13:14 <pingou> having a comment app? :)
18:13:27 <rahulrrixe_> pingou: I haven’t written updates about this week as my thesis presentation is on friend?
18:13:38 <nirik> pingou: well, or a flask module many apps could use?
18:14:07 <nerdsville> I know how to package python apps, if I can be of any help
18:14:12 <rahulrrixe_> pingou: After the git part I was working on review comments.
18:15:00 <pingou> nirik: I wonder if we could do something like this
18:15:23 <pingou> could be cool
18:15:32 <rahulrrixe_> pingou: BTW the last blog link is here https://medium.com/@rahulrrixe/becoming-git-pro-by-getting-into-under-the-hood-417054b3f4aa
18:15:42 <nirik> just a thought, as copr was talking about what to do for user feedback and was looking at disqus. ;(
18:16:06 <nirik> #idea make simple comments flask module for apps to use.
18:16:09 <pingou> yeah, saw this discussion :(
18:16:11 <tflink> I also have some interest in comments but haven't gotten to it
18:16:15 <rahulrrixe_> Sorry thesis presentation is on friday.
18:16:27 <pingou> rahulrrixe_: ok, good luck with this :)
18:16:47 <nerdsville> nirik: I am interested in helping with that
18:16:51 <nirik> Cool. Any more GSoC updates?
18:17:06 <nirik> nerdsville: cool. :) we will need to sort out what we have currently and what we want first
18:17:19 <nerdsville> no prob :)
18:17:41 <nirik> ok, on to announcements/infodump:
18:17:43 <nirik> #topic announcements and information
18:17:43 <nirik> #info Various fixes to people01 after migration - kevin
18:17:44 <nirik> #info Outage template updated - kevin
18:17:44 <nirik> #info lots of cloud instances moved to new cloud - kevin
18:17:44 <nirik> #info lots of no longer needed cloud instances killed in old cloud - kevin
18:17:45 <nirik> #info osbs01.stg setup for releng - kevin
18:17:46 <nirik> #info backups migrated for hosted04->03 and ongoing for collab04->03 - kevin
18:17:48 <nirik> #info inode count increased for backups volume when it hit 100% - kevin
18:17:50 <nirik> #info download01 iDrac fixed - patrick
18:17:52 <nirik> #info migrating floating ips from old to new cloud (backporting fix) - patrick
18:17:54 <nirik> #info Authentication infrastructure upgraded to Ipsilon - patrick
18:17:58 <nirik> #info Koschei is now monitoring f24, a few minor problems fixed - mizdebsk
18:18:00 <nirik> #info Planning to migrate qadevel from old cloud to infra-proper on Friday - tflink
18:18:02 <nirik> #info UMDL re-write (aka umdl2) is getting there - pingou
18:18:04 <nirik> #info pagure work on Fedora 22 - pingou
18:18:06 <nirik> anything in there anyone would like to especially note or talk about?
18:18:13 <nirik> (we have no discussion items listed in gobby)
18:18:16 <smooge> #info PHX2 trip in 2 weeks for physical items
18:18:33 <smooge> sorry I forgot to pu tthat in gobby
18:18:33 <nirik> ah yes, good reminder smooge
18:18:42 <nirik> also, flock is coming up fast.
18:19:09 <nirik> I am hoping we can retire our old cloud by the end of next week, but we will see
18:19:20 <pingou> nirik: how far are we?
18:19:34 <nirik> I killed a ton of instances yesterday. ;)
18:19:40 <pingou> \ó/
18:19:49 <puiterwijk> I do want to point out one thing about the Ipsilon migration: If people report any (external) apps that don't work after the update, have them restart it, as it's likely a stale cache on their end
18:19:49 <nirik> I have about 5-6 to make on new cloud today that we can migrate.
18:19:51 <tflink> nirik: is there any way to hold off for at least a few days?
18:20:12 <nirik> tflink: sure, there's no hard deadline. I just want to get it done. ;)
18:20:28 <tflink> I'm planning to migrate one of our last VMs on friday and while I don't anticipate problems, I'd rather have something to fall back to if something goes horribly wrong
18:20:42 <nirik> next friday? or tomorrow?
18:20:47 <tflink> tomorrow
18:21:04 <nirik> sure. I was hoping for next friday... so that would leave a week?
18:21:14 <tflink> oh, i thought you meant tomorrow
18:21:15 <nirik> there's 33 instances left in the old cloud right now.
18:21:28 <smooge> that is more than are left in puppet correct?
18:22:18 <nirik> puppet currently has 14: https://fedoraproject.org/wiki/Infrastructure/PuppetToAnsibleMigration
18:22:29 <nirik> however, later today I am going to kill 3 more, taking us down to 11.
18:22:38 <pingou> cool!
18:22:48 <nerdsville> Destruction
18:23:04 <nirik> 2 more should also go soon (bapp02/app01), down to 9
18:24:01 <nirik> I am pondering the idea of migrating lockbox01 at that point and just making sure we have good backups of the rest and stopping puppet, but perhaps thats a bad idea, still thinking about it.
18:24:37 <pingou> kinda tempting
18:25:07 <nirik> collab03/hosts-lists01 will go away in favor of mailman01/02 as they finish migrating to mailman3
18:25:17 <smooge> oh rebuilding bodhost01....
18:25:20 <nirik> releng04/relepel01 will go away in favor of bodhi2 stuff
18:25:40 <nerdsville> This is all greek to me lol
18:25:44 <nirik> hosted03 we need to really migrate, but might take a bit with packages.
18:26:00 <nirik> nerdsville: sorry. :)
18:26:05 <nerdsville> lol np
18:26:10 <nirik> anyhow, lets move on to the learn section?
18:26:16 <pingou> nirik: what's with hosted03?
18:26:25 <pingou> ah trac
18:26:27 <pingou> yeah :/
18:26:28 <nirik> pingou: we need to branch all the trac stuff
18:26:38 <nirik> and build it and make a ansible playbook for it.
18:26:44 <nirik> not hard, just time consuming.
18:26:57 <puiterwijk> yeah, I have a lot of packages already branches and built
18:27:10 <puiterwijk> will check the rest and start the playbook soon
18:27:11 <nirik> puiterwijk: oh? cool.... might not be as much as I thought then
18:27:33 <nirik> #topic Learn about: backups with rdiff-backup - kevin
18:27:43 <nirik> ok, I thought I would talk a bit about our backups today.
18:27:57 <nirik> We currently have a backup machine (backup01) in our main datacenter.
18:28:19 <nirik> it uses rdiff-backup to reach out to machines with data we care about on them and backs them up to a local netapp volume
18:28:55 <nirik> it runs backups daily. All machines that are backed up have /etc and /home backed up and many have additional dirs like /srv or the like backed up.
18:29:37 <nirik> We don't currently have off-site backups anoyingly, but we have plans to add that in the 4th Quarter (netapp sync to another datacenter)
18:30:04 <nirik> rdiff-backup is sadly not very active upstream anymore, but it works pretty well overall.
18:30:16 <nerdsville> is it like rsync?
18:30:37 <nirik> nerdsville: it uses librsync yeah, but it can do incrementals
18:30:56 <nirik> so it stores just the changes in each days backup and you can restore as of anytime in the past you have backups for
18:31:09 <nerdsville> ooh
18:31:19 * pingou needs to set it up for himself
18:31:24 <nerdsville> me too lol
18:31:34 <nirik> backup01 runs a cron job for doing the backups.
18:31:47 <jcvicelli> Do we have a continuity plan or something like?
18:31:58 <nirik> It uses ansible actually. It pulls our ansible repo to find out what needs to be backed up and then runs rdiff-backup commands over those hosts.
18:32:12 <nerdsville> nice*
18:32:15 <nerdsville> nice!*
18:32:28 <nirik> http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/playbooks/rdiff-backup.yml is the playbook.
18:32:48 <nirik> jcvicelli: not sure. what do you mean by that? ;)
18:32:53 <nerdsville> what is the git seed stuff
18:33:27 <jcvicelli> A plan for what to do if something bad happens
18:33:29 <nirik> that is a checkout of all our pkgs git repos people can download to have them all.
18:33:35 <nerdsville> ah
18:33:43 <nirik> we don't want to back up that as we back up the git repos.
18:34:16 <nirik> jcvicelli: well it would depend on what that bad thing was. ;)
18:34:41 <nirik> if that datacenter is unreachable there's not much we can do... but after we have offsite backups later this year we could restore from them and bring some things up.
18:34:42 <jcvicelli> Like, if an airplane crash on the datacenter, what steps to follow
18:34:52 <nerdsville> Lol or if y2k2 happened
18:35:08 <smooge> jcvicelli, there isn't much we could do in that case.
18:35:23 <jcvicelli> Got it...
18:35:24 <smooge> we do not have alternate backup centers
18:35:29 <nirik> yeah, not much there. Actually we do have some offsite stuff already
18:35:52 <nirik> all the dl.fedoraproject.org released content and all of koji is mirrored already
18:35:59 <pingou> we have mirrors and clone of the ansible repo :)
18:36:00 <nirik> it's just not the backups space (not enough room yet)
18:36:15 <smooge> yes sorry .. I meant more like "alternate servers to restore to"
18:36:18 <pingou> the DB would be the hardest
18:36:23 <nirik> smooge: yeah, that too.
18:36:36 <nerdsville> Why would the db be hard?
18:37:05 <nirik> well, if we don't have a dump of it?
18:37:06 <pingou> nerdsville: that's probably what would be the hardest to reconstruct if we were to loose the datacenter
18:37:12 <nerdsville> ah lol
18:37:35 <smooge> I would expect that would be a "Hey everyone gets a new account." day
18:37:35 <nerdsville> Get a schema dump at least backed up
18:37:47 <jcvicelli> Yes, so that is the purpose of a bcp plan, so it should be easier to put services back
18:37:50 <pingou> nerdsville: that's upstream :)
18:37:50 <nirik> we have backups, they are just in the same datacenter
18:38:20 <nirik> once we have offsite copy of our backups volume we should be in much better shape.
18:38:34 <smooge> it is just a lot of space needed
18:38:34 <nirik> unless of course _both_ datacenters go away.
18:38:35 <jcvicelli> Sure
18:38:42 <nerdsville> lol amazon? :P
18:38:45 <nirik> there's always a worse case. :)
18:38:49 <smooge> could not afford amazon
18:38:56 <nerdsville> I will donate
18:39:10 <nerdsville> Let's get donations
18:39:11 <nerdsville> :P
18:39:12 <pingou> nerdsville: not enough :)
18:39:53 <nirik> we may also do some more offsite at ibiblio once our new machine there comes on line
18:40:03 <nirik> (just critical stuff)
18:40:25 <nirik> any other backup questions?
18:40:49 <nerdsville> What happens if all internet connectivity is lost! :O
18:40:54 <nirik> we currently have 14TB of backups (but that also includes gnome folks): /fedora_backups   26T   14T   12T  55% /fedora_backups
18:41:04 <nirik> nerdsville: we might have to go out into the big blue room. ;)
18:41:13 <nerdsville> lol
18:41:22 <nirik> #topic Open Floor
18:41:27 <nirik> anyone have anything for open floor?
18:41:50 <nirik> less than a month to flock. Everyone work on their talks/slides. ;)
18:42:06 <adrianr_> about the current fedora-23 repository status
18:42:16 <adrianr_> I can give an update of the current situation
18:42:22 <nirik> adrianr_: sure. please do.
18:42:35 <adrianr_> umdl was not picking up the new release in development/23
18:42:43 <pingou> (is still not)
18:42:51 <adrianr_> that part of the MM2 rewrite has not been required since deployment
18:43:04 <nirik> ok. ;(
18:43:25 <adrianr_> I have added a few fixes to umdl and I am pretty positive this run could be the one to create the repositories correctly
18:43:36 <adrianr_> let's see
18:43:44 <nirik> oh nice. ;)
18:43:47 <pingou> adrianr_: fixed repomap?
18:43:57 <adrianr_> pingou: yes, also
18:44:41 <adrianr_> the problem with fixes in repomap is that it is always hard to tell how much in influenced by change that looks pretty simple
18:45:11 <nirik> yeah.
18:45:25 <adrianr_> and I have prepared a PR for the crawler which supports continent based crawling
18:45:38 <adrianr_> we talked last week about it
18:45:50 * pingou looked at it
18:45:52 <adrianr_> I would say the code is ready
18:45:56 <nirik> ok.
18:46:15 <nirik> this was with the idea that we might make a crawler in or nearer eu and scan eu stuff from there?
18:46:19 <adrianr_> so, if there is the possibility to have a crawler in europe we can try it
18:46:39 <nirik> yeah, I can look into that. :) If you like can you file an infra ticket on it so we don't forget?
18:46:49 <adrianr_> nirik: will do
18:47:11 <nirik> cool.
18:47:40 <nirik> oh, does someone want to teach about something else next week?
18:47:48 <pingou> I added a --start-at to umdl2
18:47:54 <pingou> so you can run it on a part of a tree :)
18:48:11 <nerdsville> what is umdl
18:48:11 <nirik> excellent. also helpfull.
18:48:12 <adrianr_> pingou: that is really helpful
18:48:22 <nirik> nerdsville: it's part of mirrormanager.
18:48:27 <nirik> update master directory list
18:48:30 <nerdsville> ah
18:48:41 <nirik> it runs and looks for repos that we have and if they have changed (we pushed updates, etc)
18:48:57 <nirik> then the crawler runs against mirrors to see if they are up to date with that info
18:49:10 <nerdsville> ah thanks! :)
18:49:38 <Cydrobolt> hey
18:49:40 <Cydrobolt> did I miss the meeting?
18:49:46 <nerdsville> hi
18:49:48 <nirik> Cydrobolt: just about. we are in open floor. ;)
18:49:49 <pingou> Cydrobolt: just finishing up :)
18:49:54 <Cydrobolt> oh, cool!
18:50:48 <nirik> Cydrobolt: you have anything to bring up? :)
18:51:01 <Cydrobolt> nirik, nope!
18:51:08 <Cydrobolt> I pushed an update to mote earlier this week
18:51:11 <Cydrobolt> but nothing important
18:51:22 <nirik> cool.
18:51:30 <Cydrobolt> has anyone reported any issues with it recently? haven't been on IRC as often these past couple of weeks
18:51:56 <nirik> not that I am aware of.
18:52:24 <nirik> anyhow, if nothing else will close out the meeting in a minute...
18:52:29 <AadityaN1ir> I wish to bring something up
18:52:43 <AadityaN1ir> https://fedoraproject.org/wiki/Category:Infrastructure_SOPs
18:52:45 <nirik> AadityaN1ir: ok, fire away
18:53:02 <AadityaN1ir> Many of the links here are broken.
18:53:12 <nirik> yes, because we moved from .txt to .rst
18:53:22 <nirik> if someone could fix those up that would be great. ;)
18:53:36 <AadityaN1ir> i could do that
18:53:39 <smooge> cool
18:53:48 <nirik> that would be lovely. ;) thank you.
18:53:53 <AadityaN1ir> can you tell me where it is hosted ?
18:54:16 <nirik> see the note at the top
18:54:20 <nirik> http://infrastructure.fedoraproject.org/infra/docs/
18:55:07 <AadityaN1ir> ok, I see it.
18:55:12 <AadityaN1ir> Thanks
18:55:27 <nirik> thanks for pointing it out
18:55:32 <nirik> thanks for coming everyone!
18:55:34 <nirik> #endmeeting