www.asplund.nu

Comments (35)

Here you'll find too some helping advices, but not so detailed as the howto here! http://www.thomas-krenn.com/de/wiki/Kategorie:Xen Go cluster! Laughing

Hi! I discovered your great project just after starting to build my own xen cluster setup similar to yours. The only difference is that my storage is on a separate storage cluster (two identical servers with software raid + DRBD + LVM + Enterprise iSCSI Target + Heartbeat) and the number of xen dom0 nodes is 9 instead of 2 (11 coputers in total). I choose iSCSI as the storage interface because it is very scalable and a "safe choice" since there are a lot of large companys selling huge hardware iSCSI enabled SAN solutions. So when i grow out of my current homebuilt iSCSI san i can seamlessly migrate to a more powerful solution. I really love XEN becuase it seems to have everything I could possibly dream up. For example XEN supports live migration with iSCSI. Lovely! Anyways if you are interrested here is a little bit of info on how to set up booting your XEN Dom0 and DomUs from iSCSI with Debian Lenny: http://www.etherboot.org/wiki/sanboot/debian_lenny_iscsi I went though wuite a bit having that work in my config and i edited the wiki while workinf so its a bit messy. But it should be complete! PS: The XEN spesific info is at the bottom of the article. Good luck with your project!

when I get to the xm migrate step I receive the following error
Error: /usr/lib64/xen/bin/xc_save 22 1 0 0 1 failed

when I cat /var/log/xen/xend.log I see the same error on the primary and pretty much the same on the secondary
XendError: /usr/lib/xen/bin/xc_restore 16 3 1 2 0 0 0 failed

an xm save returns the same error. Any thoughts on where to look?

Hi!
Thanks for your guide!
I have an issue:
When I take out the lan cable on first node vm's start very well on the second node, but they are starting on the first node too, because he considers himself primary. If I connect the cable i get duplicates of the vm's running on second node.
What can I do to that would first node shutdown or shutdown heartbeat and drbd that I could to start them manual?

Hi.

Super help-full explanation!

I followed article closely, and have only one issue - when I start the heartbeat on both servers, I'm getting the VM running on both machines - and the DRBD switches to both primary mode.

Does anyone has an idea about this - or can provide a 100% working xendomains scripts?

Thanks!

Hi and greetings from Sweden!

I've successfully used the guide above and have it setup and running. However i wonder if anyone have a nice script or workflow to easy add new domU's to the config?

Hälsningar /Johan

Kim, this is covered in the following bug report: https://bugs.launchpad.net/ubuntu/+source/xen-3.2/+bug/216761 Haven't added this in the main HowTo but the link was provided by Frederico above. Probably I wait to update the HowTo until I perform an upgrade to Ubuntu 8.10

first great howto, i did as many other and run apt-get upgrade first and that broke a script, well here is a quick fix

###CUT###
--- xendomains 2008-06-04 21:21:55.000000000 +0200
+++ xendomains.fix 2008-06-04 21:23:06.000000000 +0200
@@ -183,7 +183,7 @@
{
name=`echo "$1" | cut -d\ -f1`
name=${name%% *}
- rest=`echo "$1" | cut cut -d\ -f2-`
+ rest=`echo "$1" | cut -d\ -f2-`
read id mem cpu vcpu state tm

Even though I have listed all modification to the default scripts it might be a good idea to list all scripts in full. I will make a section for that and post it when I get some time over.

Cheers, Daniel

Sure, here's the version I'm using -> http://pastebin.com/m3c9625a0

Thanks for your answer! I think you are right with your guess that something is wrong with the xendomains* script. I found another post where somebody else has exactly the same problem. The problem was that i updated everything before this howto and now the xendomain script is somehow buggy.

May I ask for posting a running xendomains* script or sending it per email (stephanheck'AT'gmx'dot'de)? Then I could test if thats the problem.

Thanks!
Stephan

I'd try looking for clues at /var/log/ha-debug on both nodes, maybe xendomains* is getting something wrong :/

First of all thx for the great howto - and merry chrismas Wink

I just followed your how-to line by line and everything seems to be perfect, but after a little bit more testing I run into a problem:

If I reboot ha1 (with vm test running on it), the vm will be migrated correctly to ha2. But when ha1 comes back up it starts a SECOND vm test and drbd says primary/primary.

Do you have a clue what may be wrong.

Thanks in advance!
Stephan

I think I found another bug.. If you put all your VMs inside one of the /etc/xen/auto/* dirs and leave the other one empty, during failback $NAMES (line 333) becomes empty and the script throws a syntax error (resulting in unintended migration of the VMs, in my case Wink

)
I patched the scripts and wrote to xen-devel about it.

Why not run LVM on DRBD? With this method you need to build a DRBD config for every DomU, if you move LVM up a notch you don't need to worry about that. Did you find your method to be faster? I am currently running LVM on DRBD in production for BlinkMind, http://www.blinkmind.com The only downside I found to running LVM on DRBD is the 4 TB limit.

-Nathan

Oops, it's the same problem, you're right Embarrassed

Anyway I tested the patch yesterday, works nicely.

I believe you are referring to the same problem as above.
But this patch looks very neat and way simpler than mine. Did you try it and can confirm it works?

In case of more than one DomU with XEN_DOMAINS_AUTO, be sure to apply the patch at http://xenbits.xensource.com/xen-unstable.hg?diff/01c8ccb551b0/tools/examples/init.d/xendomains otherwise Heartbeat won't be able to migrate VMs during "xendomains stop", and you'll end up with the VMs on both nodes Crying or Very Sad

Well I tried setting up dopd to prevent splitbrains, but it didn't work as expected.. Maybe I did something wrong, I didn't investigate the matter because I had already spent a lot of time to build the cluster :/ So for the moment I extended deadtime to one minute and wrote notes to *watch out* in case of an unplugged cable.. When there's more time I'd like to try to bind to xendomainsX a script that would check if the actual node is the off-the-net one (by pinging a router maybe) and then shutting down the VMs..

Did you improve you configuration to prevent this state?

This is something I need to look into myself, but I have had no time so far. I'm currently not using fencing like STONITH.

I figured it out (by asking on the drbd ML actually Mr. Green

).. What I had when I reattached the cables was a split-brain situation, that's why drbd couldn't resynchronize Smile

Thanks all the same!

I'm sorry, what's the expected behavior on primary node failure, with your setup? For example, imagine the following:
I have a Samba server on a VM, I pull the network cables from the primary node, and the VM starts on the secondary.. After a while the Samba server comes up again, and I copy a file to it. Then I re-attach the cables.
From my tests, heartbeat shuts the VM down on node2 and restarts it on node1, while DRBD goes in StandAlone mode on both nodes..
What should I do now to prevent losing the file that I copied during node1 failure?
Many thanks for your time, I'm really a HA-newbie Embarrassed

As I understand it, live migration is a maintenance/balancing tool, not a high-availability one.. So everything must be in place for it to work, no pulled cables Smile

Yes manual migration is working fine. Not lively migrated when rebooted and shutdown ha1. I have checked haresources and associated files at resources.d and /etc/default/xendomains, all of them are equipped with --live option.

Can anyone tell me one more thing. Is the automatic live migration of Virtual machine possible from ha1 to ha2 if I pull the network cable immediately from ha1?

Thanks
Paras.

I didn't try with more than one DomU, but I had to apply the patch because of many scripting errors :/

I solved the crashing problem by sheer luck Mr. Green

I changed the NIC (I had a 3Com SOHO100TX, switched with a Realtek RTL-8169) and the problem went away..
Many thanks for your helpfulness Smile

Look in your /etc/defaults/xendomains file(s) and check your line with: XENDOMAINS_MIGRATE=
You need to have --live added to that string, like this:
XENDOMAINS_MIGRATE="ha1 --live"

I see now, when looking at this line of the article using Firefox, it looks like a single long dash "-" instead of two dashes. If a manual live migration is working I guess this is your problem, or maybe your /etc/ha.d/haresources file is pointing to the wrong resource-file.

Instead of live migration, relocating is going on when I reboot the primary node. Paras.

I think I ran that patch on my system as well, but I didn't have any notes to confirm if I did or not. Should add that to my instructions. But that patch is not resolving the issue described above. Did you confirm if you have the same behavior when running more than two DomUs?

For reference, there's a bug report on Launchpad for xendomains: https://bugs.launchpad.net/ubuntu/+source/xen-3.2/+bug/216761

I followed your steps, but that didn't cause my system to crash. When I checked the logs I found out that crossover link didn't go down as I had Wake on LAN enabled. So I did another test where I disconnected the crossover cable just after shutdown of the master node. But I still did not experience a crash. Let me know if you want to try something else.

I'm running Ubuntu 8.04 Server, but the 32bit version.. Could you please try a link failure with your setup?
Steps to reproduce the crash on my system:
- Shutdown -h on the "master" node
- Wait a few minutes
- Panic!

If I ping the vm during the shutdown phase, I get just a 4-seconds gap before the connection is up again (until the kernel crashes, of course) so I think everything is set up correctly.
Many thanks for your help!

I have no issues with kernel panics as described in the bugreport at normal reboots. Have still to do some more extensive testing with heartbeat and effects of certain failures and also I am not sure the DRBD config is optimal.

Are you also running Ubuntu 8.04 Server 64 bit or something else?

Hi! Thanks for your guide, it helped me greatly Mr. Green

Anyway, didn't you have any issue with kernel panics on ethernet link failure? (Due to kernel 2.6.24.. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=399981

Got the link from the ubuntu forums. Thank you very much. Tried it out and it works really nice. Only had one little problem, but that was my fault Smile

Add your comment

Your name:
Your website:
Subject:
Comment:

yvComment v.1.16.0

www.asplund.nu

Add your comment

Main Menu

Latest News

Popular