Fail to run benchmark with runspecvirt.sh

[lroderic]

I don’t know if this is the solution, but you’re running Java JRE on the VMs: “Java SE Runtime”.  Please update to Java JDK and retry.

Lisa

[Tang]

Hi,Lisa

I installed java with jdk package 'jdk-8u112-linux-x64.rpm' downloaded on oracle.com.
I think java jdk is installed correctly and no error printed during installation.
Then I updated java to latest version with package 'jdk-8u112-linux-x64.rpm' and run again, but the result is the same.

thanks,

Tang

[lroderic]

When you do a java -version, do you see JDK? Did you set $JAVA_HOME correctly in ~/.bash_profile? What happens when you log out then log back in again and:

Code: [Select]

echo $JAVA_HOME
ls -l $JAVA_HOME

I'd also like to know more about the vNICs and your bridges. On the host, please issue and post:

Code: [Select]

ifconfig
ls /etc/libvirt/qemu/networks

According to ifconfig on the guest, you're using the same bridge for both internal and external network, and that won't work. You need one external vNIC on the VM that connects to an external bridge to communicate with the client (such as 10.10.xx.xx, 255.255.254.0) and one internal vNIC to communicate between the VMs (such as 192.168.xx.xx, 255.255.0.0).

Here's a sample of a VM definition where the external vNIC is connected to host bridge br6 (bridged to eth1 on the host) and the internal vNIC is using the default virbr1 that libvirtd creates:

Code: [Select]

<interface type='bridge'>
      <mac address='52:54:00:dd:ca:8b'/>
      <source bridge='br6'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
 <interface type='network'>
      <mac address='52:54:00:6f:b7:c6'/>
      <source network='virbr1'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

See https://www.cyberciti.biz/faq/rhel-linux-kvm-virtualization-bridged-networking-with-libvirt for details on setting up bridges for KVM.

Lisa

[Tang]

Hi, Lisa

In ~/.bash_profile, I've added JAVA_HOME to the end. Follow is ~/.bash_profile

Code: [Select]

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

JAVA_HOME=/usr/java/latest
export JAVA_HOME
PATH=$JAVA_HOME/bin:$PATH
export PATH

Follow is $JAVA_HOME.
Issue JAVA_HOME:

Code: [Select]

[root@client1 ~]# echo $JAVA_HOME
/usr/java/latest
[root@client1 ~]# ls -l $JAVA_HOME
total 0
lrwxrwxrwx 1 root root 8 Mar 15 20:35 bin -> /usr/bin
I don't see JDK in java -version, but I can see jdk1.8.0_171-amd64 that I newly install in /usr/java.
Googling that, in the second line of java -version, it may be something like
"OpenJDK Runtime Environment xxx" or "Java(TM) SE Runtime Environment xxx".
I remember that before updated java to 1.8 version in vm, it started with OpenJDK.
Do you think I should use OpenJDK?

Follow is some other information about my java.

Code: [Select]

[root@client1 ~]# which java
/usr/java/latest/bin/java
[root@client1 ~]# java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
[root@client1 opt]# ls /usr/java/
default  jdk1.8.0_112  jdk1.8.0_171-amd64  latest
About the network, on host, I've set up two bridges, br3 and br4.
Br3 for external network for vms and client.
Br4 for internal network for vms. (I also add client to this bridge for debug reason, I'm not sure if it will cause any problems?)

host   br3     192.1.1.254
infra  tap1-1  192.1.1.1
web    tap1-2  192.1.1.2
mail   tap1-3  192.1.1.3
app    tap1-4  192.1.1.4
db     tap1-5  192.1.1.5
batch  tap1-6  192.1.1.6
client tap1-7  192.1.1.7

host   br4       192.2.1.254
infra  tap1-1-i  192.2.1.1
web    tap1-2-i  192.2.1.2
app    tap1-4-i  192.2.1.4
db     tap1-5-i  192.2.1.5
client tap1-7-i  192.2.1.7

I set up bridges and taps on host with br_tap.sh(attached below).
'ifconfig' output is in ifconfig.txt(attached below).
Actually I use qemu directly to start and config vms, not with libvirt.
In vms, ip connection is ok via 192.1.1.x or 192.2.1.x.
All vm start command is attached below.

thanks,

Tang

[lroderic]

The client doesn't use the internal network. Only appserver, dbserver, webserver, and infraserver use the internal network. So remove the second vNIC from the client and edit its /etc/hosts to remove references to internal addresses. One bridge works for a start. You can add bridges later as you add tiles.

I still think the problem is that you have two bridges on the same network (192.xx), so the vNICs are on the same network (192.xx) with the same NETMASK (255.255.255.0). This is why ping works between them but the test hangs. But for our environment, the -int hostnames need to map to an internal-only VM network with a different NETMASK (for example, 255.255.0.0).

Here's a sample VM definition for the network pieces where the internal network is labeled "default" (supplied with libvirtd):

Code: [Select]

   
   <interface type='network'>
      <mac address='52:54:00:f2:1d:a7'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:19:9e:f3'/>
      <source bridge='br1'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>

Please copy/paste the VM XML definition for appserver and dbserver.

Also, do you have 127.0.0.1 in /etc/hosts? Either remove it entirely or make sure it only maps to locahost:

Code: [Select]

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
Can you run nslookup appserver manually on the dbserver VM?

Lisa

[Tang]

Hi, Lisa

In physical server, br3 is for externel network. br4 is for internel network.
All externel vNICs of vms are added to br3, and not added to br4.
Internel vNICs of app,db,web,infraserver are added to br4, and not added to br3.
I think I can't just configue only one bridge. Without br3, vms can't connect through externel network. Without br4, vms
can't connect through internel network.

What do you think if I configue total different network in br3 and br4?

for example:
externel network 192.x.x.x netmask 255.0.0.0
internel network 193.x.x.x netmask 255.0.0.0

Also, I've double checked /etc/hosts in all vms, and 127.0.0.1 is mapped to localhost.
First line "client1" is different in each vm.

/etc/hosts in client(I have not rerun yet):

Code: [Select]

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 client1
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.1.1.1          infraserver1 infraserver1-ext infraserver
192.1.1.2          webserver1 webserver1-ext webserver
192.1.1.3          mailserver1 mailserver1-ext mailserver
192.1.1.4          appserver1 appserver1-ext specdelivery specemulator appserver
192.1.1.5          dbserver1  dbserver1-ext specdb dbserver
192.1.1.6          batchserver1 batchserver1-ext batchserver

192.2.1.1          infraserver1-int besim infraserver-int
192.2.1.2          webserver1-int webserver-int
192.2.1.3          mailserver1-int mailserver-int
192.2.1.4          appserver1-int appserver-int
192.2.1.5          dbserver1-int dbserver-int
192.2.1.6          batchserver1-int batchserver-int

192.1.1.7   specclient1 client1 specdriver specclient primeclient1
192.1.2.7   specclient2 client2
192.1.3.7   specclient3 client3
192.1.4.7   specclient4 client4
192.1.5.7   specclient5 client5
192.1.6.7   specclient6 client6
192.1.7.7   specclient7 client7
192.1.8.7   specclient8 client8
192.1.9.7   specclient9 client9
thank you so much

Tang

[lroderic]

> Also, I've double checked /etc/hosts in all vms, and 127.0.0.1 is mapped to localhost.
> First line "client1" is different in each vm.

Please remove "client1" and any VM names you have in every /etc/hosts. For all VMs, the first line needs to be the following and only the following:

Code: [Select]

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
> externel network 192.x.x.x netmask 255.0.0.0
> internel network 193.x.x.x netmask 255.0.0.0

These are still the same network. You need entirely different NETMASKs for this to work on the app/dbserver and web/infraserver VMs. The internal network doesn't map to a physical bridge or host NIC. In the User Guide:

Code: [Select]

1.2.2 VM creation and network configuration

Once your virtual machine is created, install your preferred operating system in the newly created virtual machine. Configure a network interface to be connected to an external network over which the clients and VMs communicate. We recommend configuring another virtual network interface on the appserver VMs and webserver VMs to be connected to an internal network that the VM can use to communicate with the dbserver and infraserver VMs respectively. Since the communications are intra-SUT, this internal network does not have to be associated with a physical network port. For example, on the VM:

cat /etc/sysconfig/network-scripts/ifcfg-eth0
#public IP to/from clients
NAME=eth0
ONBOOT=yes
BOOTPROTO=static
NETMASK=255.255.0.0
IPADDR=172.168.122.64

cat /etc/sysconfig/network-scripts/ifcfg-eth1
#private intra-SUT IP communicates over Virtual NIC to other VMs only
NAME=eth1
ONBOOT=yes
BOOTPROTO=static
NETMASK=255.255.255.0
IPADDR=10.0.0.64

On the host you need a setup such as: https://vpsboard.com/threads/kvm-xen-internal-private-network.71/ and https://serverfault.com/questions/725795/vm-vm-only-network-with-kvm: "...as simple as creating another bridge "bridge1" and not attaching a physical interface AND not assigning an ip address".

Or you might consider using libvirt since the virt-manager GUI lets you set up the internal network quickly and easily.

Lisa

[Tang]

Thank you all, i have solved this problem.
The network between vm servers is not stable because i did not set mac-addr when i started these vms.
The mac-addr conflict might result in unstable network, finnally resulted in this problem.
After i configed mac-addr of each vm and reran, this problem is solved.

[dongqing]

hello tang ,
i am also running specvirt on arm64 , Can you give me your email? I got some questions about it