Do you need help with with essay, research paper, homework or even dissertation?
Visit our website Where Can I Find Someone to Write My College Paper - https://goo.gl/HKbmHS (assignmenthelp24.com)
=================>>>GET ASSIGNMENT HELP<<<=================
=================>>>GET ASSIGNMENT HELP<<<=================
=================>>>GET ASSIGNMENT HELP<<<=================
dxtr illustration essay
comic essay topics
iisd holiday assignment class 1
article 2314 code civil explication essay
litterbugs essay contest
foxglove manor deed of assignment
roger scruton why beauty matters essay definition
the domain and range of a function homework online
frederick douglass narrative of the life summary essay
northern illinois university admissions essay
arcadia cover letter
mba essays india
replace in shell variable assignment
which person is customarily used in a research paper
jacques ranciere bibliography apa
natalie dessay haendel
essay on old age homes wikipedia dictionary
example of thesis statement for essays
essay examples for Where Can I Find Someone to Write My College Paper middle school
changing face of terrorism essay topics
sample uva supplement essays
breast cancer research paper thesis samples
en1320 research paper
global warming essay titles on pride
example of nursing student cover letter
writing descriptive essay place
ponnala lakshmaiah mla bibliography
gmfrs critical thinking
altpreussische bibliography de mariama ba
free school homework passes
int213 assignment 1 skeletal system
picnic with my family essay examples
latest research papers in plant biotechnology course
ceramium rubrum classification essay
animal cruelty essay conclusion maker
essays on wisdom comes with age
a2 english literature coursework bookshelves
dr essay albany ny
your favourite food essay samples
pgce personal statement secondary physical education
appendix in apa 6 format bibliography
acknowledgement sample for student assignment sheets
ignou pgdt assignment 2011 super
dean s list cornell ilr essay
topics on psychology for a research paper
list of topics for compare and contrast essay example
street crime in pakistan essay in english
philosophy essay prompts for common
abstracts for research papers example
the person i admire the most essay mother
ltc broadening assignments
humanness essay topics
class participation essays
organizational method for process analysis essay
personal essay for medical school
essay topics for persuasion by jane austen
essays on edward lear and nonsense
an artists personal statement ao3 down
best english words for essays
help me with my physical science homework
human genetic engineering essays
good introduction paragraph for a descriptive essay
Where Can I Find Someone to Write My College Paper preschool homework printable
essay about sea in tamil
short essay about the hunger games
sample cover letter for visa application germany
2cb rave experience essay
ipinagbabawal na gamot essay format
quality of friends essay
easy books to write essays on
apple monopoly research paper
good thesis statement comparison essay
peconic bay medical center residency personal statement
how do i find sources for a research paper
overpopulation a problem essay
persuasive essay topics about social media
essay about 2010 world cup
radar source evaluation assignment
ignou ms-03 solved assignment 2014 impala
gcse t - totals maths coursework
cornell graduate school dissertation
cover letter for second job
i graded your homework it was delicious in persian
literary analysis essay on pride and prejudice soundtrack
chateau de chaugy bessay akron
inner city youth violence essay
essay topics for ib exam 2012
animal testing argumentative essay conclusion
tennessee williams free essays
new product strategy at kellogg case study
life is like a rainbow essay
5rwm assignment submission
overcoming bias to learn about controversial topics for essays
comment aborder un sujet de dissertation
wssu admissions essay editing
reformation essay conclusion example
cal state admission essay prompt
georgetown short Where Can I Find Someone to Write My College Paper essay word limit
Where Can I Find Someone to Write My College Paper bibtex example revtex bibliography
true love Where Can I Find Someone to Write My College Paper wait essay about myself
little women essay questions
in an ideal world essay contest
essay background statement
boston college essay college confidential
how to write a good introduction for persuasive essays
thesis compare contrast research paper
research paper works cited example internet
book reference for bibliography
personal essay for job application
essay on making school education stress free
missing values are not allowed in subscripted assignments of data frames
bcsl 013 solved assignment 2014-15 nba
one love song analysis essay
chi square research paper
sample outline for an essay mla
nanako homework persona 4 dancing
ralf zeichner dissertation outline
us senate committee assignments 2013 calendar
art show assignment
masters essay for education
sasbadi spm essay sample
computer assignment for high school students
choosing career path essay
decrescendo music definition essay
french revolution 1789 essays
eco-14 solved assignment 2012-13 champions
cover letter for bill of quantities construction
paul revere biography essay requirements
list of tables and figures dissertation titles
dissertation format sample
whale rider film review essay rubric
essay for texas a&m admission requirements top 25%
objective cover letter examples
best english essay writing
private university vs public university essay writing
puneet rajkumar film names in essays
meat vs vegetarian diet term paper
act essay tips and tricks
research paper on cancer therapy
ann d ballinger scholarship essays
student affairs cover letter
essay effect of war
examples of synopsis for dissertations abstracts
parametric eigenstructure assignment of lease
Linux containers, as a lighter virtualization alternative to virtual machines, are gaining momentum. The High Performance Computing (HPC) community is eyeing Linux containers with interest, hoping that they can provide the isolation and configurability of Virtual Machines, but without the performance penalties.
In this article, I will show a simple example of libvirt-based container configuration in which I assign the container one of the ultra-low latency (usNIC) enabled Ethernet interfaces available in the host. This allows bare-metal performance of HPC applications, but within the confines of a Linux container.
Before we jump into the specific libvirt configuration details, let’s first quickly review the following points:
- What “container” means in the context of this article.
- What limitations exist making it impossible to rely solely on (the available) namespaces to assign host devices to containers and guarantee some kind of isolation.
- What tools can be used to bridge the above-mentioned gaps.
Introduction to Linux Containers
Fun fact: there is no formal definition of a Linux “container.” Most people identify a Linux container with keywords like LXC, libvirt, Docker, namespaces, cgroups, etc.
Some of those keywords identify user space tools used to configure and manage some form of containers (LXC, libvirt, and Docker). Others identify some of the building blocks used to define a container (namespaces and cgroups).
Even in the Linux kernel, there is no definition of a “container.”
However, the kernel does provide a number of features that can be combined to define what many people call a “container.” None of these features are mandatory, and depending on what level of sharing or isolation you need between containers — or between the host and containers — the definition/configuration of a “container” will (or will not) make use of certain features.
In the context of this article, I will focus on assignment of usNIC enabled devices in libvirt-based LXC containers. For simplicity, I will ignore all security-related aspects.
Network namespaces, PCI, and filesystems
Given the relationship between devices and the filesystem, I will focus on filesystem related aspects and ignore the other commonly configured parts of a container, such as CPU, generic devices, etc.
Assigning containers their own view of the filesystem, with different degrees of sharing between host filesystem and container filesystem, is already possible and easy to achieve (see
documentation for namespaces). However, what is still not possible is to partition or virtualize (i.e., make namespace-aware) certain parts of the filesystem.
Filesystem elements such as the virtual filesystems commonly mounted in
are examples that fall into that category. These special filesystems provide a lot of information and configuration knobs that you may not want to share between the host and all containers, or between containers.
Also, a number of device drivers place special files in
that user space can use to interact with the devices via the device driver.
Even though network interfaces do not normally need to add anything to
(i.e., there is no
), usNIC enabled Ethernet interfaces have entries in
because the Libfabric and Verbs libraries require to access those entries.
Sidenote: For more information on why modern Linux distribution do not use interface names like
any more, and how names like
are derived, see this document.
The tools you use to manage containers may assign a new network namespace to each container you create by default, or may need you to explicitly ask for that. Libvirt, as explained here, does that automatically when you assign a host network interface to the container. Specifically: when you create a new network namespace, you have the option of moving into the container any of the network interfaces (e.g.,
) available in the host.
You can do this by hand using the
command, or you can have that assignment taken care for you by one of the container management tools. Later we will see how libvirt does that for us.
Once you have moved a network interface into a container, that network device will be only visible and usable inside that container.
Figure 1: (a) host with no containers (b) container that has been assigned a new network namespace which shares all network interfaces with the host (c) container that has been assigned a new network namespace and one of the host network interfaces (no longer visible in the host)
However, the Ethernet adapter also has an identity as a PCI device. As such, it appears in
and can be seen via commands like
from any network namespace — not only from the one where the associated network device (
This gap derives from the fact that the Ethernet device is hooked to both the PCI layer and the networking layer, but only the latter has been assigned a namespace.
Figure 2: (a) host with no containers (b) container that has been assigned a new network namespace which can not access any of the host network interfaces (c) container that has been assigned a new network namespace and one of the host network interfaces.
Tools you can use to assign devices to containers
You can classify containers based on different criteria, such as based on what they will be used to run inside. At the two extremes, you have these options:
- Application container
- Distribution container
In the first case, you only need to populate the container filesystem with what is strictly needed to run a given application. Most likely, not much more than a set of libraries. Other parts of the filesystem may be shared with the host (including the virtual filesystems), or may not be needed at all.
In the second case, you want to assign the container a full filesystem and have less (if any) sharing with the host filesystem, including the special entries like
Even though full distribution container support is still not considered “ready for prime time” due to the limitations imposed by a few special filesystems as discussed above, there are a number of generic tools available that can be used to provide some kind of device/resource assignments and isolation between containers:
- Security infrastructures like selinux and apparmor
- Bind mounts
- Cgroup device controller (via device whitelists)
You can check LXD for an example of project whose goal is to add whatever is missing in order make containers as isolated as virtual machines in terms of resource usage/access.
In section “Example of libvirt LXC container configuration” we will see a simple example of how you can tell libvirt to use bind mounts and cgroup device controllers to assign a usNIC enabled Ethernet interface to a container.
Support for bind mounts has been available for long time (see
for the details).
cgroup device controller support may already be enabled on your distro by default. But if not, you can enable it with this kernel configuration option:
- General setup
- Control Group support
- Device controller for cgroups
- Control Group support
You can find some documentation about this feature in the kernel file Documentation/cgroups/devices.txt. We will not configure it manually as described in that document; instead, we will tell libvirt to do that for us.
Loading the required kernel modules and understanding the role of key filesystem entries
For a detailed description of how to deploy usNIC you can refer to the usNIC deployment guide (available at cisco.com). Keep in mind that:
- The installation of the kernel modules is only needed in the host (not the container).
- In the container filesystem, you only need to install the user space libraries and packages.
The only missing point, which is the focus of this article, is to make sure certain files created by step 1) will be visible and usable inside the container’s filesystem.
Normally, users do not need to have a detailed knowledge of what files are created by the kernel modules and used by the user space libraries. In our case, however, we do need to have some knowledge about these files in order to properly populate the container filesystem.
Before I show you the libvirt XML configuration, let’s first discuss the role of three key file/directories we will need to tell libvirt about.
Once you have created a “Virtual NIC” (vNIC) on the Cisco UCS Virtual Interface Card (VIC) and enabled the usNIC feature in it (per the Cisco documentation cited above), you will see the following three filesystem entries in the host:
This is a character device used by the user space library to configure a usNIC enabled network interface.
This is a directory used by the usNIC kernel driver to export a number of configuration parameters. For example, the
file in this directory tells you with which network interface (visible with
) this usNIC entry is associated to.
Among the data exported here by the Linux Verbs API, you may find useful these two files:
This is the
device ID which will match with what you will see in
. You can refer back to this information when/if you want to check if libvirt configures the cgroup device whitelist properly (see example, below).
This is the associated usnic_X entry in
directory will be populated when you load the usNIC kernel driver module (i.e.,
directories also will be populated when you load the usNIC kernel driver module.
In order to find the mapping between one of the network interfaces visible with ifconfig and the associated
, you can either use the files in
described above, or use the
command that comes with the
Example of libvirt LXC container configuration
Libvirt describes the configuration of containers (as well as virtual machines) with an XML file. Here is a link to detailed documentation of all libvirt’s XML options. In the context of this article, I recommend reading the following sections of that documentation:
- Filesystem mounts
- Device nodes
- Filesystem isolation
- Device access
Let’s start with a simple container configuration and add the delta needed to assign one usNIC enabled host Ethernet interface to the container. This example shows how to create a container on a Cisco UCS C240-M3 rack server running Centos 7.
Here is a stripped-down version of the container XML; I have removed the details that are not relevant for this discussion:
<domain type='lxc'> <name>container_1</name> <memory unit='GiB'>8</memory> <currentMemory unit='GiB'>0</currentMemory> <os> <type arch='x86_64'>exe</type> <init>/sbin/init</init> </os> <devices> <filesystem type='mount' accessmode='passthrough'> <source dir='/usr/local/var/lib/lxc/container_1/rootfs'/> <target dir='/'/> </filesystem> <console type='pty'/> </devices> </domain>
The only detail worth noting is that the container root filesystem is located at
in the host.
Note that with this basic configuration, and according to the section “Device Nodes” mentioned above, the container’s
tree will not contain any of the special entries from the host’s
tree, including the
directory that we need for usNIC:
[container_1]# ls /dev/infiniband ls: cannot access /dev/infiniband: No such file or directory
is shared with the host, you can see the entries associated to usNIC enabled Ethernet interfaces:
[container_1]# find /sys/class -name uverbs* /sys/class/infiniband_verbs/uverbs0 /sys/class/infiniband_verbs/uverbs1 /sys/class/infiniband_verbs/uverbs2 /sys/class/infiniband_verbs/uverbs3 [container_1]# find /sys/class -name usnic* /sys/class/infiniband/usnic_0 /sys/class/infiniband/usnic_1 /sys/class/infiniband/usnic_2 /sys/class/infiniband/usnic_3
But notice that none of the
devices are present (yet) in the container. Running a simple usNIC diagnostic program in the container shows warnings (one for each device I have configured on my server):
[container_1]# /opt/cisco/usnic/bin/usd_devinfo usd_open_for_attrs: No such device usd_open_for_attrs: No such device usd_open_for_attrs: No such device usd_open_for_attrs: No such device
Since we did not assign any host network interface to the container, by default, libvirt allowed the container to see all Ethernet interfaces (i.e., it did not create a new network namespace):
[container_1]# ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 8: enp6s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 00:25:b5:00:00:04 brd ff:ff:ff:ff:ff:ff 9: enp7s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 00:25:b5:00:00:14 brd ff:ff:ff:ff:ff:ff 10: enp8s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 00:25:b5:00:00:24 brd ff:ff:ff:ff:ff:ff 11: enp9s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 00:25:b5:01:01:0f brd ff:ff:ff:ff:ff:ff
Now we edit the libvirt configuration to assign one usNIC enabled interface to the container. This means that inside the container:
will show an entry for the assigned usNIC enabled interface
will also show the usNIC enabled Ethernet interface .
) to the container. Here is the new libvirt LXC container configuration (the changes compared to
are shown in red):
<domain type='lxc'> <name>container_2</name> <memory unit='GiB'>8</memory> <currentMemory unit='GiB'>0</currentMemory> <os> <type arch='x86_64'>exe</type> <init>/sbin/init</init> </os> <devices> <filesystem type='mount' accessmode='passthrough'> <source dir='/usr/local/var/lib/lxc/centos_container/rootfs'/> <target dir='/'/> </filesystem> <hostdev mode='capabilities' type='misc'> <source> <char>/dev/infiniband/uverbs1</char> </source> </hostdev> <hostdev mode='capabilities' type='net'> <source> <interface>enp7s0f0</interface> </source> </hostdev> <console type='pty'/> </devices> </domain>
You can find more details about the above two new pieces of configuration here.
If I start the container with the new “container_2” configuration, this is what I can see now from within it:
- Only one network interface (
- The device node
- The same four entries in
(as with the previous configuration
[container_2]# ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 9: enp7s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 00:25:b5:00:00:14 brd ff:ff:ff:ff:ff:ff [container_2]# ls -ls /dev/infiniband/ total 0 0 crwx------. 1 root root 231, 193 Apr 1 20:44 uverbs1 [container_2]# find /sys/class -name uverbs* /sys/class/infiniband_verbs/uverbs0 /sys/class/infiniband_verbs/uverbs1 /sys/class/infiniband_verbs/uverbs2 /sys/class/infiniband_verbs/uverbs3 [container_2]# find /sys/class -name usnic* /sys/class/infiniband/usnic_0 /sys/class/infiniband/usnic_1 /sys/class/infiniband/usnic_2 /sys/class/infiniband/usnic_3
Here is how the usNIC diagnostic command
shows the information about the visible usNIC enabled network interfaces (there are still some warnings because of the
entries that are present in
but not in
[container_2]# /opt/cisco/usnic/bin/usd_devinfo usd_open_for_attrs: No such device usnic_1: Interface: enp7s0f0 MAC Address: 00:25:b5:00:00:14 IP Address: 10.0.7.1 Netmask: 255.255.255.0 Prefix len: 24 MTU: 9000 Link State: UP Bandwidth: 10 Gb/s Device ID: UCSB-PCIE-CSC-02 [VIC 1225] [0x0085] Firmware: 2.2(2.5) VFs: 64 CQ per VF: 6 QP per VF: 6 Max CQ: 256 Max CQ Entries: 65535 Max QP: 384 Max Send Credits: 4095 Max Recv Credits: 4095 Capabilities: CQ sharing: yes PIO Sends: no usd_open_for_attrs: No such device usd_open_for_attrs: No such device
Let’s compare the content of
in the host and in the container:
[container_2]# ls -ls /dev/infiniband/ total 0 0 crwx------. 1 root root 231, 193 Apr 1 20:44 uverbs1
[host]# ls -ls /dev/infiniband/ total 0 0 crw-rw-rw-. 1 root root 231, 192 Mar 31 17:30 uverbs0 0 crw-rw-rw-. 1 root root 231, 193 Mar 31 17:30 uverbs1 0 crw-rw-rw-. 1 root root 231, 194 Mar 31 17:30 uverbs2 0 crw-rw-rw-. 1 root root 231, 195 Mar 31 17:30 uverbs3
As you can see,
— and only
— is visible in the container. The device major number for all
entries is 231, while the device minors are 192/193/194/195.
Let’s now compare the
device whitelist for the container and for the host:
[container_2]# cat /sys/fs/cgroup/devices/devices.list c 1:3 rwm c 1:5 rwm c 1:7 rwm c 1:8 rwm c 1:9 rwm c 5:0 rwm c 5:2 rwm c 10:229 rwm c 231:193 rwm c 136:* rwm
[host]# cat /sys/fs/cgroup/devices/devices.list a *:* rwm
As you can see from the two commands above:
- The hostdev/misc entry in the libvirt XML config added the 231:193 rule to the container device whitelist
- The rest of the devices are the default ones added by libvirt
We can see that “ping” works just fine from inside the container (using the enp7s0f0 interface):
[container_2]# ip addr show dev enp7s0f0 9: enp7s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000 link/ether 00:25:b5:00:00:14 brd ff:ff:ff:ff:ff:ff inet 10.0.7.1/24 brd 10.0.7.255 scope global enp7s0f0 valid_lft forever preferred_lft forever inet6 fe80::225:b5ff:fe00:14/64 scope link valid_lft forever preferred_lft forever [container_2]# ping -c 1 10.0.7.2 PING 10.0.7.2 (10.0.7.2) 56(84) bytes of data. 64 bytes from 10.0.7.2: icmp_seq=1 ttl=64 time=0.279 ms --- 10.0.7.2 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.279/0.279/0.279/0.000 ms
We can test the
interface using the
command to another container, similarly configured with usNIC enabled interface on another Cisco UCS C240-M3 rack server connected on a regular IP/Ethernet network:
[container_2]# /opt/cisco/usnic/bin/usd_pingpong -d usnic_1 -h 10.0.7.2 open usnic_1 OK, IP=10.0.7.1 QP create OK, addr -h 10.0.7.1 -p 3333 sending params... payload_size=4, pkt_size=46 posted 63 RX buffers, size=64 (4) 100000 pkts, 1.790 us / HRT
The 1.79 microsecond half-round trip ping-pong time (show in red, above) shows that we are getting bare-metal performance inside of the container.
As Linux containers become more mainstream — potentially even in HPC — it will become more important to understand how to expose native hardware functionality properly. Documentation and “best practice” knowledge is still somewhat scarce in the rapidly-evolving Linux containers ecosystem; this blog entry explains some of the underlying concepts and shows some examples of how adding just a few lines of XML allows bare-metal performance with the isolation and configurability of Linux containers.Tags: