HowTo Headless Intel NUC vPro AMT

Background

We purchased a Intel NUC (NUC7i7DNHE to be precise) with Intel's vPro AMT support in July 2018 to act as a new headless build machine when doing NST software development. This page describes some of findings related to setting up the Intel NUC as a headless server running NST 28. Many of these findings will apply to other Linux distributions especially those that are based on Fedora.

Finding Intel AMT Enabled Systems

The following nmap command can be used to scan a single host for Intel AMT related ports:

[pkb@refritos ~]$ nmap -Pn -p 623,664,5900,9971,16992-16995 192.168.7.57
Starting Nmap 7.70SVN ( https://nmap.org ) at 2018-07-23 16:21 EDT
Nmap scan report for beans.attlocal.net (192.168.7.57)
Host is up (0.0021s latency).

PORT      STATE  SERVICE
623/tcp   open   oob-ws-http
664/tcp   closed secure-aux-bus
5900/tcp  open   vnc
9971/tcp  closed unknown
16992/tcp open   amt-soap-http
16993/tcp closed amt-soap-https
16994/tcp open   amt-redir-tcp
16995/tcp closed amt-redir-tls

Nmap done: 1 IP address (1 host up) scanned in 6.54 seconds
[pkb@refritos ~]$ 

This scan can take a long time if you do an entire network. If you have enabled your AMT systems to respond to ICMP pings, you can drop the -Pn option to try and speed up the scan.

Unable to Scan Self

This may be obvious, but just to be clear, you can not scan yourself. AMT will only allow connections from external machines. If you try to scan yourself, nmap will show all ports as closed as demonstrated below (after logging into the 192.168.7.57) machine:

[pkb@beans ~]$ nmap -p 623,664,5900,9971,16992-16995 192.168.7.57
Starting Nmap 7.70SVN ( https://nmap.org ) at 2018-07-24 00:25 EDT
Nmap scan report for beans.attlocal.net (192.168.7.57)
Host is up (0.000038s latency).

PORT      STATE  SERVICE
623/tcp   closed oob-ws-http
664/tcp   closed secure-aux-bus
5900/tcp  closed vnc
9971/tcp  closed unknown
16992/tcp closed amt-soap-http
16993/tcp closed amt-soap-https
16994/tcp closed amt-redir-tcp
16995/tcp closed amt-redir-tls 

Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds
[pkb@beans ~]$ 

While this makes sense (AMT is designed to manage servers out of band), it can sometimes be easy to forget.

Connecting

If the scan shows that port 16992 or 16993 is open, you should be to open up a connection to the simple AMT web interface using a web browser. For example, if in the previous scan, 192.168.7.57 showed that port 16992 was open. This indicates that a HTTP connection (http://192.168.7.57:16992/) should work. However, since port 16993 was shown as closed, we would not expect to be able to open a HTTPS connection (https://192.168.7.57:16993/ will not work).

Again, you will need to be on a different system in order to access the AMT web interface of the system you want to connect to.

Headless Configuration

Enabling KVM (VNC) Without A Monitor

The default BIOS setting for the Intel NUC7i7DNHE made it impossible to manage the system using VNC unless a monitor was physically plugged into the system. We found that we needed to go into the BIOS and configure the display for Headless display emulation operation as shown below in a remote VNC session running in meshcommander.

Enabling KVM (VNC) Without A Keyboard

We also found that if we attempted to use the meshcommander web interface to power our NUC directly to the BIOS, we could only view the screen in a VNC connection if a USB keyboard/mouse was physically connected to the system. While we were unable to find a BIOS setting to address this issue, we did find a suitable work around. Instead of issuing a command to powering up directly to the BIOS, we did the following:

• Started a VNC session to the AMT system (using the meshcommander web interface).
• Powered up the system remotely (using the meshcommander web interface).
• Pressed the F2 key at the boot screen when it appeared in our VNC session to enter the BIOS.

Entering the BIOS in this manner fooled the system into thinking that our VNC connection was providing the necessary keyboard/mouse to control the BIOS settings.

VNC Client Limitations

We found that VNC communications used by the AMT interface for remote desktop applications is not widely supported. The VNC client in the meshcommander web interface had no problem. Neither did the RealVNC Google Chrome extension. But, the tigervnc client that comes with Fedora would not work.

AMT Serial Port

The out of band serial port services provided by AMT on our NUC had limited benefit.

• We could not determine how to configure grub to recognize the AMT serial port. We think this is due to the EFI based installation of NST 28, but are not 100% certain.
• We were able to determine how to to use the AMT serial port after the OS was loaded.

These limitations mean that we can make use of the AMT serial port remotely as long as the host OS is up and running (even if the host OS has disabled networking). However, this is a subset of the features available through a VNC connection to the AMT remote desktop making the AMT serial port of limited use.

If you want to make use of the AMT serial port on an NST or Fedora based Linux system, you will first need to find out what serial device the AMT port is mapped to.

[root@beans ~]# dmesg | grep ttyS
[    1.024061] 00:01: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.046709] 0000:00:16.3: ttyS4 at I/O 0xf0a0 (irq = 19, base_baud = 115200) is a 16550A
[  127.583269] audit: type=1130 audit(1532408147.625:181): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=serial-getty@ttyS4 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[  127.583282] audit: type=1131 audit(1532408147.625:182): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=serial-getty@ttyS4 comm="systemd" exe="/usr/lib/systemd/systemd" 
hostname=? addr=? terminal=? res=success'
[root@beans ~]# 

The output above indicates that our NUC has two serial ports ttyS0 and ttyS4. Experience and Google searches lead us to believe that ttyS0 is a real serial port available on the NUC whereas ttyS4 is the virtual AMT serial port. To enable and start a serial port connection on this port (without system messages being logged), we can use the following commands:

[root@beans ~]# systemctl enable serial-getty@ttyS4.service
Created symlink /etc/systemd/system/getty.target.wants/serial-getty@ttyS4.service → /usr/lib/systemd/system/serial-getty@.service.
[root@beans ~]# systemctl start serial-getty@ttyS4.service
[root@beans ~]# 

Once enabled, we can used the Serial-over-LAN interface in meshcommander to remotely connect to the serial port. Alternatively we could use the amtterm (console) or gamt (GUI) terminals to remotely connect.

If we wanted to have system log messages appear in our serial console, we could enable an additional console in our kernel boot options instead of enabling a service via systemctl by adding console=tty0 console=ttyS4,115200n8 to the end of the kernel boot options:

linuxefi /vmlinuz-4.17.7-200.fc28.x86_64 root=/dev/mapper/nst_beans-root ro resume=/dev/mapper/nst_beans-swap rd.lvm.lv=nst_beans/root rd.lvm.lv=nst_beans/swap acpi=force rhgb quiet audit=0 systemd.unit=multi-user.target console=tty0 console=ttyS4,115200n8

As far as we know, you must set the serial port speed to 115200 for the AMT serial port.

Also, if you have previously enabled the serial port via systemctl, you should disable it now as the kernel will set it up for you:

[pkb@beans ~]$ sudo systemctl disable serial-getty@ttyS4.service
Removed /etc/systemd/system/getty.target.wants/serial-getty@ttyS4.service.
[pkb@beans ~]$ 

For NST systems, you can enable this in the NST boot options by setting the NST_SERIAL options the /etc/nst/grub2/nst_grub2_defaults file. For example, the following changes to the NST Console boot entry enable a serial console connected to the AMT serial port (/dev/ttyS4):

#
# NST Console (Boot)
# --- ------- ------
NST_SERIAL[idx]="true";
NST_SERIAL_DEV[idx]="ttyS4";
NST_SERIAL_BAUD[idx]="115200";
NST_TITLE[idx]="Console";
NST_GRUB_CMDLINE_LINUX[idx]="audit=0 systemd.unit=multi-user.target";
NST_GRUB_SAVEDEFAULT[idx]="true";
NST_GRAPHICAL_BOOT[idx]="true";
idx=idx+1;

Do not forget to run the nstboot command after making changes to the boot configuration file.

[pkb@beans ~]$ sudo nstboot --grub2
[pkb@beans ~]$

Speeding Up Boot Time

Though it may sound backwards, enabling the graphical boot feature in your grub configuration will speed up the boot time. This is due to the fact that the scrolling of output on the console during boot is extremely slow. To enable graphical boot, make sure that your kernel boot line contains rhgb. For example, the following enabled graphical boot, but goes to a non-graphical console mode once the boot process is complete (systemd.unit=multi-user.target).

linuxefi /vmlinuz-4.17.7-200.fc28.x86_64 root=/dev/mapper/nst_beans-root ro resume=/dev/mapper/nst_beans-swap rd.lvm.lv=nst_beans/root rd.lvm.lv=nst_beans/swap acpi=force rhgb quiet audit=0 systemd.unit=multi-user.target

For NST systems, you can enable this in the NST boot options by setting the NST_GRAPHICAL_BOOT[idx] options to true in the /etc/nst/grub2/nst_grub2_defaults file. For example:

#
# NST Console (Boot)
# --- ------- ------
NST_SERIAL[idx]="false";
NST_SERIAL_DEV[idx]="ttyS4";
NST_SERIAL_BAUD[idx]="115200";
NST_TITLE[idx]="Console";
NST_GRUB_CMDLINE_LINUX[idx]="audit=0 systemd.unit=multi-user.target";
NST_GRUB_SAVEDEFAULT[idx]="true";
NST_GRAPHICAL_BOOT[idx]="true";
idx=idx+1;

Do not forget to run the nstboot command after making changes to the boot configuration file.

[pkb@beans ~]$ sudo nstboot --grub2
[pkb@beans ~]$ 

Additional Hardware Notes

Running gnome-disks to benchmark the disk read speed showed that the Samsung 970 EVO NVME SSD was operating at about half speed. While the access times looked good, the transfer rates looked slow (at least compared to what you would expect form the 970 EVO). After a bit of googling it looks like this is a known issue with this combination (see: https://communities.intel.com/thread/126725). Hopefully there will be a BIOS update that will address this in the near future.

Linux Tools

MeshCommander

The MeshCommander project (see: http://www.meshcommander.com/meshcommander) provides an incredible tool for working with Intel vPro AMT systems on your network. It works as follows:

• You install the npm and nodejs package for your platform on the system that will act as the central manager.
  • On macOS one can use Homebrew and the on a command line run: brew install npm.
• You npm install the meshcommander package. On NST run: dnf install meshcommander. One macOS running Homebrew run: npm install meshcommander.
• You start the meshcommander server.
• You point your browser at http://127.0.0.1:3000/ (or alternate IPv4 address if you enabled the --any option).
• Using the web interface, you add and manage Intel AMT systems on your LAN to the meshcommander web interface.

Once a system has been successfully added and configured, you can bring up an out of band serial or remote desktop (VNC) to each managed system.

For more information about setting up and running the meshcommander server, see: https://www.npmjs.com/package/meshcommander. The server is fairly light and can run on minimal hardware if desired (we have run it on a Raspberry PI 3 to manage Intel systems on the network).

The amtterm Package

The amtterm package is available in Fedora, to install:

 sudo yum install amtterm

It provides the amtterm, gamt and amttool utilities and maybe most importantly, the amt-howto man page.

Both amtterm and gamt term can be used to connect to the out of band serial port on an Intel system with AMT support (provided out of band serial access is enabled). The amtterm utility is a console application where as gamt is a GUI version.

The amttool relies on an older SOAP method that is no longer supported in recent AMT releases. If you need to use this tool (most likely you won't), you will need to install the perl-SOAP-Lite package.

The mei-amt-check Tool

The AMT Status Checker for Linux (mei-amt-check) tool is a simple command that allows you detect whether or not your Intel based system supports Intel's vPro AMT out of band management. The following shows what the command reports on a system with AMT support (an Intel NUC7i7DNHE):

[pkb@beans ~]$ sudo /usr/sbin/mei-amt-check
AMT present: true
AMT provisioning state: provisioned
Flash:	11.8.50
Netstack:	11.8.50
AMTApps:	11.8.50
AMT:	11.8.50
Sku:	16392
VendorID:	8086
Build Number:	3425
Recovery Version:	11.8.50
Recovery Build Num:	3425
Legacy Mode:	False
[pkb@beans ~]$ 

The following shows what the tool displays on an Intel based system that does not have AMT support:

[pkb@refritos ~]$ sudo /usr/sbin/mei-amt-check 
Error: Management Engine refused connection. This probably means you don't have AMT
[pkb@refritos ~]$ 

Information for building and running this handy tool can be found at: https://github.com/mjg59/mei-amt-check.

Starting with NST 28, the NST source code contains the tools to build an mei-amt-check package.

WS Management Command Line Interface

The wsmancli package provides provides the wsman and wseventmgr utilities that can be used to query and monitor AMT based systems. To install:

yum install wsmancli

These tools are a bit difficult to use. Here is an example of using the wsman command to get identity information from an AMT based system (change AMT_PASS to the password your AMT system requires and 192.168.8.13 to the IPv4 address of your AMT based system):

[pkb@refritos ~]$ export WSMAN_PASS=AMT_PASS
[pkb@refritos ~]$ export WSMAN_USER=admin
[pkb@refritos ~]$ wsman -h 192.168.8.13 -P 623 identify
<?xml version="1.0" encoding="UTF-8"?>
<a:Envelope xmlns:a="http://www.w3.org/2003/05/soap-envelope" xmlns:b="http://schemas.dmtf.org/wbem/wsman/identity/1/wsmanidentity.xsd" xmlns:c="http://schemas.dmtf.org/wbem/dash/1/dash.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <a:Header/>
  <a:Body>
    <b:IdentifyResponse>
      <b:ProtocolVersion>http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd</b:ProtocolVersion>
      <b:ProductVendor>Intel(r)</b:ProductVendor>
      <b:ProductVersion>AMT 11.8</b:ProductVersion>
      <c:DASHVersion>1.0.0</c:DASHVersion>
      <b:SecurityProfiles>
        <b:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/digest</b:SecurityProfileName>
        <b:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/https/digest</b:SecurityProfileName>
        <b:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/https/mutual/digest</b:SecurityProfileName>
        <b:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/spnego-kerberos</b:SecurityProfileName>
        <b:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/https/spnego-kerberos</b:SecurityProfileName>
        <b:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/https/mutual/spnego-kerberos</b:SecurityProfileName>
      </b:SecurityProfiles>
    </b:IdentifyResponse>
  </a:Body>
</a:Envelope>
[pkb@refritos ~]$