View source for HowTo Headless Intel NUC vPro AMT

== Background ==

We purchased a Intel NUC (NUC7i7DNHE to be precise) with Intel's vPro AMT support in July 2018 to act as a new headless build machine when doing NST software development. This page describes some of findings related to setting up the Intel NUC as a headless server running NST 28. Many of these findings will apply to other Linux distributions especially those that are based on Fedora.

== Finding Intel AMT Enabled Systems ==

The following '''nmap''' command can be used to scan a single host for Intel AMT related ports:

 [pkb@refritos ~]$ nmap -Pn -p 623,664,5900,9971,16992-16995 192.168.7.57
 Starting Nmap 7.70SVN ( https://nmap.org ) at 2018-07-23 16:21 EDT
 Nmap scan report for beans.attlocal.net (192.168.7.57)
 Host is up (0.0021s latency).
 
 PORT      STATE  SERVICE
 623/tcp   open   oob-ws-http
 664/tcp   closed secure-aux-bus
 5900/tcp  open   vnc
 9971/tcp  closed unknown
 16992/tcp open   amt-soap-http
 16993/tcp closed amt-soap-https
 16994/tcp open   amt-redir-tcp
 16995/tcp closed amt-redir-tls
 
 Nmap done: 1 IP address (1 host up) scanned in 6.54 seconds
 [pkb@refritos ~]$ 

This scan can take a long time if you do an entire network. If you have enabled your AMT systems to respond to ''ICMP'' pings, you can drop the ''-Pn'' option to try and speed up the scan.

=== Unable to Scan Self ===

This may be obvious, but just to be clear, you can not scan yourself. AMT will only allow connections from external machines. If you try to scan yourself, '''nmap''' will show all ports as closed as demonstrated below (after logging into the ''192.168.7.57'') machine:

 [pkb@beans ~]$ nmap -p 623,664,5900,9971,16992-16995 192.168.7.57
 Starting Nmap 7.70SVN ( https://nmap.org ) at 2018-07-24 00:25 EDT
 Nmap scan report for beans.attlocal.net (192.168.7.57)
 Host is up (0.000038s latency).
 
 PORT      STATE  SERVICE
 623/tcp   closed oob-ws-http
 664/tcp   closed secure-aux-bus
 5900/tcp  closed vnc
 9971/tcp  closed unknown
 16992/tcp closed amt-soap-http
 16993/tcp closed amt-soap-https
 16994/tcp closed amt-redir-tcp
 16995/tcp closed amt-redir-tls 
 
 Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds
 [pkb@beans ~]$ 

While this makes sense (AMT is designed to manage servers out of band), it can sometimes be easy to forget.

=== Connecting ===

If the scan shows that port 16992 or 16993 is open, you should be to open up a connection to the simple AMT web interface using a web browser. For example, if in the previous scan, ''192.168.7.57'' showed that port 16992 was open. This indicates that a HTTP connection (http://192.168.7.57:16992/) should work. However, since port 16993 was shown as closed, we would not expect to be able to open a HTTPS connection (https://192.168.7.57:16993/ will not work).

Again, you will need to be on a different system in order to access the AMT web interface of the system you want to connect to.

== Headless Configuration ==

=== Enabling KVM (VNC) Without A Monitor ===

The default BIOS setting for the Intel NUC7i7DNHE made it impossible to manage the system using VNC unless a monitor was physically plugged into the system. We found that we needed to go into the BIOS and configure the display for ''Headless display emulation'' operation as shown below in a remote VNC session running in '''meshcommander'''.

[[File:NUC-Headless.png]]

=== Enabling KVM (VNC) Without A Keyboard ===

We also found that if we attempted to use the '''meshcommander''' web interface to power our NUC directly to the BIOS, we could only view the screen in a VNC connection if a USB keyboard/mouse was physically connected to the system. While we were unable to find a BIOS setting to address this issue, we did find a suitable work around. Instead of issuing a command to powering up directly to the BIOS, we did the following:

* Started a VNC session to the AMT system (using the '''meshcommander''' web interface).
* Powered up the system remotely (using the '''meshcommander''' web interface).
* Pressed the ''F2'' key at the boot screen when it appeared in our VNC session to enter the BIOS.

Entering the BIOS in this manner fooled the system into thinking that our VNC connection was providing the necessary keyboard/mouse to control the BIOS settings.

=== VNC Client Limitations ===

We found that VNC communications used by the AMT interface for remote desktop applications is not widely supported. The VNC client in the '''meshcommander''' web interface had no problem. Neither did the RealVNC Google Chrome extension. But, the '''tigervnc''' client that comes with Fedora would not work. 

=== AMT Serial Port ===

The out of band serial port services provided by AMT on our NUC had limited benefit.

* We could not determine how to configure '''grub''' to recognize the AMT serial port. We think this is due to the EFI based installation of NST 28, but are not 100% certain.
* We were able to determine how to to use the AMT serial port after the OS was loaded.

These limitations mean that we can make use of the AMT serial port remotely as long as the host OS is up and running (even if the host OS has disabled networking). However, this is a subset of the features available through a VNC connection to the AMT remote desktop making the AMT serial port of limited use.

If you want to make use of the AMT serial port on an NST or Fedora based Linux system, you will first need to find out what serial device the AMT port is mapped to.

 [root@beans ~]# dmesg | grep ttyS
 [    1.024061] 00:01: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
 [    1.046709] 0000:00:16.3: ttyS4 at I/O 0xf0a0 (irq = 19, base_baud = 115200) is a 16550A
 [  127.583269] audit: type=1130 audit(1532408147.625:181): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=serial-getty@ttyS4 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
 [  127.583282] audit: type=1131 audit(1532408147.625:182): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=serial-getty@ttyS4 comm="systemd" exe="/usr/lib/systemd/systemd" 
 hostname=? addr=? terminal=? res=success'
 [root@beans ~]# 

The output above indicates that our NUC has two serial ports ''ttyS0'' and ''ttyS4''. Experience and Google searches lead us to believe that ''ttyS0'' is a real serial port available on the NUC whereas ''ttyS4'' is the virtual AMT serial port. To enable and start a serial port connection on this port (without system messages being logged), we can use the following commands:

 [root@beans ~]# systemctl enable serial-getty@ttyS4.service
 Created symlink /etc/systemd/system/getty.target.wants/serial-getty@ttyS4.service → /usr/lib/systemd/system/serial-getty@.service.
 [root@beans ~]# systemctl start serial-getty@ttyS4.service
 [root@beans ~]# 

Once enabled, we can used the ''Serial-over-LAN'' interface in '''meshcommander''' to remotely connect to the serial port. Alternatively we could use the ''amtterm'' (console) or ''gamt'' (GUI) terminals to remotely connect.

If we wanted to have system log messages appear in our serial console, we could enable an additional console in our kernel boot options instead of enabling a service via '''systemctl''' by adding ''console=tty0 console=ttyS4,115200n8'' to the end of the kernel boot options:

 linuxefi /vmlinuz-4.17.7-200.fc28.x86_64 root=/dev/mapper/nst_beans-root ro resume=/dev/mapper/nst_beans-swap rd.lvm.lv=nst_beans/root rd.lvm.lv=nst_beans/swap acpi=force rhgb quiet audit=0 systemd.unit=multi-user.target console=tty0 console=ttyS4,115200n8

As far as we know, you must set the serial port speed to 115200 for the AMT serial port.

Also, if you have previously enabled the serial port via '''systemctl''', you should disable it now as the kernel will set it up for you:

 [pkb@beans ~]$ sudo systemctl disable serial-getty@ttyS4.service
 Removed /etc/systemd/system/getty.target.wants/serial-getty@ttyS4.service.
 [pkb@beans ~]$ 

For NST systems, you can enable this in the NST boot options by setting the ''NST_SERIAL'' options the ''/etc/nst/grub2/nst_grub2_defaults'' file. For example, the following changes to the NST Console boot entry enable a serial console connected to the AMT serial port (''/dev/ttyS4''):

 #
 # NST Console (Boot)
 # --- ------- ------
 NST_SERIAL[idx]="true";
 NST_SERIAL_DEV[idx]="ttyS4";
 NST_SERIAL_BAUD[idx]="115200";
 NST_TITLE[idx]="Console";
 NST_GRUB_CMDLINE_LINUX[idx]="audit=0 systemd.unit=multi-user.target";
 NST_GRUB_SAVEDEFAULT[idx]="true";
 NST_GRAPHICAL_BOOT[idx]="true";
 idx=idx+1;

Do not forget to run the '''nstboot''' command after making changes to the boot configuration file.

 [pkb@beans ~]$ sudo nstboot --grub2
 [pkb@beans ~]$

=== Speeding Up Boot Time ===

Though it may sound backwards, enabling the graphical boot feature in your '''grub''' configuration will speed up the boot time. This is due to the fact that the scrolling of output on the console during boot is extremely slow. To enable graphical boot, make sure that your kernel boot line contains ''rhgb''. For example, the following enabled graphical boot, but goes to a non-graphical console mode once the boot process is complete (''systemd.unit=multi-user.target'').

 linuxefi /vmlinuz-4.17.7-200.fc28.x86_64 root=/dev/mapper/nst_beans-root ro resume=/dev/mapper/nst_beans-swap rd.lvm.lv=nst_beans/root rd.lvm.lv=nst_beans/swap acpi=force rhgb quiet audit=0 systemd.unit=multi-user.target

For NST systems, you can enable this in the NST boot options by setting the ''NST_GRAPHICAL_BOOT[idx]'' options to ''true'' in the ''/etc/nst/grub2/nst_grub2_defaults'' file. For example:

 #
 # NST Console (Boot)
 # --- ------- ------
 NST_SERIAL[idx]="false";
 NST_SERIAL_DEV[idx]="ttyS4";
 NST_SERIAL_BAUD[idx]="115200";
 NST_TITLE[idx]="Console";
 NST_GRUB_CMDLINE_LINUX[idx]="audit=0 systemd.unit=multi-user.target";
 NST_GRUB_SAVEDEFAULT[idx]="true";
 NST_GRAPHICAL_BOOT[idx]="true";
 idx=idx+1;

Do not forget to run the '''nstboot''' command after making changes to the boot configuration file.

 [pkb@beans ~]$ sudo nstboot --grub2
 [pkb@beans ~]$ 


=== Additional Hardware Notes ===

Running '''gnome-disks''' to benchmark the disk read speed showed that the Samsung 970 EVO NVME SSD was operating at about half speed. While the access times looked good, the transfer rates looked slow (at least compared to what you would expect form the 970 EVO). After a bit of ''googling'' it looks like this is a known issue with this combination (see: https://communities.intel.com/thread/126725). Hopefully there will be a BIOS update that will address this in the near future.

== Linux Tools ==

=== MeshCommander ===

The ''MeshCommander'' project (see: http://www.meshcommander.com/meshcommander) provides an incredible tool for working with Intel vPro AMT systems on your network. It works as follows:

* You install the '''npm''' and '''nodejs''' package for your platform on the system that will act as the central manager.
* You ''npm install'' the '''meshcommander''' package.
* On MacOS
* You start the '''meshcommander''' server.
* You point your browser at http://127.0.0.1:3000/ (or alternate IPv4 address if you enabled the ''--any'' option).
* Using the web interface, you add and manage Intel AMT systems on your LAN to the '''meshcommander''' web interface.

Once a system has been successfully added and configured, you can bring up an out of band serial or remote desktop (VNC) to each managed system.


For more information about setting up and running the '''meshcommander''' server, see: https://www.npmjs.com/package/meshcommander. The server is fairly light and can run on minimal hardware if desired (we have run it on a Raspberry PI 3 to manage Intel systems on the network).

=== The amtterm Package ===

The '''amtterm''' package is available in Fedora, to install:

  sudo yum install amtterm

It provides the '''amtterm''', '''gamt''' and '''amttool''' utilities and maybe most importantly, the ''amt-howto'' man page.

Both '''amtterm''' and '''gamt''' term can be used to connect to the out of band serial port on an Intel system with AMT support (provided out of band serial access is enabled). The '''amtterm''' utility is a console application where as '''gamt''' is a GUI version.

The '''amttool''' relies on an older SOAP method that is no longer supported in recent AMT releases. If you need to use this tool (most likely you won't), you will need to install the '''perl-SOAP-Lite''' package.

=== The mei-amt-check Tool ===

The ''AMT Status Checker'' for Linux ('''mei-amt-check''') tool is a simple command that allows you detect whether or not your Intel based system supports Intel's vPro AMT out of band management. The following shows what the command reports on a system with AMT support (an Intel NUC7i7DNHE):

 [pkb@beans ~]$ sudo /usr/sbin/mei-amt-check
 AMT present: true
 AMT provisioning state: provisioned
 Flash:	11.8.50
 Netstack:	11.8.50
 AMTApps:	11.8.50
 AMT:	11.8.50
 Sku:	16392
 VendorID:	8086
 Build Number:	3425
 Recovery Version:	11.8.50
 Recovery Build Num:	3425
 Legacy Mode:	False
 [pkb@beans ~]$ 

The following shows what the tool displays on an Intel based system that does not have AMT support:

 [pkb@refritos ~]$ sudo /usr/sbin/mei-amt-check 
 Error: Management Engine refused connection. This probably means you don't have AMT
 [pkb@refritos ~]$ 

Information for building and running this handy tool can be found at: https://github.com/mjg59/mei-amt-check.

Starting with NST 28, the NST source code contains the tools to build an '''mei-amt-check''' package. 

=== WS Management Command Line Interface ===

The '''wsmancli''' package provides provides the '''wsman''' and '''wseventmgr''' utilities that can be used to query and monitor AMT based systems. To install:

 yum install wsmancli

These tools are a bit difficult to use. Here is an example of using the '''wsman''' command to get ''identity'' information from an AMT based system (change ''AMT_PASS'' to the password your AMT system requires and ''192.168.8.13'' to the IPv4 address of your AMT based system):

 [pkb@refritos ~]$ export WSMAN_PASS=AMT_PASS
 [pkb@refritos ~]$ export WSMAN_USER=admin
 [pkb@refritos ~]$ wsman -h 192.168.8.13 -P 623 identify
 <?xml version="1.0" encoding="UTF-8"?>
 <a:Envelope xmlns:a="http://www.w3.org/2003/05/soap-envelope" xmlns:b="http://schemas.dmtf.org/wbem/wsman/identity/1/wsmanidentity.xsd" xmlns:c="http://schemas.dmtf.org/wbem/dash/1/dash.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <a:Header/>
   <a:Body>
     <b:IdentifyResponse>
       <b:ProtocolVersion>http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd</b:ProtocolVersion>
       <b:ProductVendor>Intel(r)</b:ProductVendor>
       <b:ProductVersion>AMT 11.8</b:ProductVersion>
       <c:DASHVersion>1.0.0</c:DASHVersion>
       <b:SecurityProfiles>
         <b:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/digest</b:SecurityProfileName>
         <b:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/https/digest</b:SecurityProfileName>
         <b:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/https/mutual/digest</b:SecurityProfileName>
         <b:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/spnego-kerberos</b:SecurityProfileName>
         <b:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/https/spnego-kerberos</b:SecurityProfileName>
         <b:SecurityProfileName>http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/https/mutual/spnego-kerberos</b:SecurityProfileName>
       </b:SecurityProfiles>
     </b:IdentifyResponse>
   </a:Body>
 </a:Envelope>
 [pkb@refritos ~]$