Mandiant - Alan Chen

Nginx

LocationEast Asia
Public IP20.239.168.223
Private IP10.0.0.4
Gateway IP10.0.0.1
MAC address00:22:48:15:ba:47 (eth0)
SSHssh mandiant@20.239.168.223

Apache

LocationEast Asia
Public IP104.208.79.117
Private IP10.0.0.5
Gateway IP10.0.0.1
MAC address00:22:48:16:14:90 (eth1)
SSHssh mandiant@104.208.79.117

Mandiant Interview Challenge


  Mandiant Interview Challenge
  ============================
  
  [Website](http://mandiant.alancf.cc/)
  [Slide](https://docs.google.com/presentation/d/1B08yv_hLtwZ7pX1ByX_tRSUiqRfzxIqFUgbKtL09ECc/)
  
  High level overview
  -------------------
  The end goal is to host a multi-server web instance on a single machine instance focusing on 
  capturing any network issues on configuration and proof of traffic routing independently across 
  both interfaces. You will be creating a virtual machine using the Ubuntu 18.04-LTS offering in 
  Microsoft Azure's Marketplace utilizing two dedicated network interfaces with associated external 
  IPs; each hosting a unique (not shared) web-server and serving unique content. 
  
  Background 
  ----------
  Linux provides for unique routing challenges when using multiple interfaces on the same subnet 
  associated with external IPs and how requests are made in and out of the system. The goal of 
  this project is to demonstrate an understanding of the issues and how to work around them inside 
  the Azure environment. 
  
  Specifics
  ---------
  Launch a Microsoft Azure virtual machine with multiple "physical" network interfaces and associated 
  public external IPs. When setting up the instance, the internal interfaces should reside in the same 
  virtual network on the same class-C subnet, utilizing the same gateway address. External IPs shall 
  not be load-balanced and should implement network security groups for access control (we can provide 
  you with the public IP addresses to add to the ACL during the presentation for us to check it on completion). 
  
  You may choose any web-server to host unique directories with unique content; make it as simple as 
  possible, feel free to make use of full setups like Python SimpleHTTPServer. To any outside user, 
  these two servers should appear as distinct hosts on the internet.  
  
  To demonstrate execution of these services and their communication, tcpdump will be run on the 
  Microsoft Azure host, demonstrating a simple GET request and the traffic flow. Show where the request 
  was made and routing information at both the MAC and IP level tracing back to each interface. 
  
  
  Output/Presentation 
  -------------------
  For the presentation, looking for: 
  * Simple live demo 
  * Challenges faced during setup and verification 
  * Any design decisions or trade off made in the process 
  * How an implementation could be improved or scaled for production use 
  * Only expecting a few slides, you can talk to and any visualization that helps demonstrate the goal 
    and work in the project. Be prepared to answer questions related to the learning process and testing 
    methodologies used in construction of the project, how any solutions were identified and how these 
    solutions were implemented to achieve the desired outcomes defined above. Audience should be considered 
    reasonably technical but possibly new to Microsoft Azure and running multiple services on a single instance. 


  Parameters and limitations
  --------------------------
  * The Microsoft Azure virtual machine/size should be free service eligible 
  * The Microsoft Azure image used should be Ubuntu 18.04-LTS from Azure marketplace 
  * Limit access to the environment via properly implemented network security groups 
  * No purchased software should be used in the construction of this system 
  * No authentication or session management is necessary for this application as security group restrictions are sufficient 
  * A user interface OR API is sufficient for accessing the application 
  * Raw command output is sufficient for this project as no UI programming is expected

Create VMs on Azure portal


  # Create Resource Group

  # Create Virtual Networks
  
  # Create Network Security Group

  # Create Network Security Rules

  # Create 2 Public IPs
  
  # Create 2 NICs

  # Create VM

Login to VM


  chmod 400 VM1.pem 

  ssh azureuser@20.239.168.223
  ssh azureuser@104.208.79.117 (unreachable at the moment)

System update


  sudo apt update
  sudo apt upgrade -y

Change ARP setting


  # Change ARP setting, since I thought I met this issue in the past (Linux weak host model)
  # https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/Documentation/networking/ip-sysctl.txt?h=v4.9#n1130

  sudo echo "net.ipv4.conf.all.arp_ignore=1" >> /etc/sysctl.conf
  sudo echo "net.ipv4.conf.all.arp_announce=2" >> /etc/sysctl.conf
  sudo sysctl --system

  sudo cat /proc/sys/net/ipv4/conf/all/arp_ignore
  # 1
  sudo cat /proc/sys/net/ipv4/conf/all/arp_announce
  # 2

  # Not working

  # Check ARP table on another VM
  ip neigh show

  # All MAC address is 12:34:56:78:9a:bc
  # Changing the ARP setting is meaningless (rollback), Azure already take care of this

Change rp_filter


  # Check packets using ping and tcpdump
  tcpdump -i eth0 -nn icmp -e
  tcpdump -i eth1 -nn icmp -e

  # 1 interface/IP looks good
  # No return packets on an interface

  # Change rp_filter 
  sudo echo "net.ipv4.conf.all.rp_filter=0" >> /etc/sysctl.conf
  sudo echo "net.ipv4.conf.eth0.rp_filter=0" >> /etc/sysctl.conf
  sudo echo "net.ipv4.conf.eth1.rp_filter=0" >> /etc/sysctl.conf
  sudo sysctl --system

  sudo cat /proc/sys/net/ipv4/conf/all/rp_filter
  # 0
  sudo cat /proc/sys/net/ipv4/conf/eth0/rp_filter
  # 0
  sudo cat /proc/sys/net/ipv4/conf/eth1/rp_filter
  # 0 

  # Check packets using ping and tcpdump
  tcpdump -i eth0 -nn icmp -e
  tcpdump -i eth1 -nn icmp -e

  # 1 interface/IP looks good
  # Return packets on the wrong interface for the other IP

Change routing


  # Check routing table
  route -n

  # Disable cloud-init first
  sudo echo "network: {config: disabled}" > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
  sudo rm /etc/netplan/50-cloud-init.yaml

  # Deploy new netplan
  vi /etc/netplan/10-static.yaml
  netplay try

netplan yaml


  network:
    ethernets:
      eth0:
        dhcp4: no
        addresses: [10.0.0.4/24]
        gateway4: 10.0.0.1
        nameservers:
          addresses: [1.1.1.1]
        routes:
          - to: 0.0.0.0/0
            via: 10.0.0.1
            table: 1
        routing-policy:
          - from: 10.0.0.4
            table: 1
  
      eth1:
        dhcp4: no
        addresses: [10.0.0.5/24]
        gateway4: 10.0.0.1
        nameservers:
          addresses: [1.1.1.1]
        routes:
          - to: 0.0.0.0/0
            via: 10.0.0.1
            table: 2
        routing-policy:
          - from: 10.0.0.5
            table: 2
    version: 2

Check result


  # Check routing
  route -n
  # default via 10.0.0.1 dev eth1 proto static 
  # default via 10.0.0.1 dev eth0 proto static 
  # 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.4 
  # 10.0.0.0/24 dev eth1 proto kernel scope link src 10.0.0.5 

  # Check ping
  tcpdump -i eth0 -nn icmp -e
  tcpdump -i eth1 -nn icmp -e

  # Looks good!

Install HTTP servers


  # Install Nginx, Apache HTTP server
  sudo apt install apache2 -y
  sudo apt install nginx -y

Web server setting


  # Set Apache HTTP server listen IP
  echo "ServerName 10.0.0.5" >> /etc/apache2/apache2.conf
  vi /etc/apache2/ports.conf

  # Set Nginx default html page and listen IP
  mv /etc/nginx/sites-enabled/default /etc/nginx/conf.d/default.conf
  vi /etc/nginx/conf.d/default.conf

  # Nginx auto-start after reboot
  vi /lib/systemd/system/nginx.service
  # Add in [Unit]
  # Wants=network-online.target
  # After=network-online.target

  systemctl daemon-reload
  systemctl restart apache2
  systemctl enable nginx --now

Check Web server


  # Web servers links above

  ss -tulp |grep http
  tcpdump -n -e tcp port 80 and not host 168.63.129.16 and not 169.254.169.254 -i eth0
  tcpdump -n -e tcp port 80 and not host 168.63.129.16 and not 169.254.169.254 -i eth1

Automation


  # Create VMs on Azure using Azure CLI

  # Create ResourceGroup
  az group create --name Mandiant01 --location eastasia

  # Create Virtual Networks
  az network vnet create \
      --resource-group Mandiant01 \
      --name mVnet \
      --address-prefix 10.0.0.0/16 \
      --subnet-name mSubnet \
      --subnet-prefix 10.0.0.0/24
  
  # Create Network Security Group
  az network nsg create \
      --resource-group Mandiant01 \
      --name myNetworkSecurityGroup

  # Create Network Security Rules
  az network nsg rule create --name Alan \
      --nsg-name myNetworkSecurityGroup \
      --priority 100 \
      --resource-group Mandiant01 \
      --access Allow \
      --destination-address-prefixes '*' \
      --destination-port-ranges '*' \
      --protocol '*' \
      --source-address-prefixes 182.55.110.87/32 \
      --source-port-ranges '*'

  # Create Network Security Rules
  az network nsg rule create --name Huan \
      --nsg-name myNetworkSecurityGroup \
      --priority 110 \
      --resource-group Mandiant01 \
      --access Allow \
      --destination-address-prefixes '*' \
      --destination-port-ranges 22 80 \
      --protocol '*' \
      --source-address-prefixes 58.182.20.3/32 \
      --source-port-ranges '*'

  # Create 2 Public IPs
  az network public-ip create --name IP1 \
      --resource-group Mandiant01 \
      --allocation-method Static
  az network public-ip create --name IP2 \
      --resource-group Mandiant01 \
      --allocation-method Static
  
  # Create 2 NICs
  az network nic create \
      --accelerated-networking false \
      --resource-group Mandiant01 \
      --name Nic1 \
      --vnet-name mVnet \
      --subnet mSubnet \
      --private-ip-address 10.0.0.4 \
      --public-ip-address IP1 \
      --network-security-group myNetworkSecurityGroup
  az network nic create \
      --accelerated-networking false \
      --resource-group Mandiant01 \
      --name Nic2 \
      --vnet-name mVnet \
      --subnet mSubnet \
      --private-ip-address 10.0.0.5 \
      --public-ip-address IP2 \
      --network-security-group myNetworkSecurityGroup

  # Create VM
  az vm create \
    --location eastasia \
    --resource-group Mandiant01 \
    --name VM1 \
    --computer-name VM1 \
    --os-disk-delete-option Delete \
    --image Canonical:UbuntuServer:18.04-LTS:latest \
    --size Standard_D2s_v3 \
    --admin-username azureuser \
    --admin-password *** \
    --authentication-type password \
    --nics Nic1 Nic2 \
    --nic-delete-option Detach \
    --custom-data cloud-init.yml

cloud-init.yml


  #cloud-config

  bootcmd:
    - apt update
    - apt upgrade -y
    - apt install -y apache2
    - apt install -y nginx
  
  users:
    - default
    - name: mandiant
      groups: sudo
      shell: /bin/bash
      sudo: ['ALL=(ALL) NOPASSWD:ALL']
      passwd: "$6$scu0b/Vf$dnYDyR.PixUGKbHpqGuyJxOv53/B29jn4ZU6psvr72FrdIu8j.q68fesnQ1Hsz6PphUMO0qppctb2eYnTYHOI/"
  
  write_files:
    - path: /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
      content: |
        network: {config: disabled}
      owner: root:root
      permissions: '0644'
    - path: /etc/netplan/10-static.yaml
      content: |
        network:
          ethernets:
            eth0:
              dhcp4: no
              addresses: [10.0.0.4/24]
              gateway4: 10.0.0.1
              nameservers:
                addresses: [1.1.1.1]
              routes:
                - to: 0.0.0.0/0
                  via: 10.0.0.1
                  table: 1
              routing-policy:
                - from: 10.0.0.4
                  table: 1
  
            eth1:
              dhcp4: no
              addresses: [10.0.0.5/24]
              gateway4: 10.0.0.1
              nameservers:
                addresses: [1.1.1.1]
              routes:
                - to: 0.0.0.0/0
                  via: 10.0.0.1
                  table: 2
              routing-policy:
                - from: 10.0.0.5
                  table: 2
          version: 2
      owner: root:root
      permissions: '0644'
    - path: /lib/systemd/system/nginx.service
      content: |
        [Unit]
        Description=A high performance web server and a reverse proxy server
        Documentation=man:nginx(8)
        After=network.target
        Wants=network-online.target
        After=network-online.target
  
        [Service]
        Type=forking
        PIDFile=/run/nginx.pid
        ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
        ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
        ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
        ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
        TimeoutStopSec=5
        KillMode=mixed
  
        [Install]
        WantedBy=multi-user.target
      owner: root:root
      permissions: '0644'
    - path: /etc/nginx/conf.d/default.conf
      owner: www-data:www-data
      content: |
        server {
            listen 10.0.0.4:80;
            root /var/www/html;
            index index.nginx-debian.html;
            server_name _;
            location / {
                try_files $uri $uri/ =404;
            }
        }
    - path: /etc/apache2/ports.conf
      owner: www-data:www-data
      content: |
        Listen 10.0.0.5:80
    - path: /etc/apache2/apache2.conf
      owner: www-data:www-data
      content: |
        ServerName 10.0.0.5
      append: true
  
  runcmd:
    - rm /etc/netplan/50-cloud-init.yaml
    - rm /etc/nginx/sites-enabled/default
    - netplan generate
    - netplan apply
    - systemctl daemon-reload
    - systemctl restart apache2
    - systemctl enable nginx --now
    - passwd mandiant -u