11.5. Group Walkthrough: Troubleshooting a Broken Linux Deployment¶

11.5.1.1.1. SSH Into the Machine¶

Within the Setup section of this walkthrough your TA will invite you to your Reader role for accessing the Azure resources related to this broken deployment. After you have accepted the invitation and configured your default resource group and VM in the az CLI you can request the public IP address of the VM.

Using the az CLI you can store the public IP in a variable so it can be used as the host component of the ssh command:

> $VmPublicIp = az vm list-ip-addresses --query '[0].virtualMachine.network.publicIpAddresses[0].ipAddress'

The first time you connect to a machine over SSH you will be prompted to trust or reject the remote host. When prompted enter yes to trust the host and connect to it:

> ssh student@$VmPublicIp
# trust the remote host
# enter password: LaunchCode-@zure1

Once you have connected to the machine, you will be able to navigate and use Bash and the tools listed above to gather information and plan a solution with your group.

$ vm_public_ip=$(az vm list-ip-addresses -o tsv --query '[0].virtualMachine.network.publicIpAddresses[0].ipAddress')
$ ssh student@"$vm_public_ip"
# trust the remote host
# enter password: LaunchCode-@zure1

11.5.1.1.2. Using service¶

The service program is a wrapper that simplifies how several of the init systems on a Linux machine can be managed through a single tool. Init systems are used to initialize and manage background processes running on Linux systems.

On Ubuntu machines the systemd init system and its client program systemctl (system control manager) are used by default to manage service units. In the configuration script of our final deployments we created a systemd unit file to define how our Coding Events API would be operated as a background service on the Ubuntu VM. The script also used the service tool (rather than the underlying systemctl it wraps) to make our script portable across supporting Linux distributions.

In addition to controlling services, the service tool can be used to view the status of any registered service units like our coding-events-api, nginx, and mysql:

 service <service-name> status

For example, if you were to check the status of a functioning API service you would receive the following output:

$ service coding-events-api status

● coding-events-api.service - Coding Events API
  Loaded: loaded (/etc/systemd/system/coding-events-api.service; disabled; vendor preset: enabled)
  Active: active (running) since Tue 2020-10-31 19:04:51 UTC; 1 day 4h ago
Main PID: 18196 (dotnet)
    Tasks: 16 (limit: 4648)
  CGroup: /system.slice/coding-events-api.service
          └─18196 /usr/bin/dotnet /opt/coding-events-api/CodingEventsAPI.dll

11.5.1.1.3. Using journalctl¶

The journalctl tool can be used to view the logs written by systemd services. You can use it to view the logs of a particular service unit using the -u (unit name) option:

$ journalctl -u <service-name>

The systemd journal can store thousands of logs and lines within them. Often it is useful to view just the most recent logs. The -f option will follow the logs starting from the last 10 lines and continuously display new lines as they are written:

$ journalctl -f -u <service-name>

# shorthand (-u comes after to pair with the service name argument)
$ journalctl -fu <service-name>

11.5.1.1.4. Working with Self-Signed Certificates¶

A self-signed certificate means that the signature used to sign the certificate is not recognized by an external certificate authority (CA). These certificates can still be used for TLS encryption but are not inherently trusted like traditional SSL certificates due to their unknown signing authority. By default, HTTP client applications like browsers or CLI tools will automatically reject self-signed certificates as a security measure.

In our configure-ssl.sh deployment script our VM internally generated the signing key used to sign the SSL certificate with the openssl tool. Recall that when you first connected to the Swagger documentation of your API in the browser, you had to bypass the warning and accept (explicitly trust) the self-signed certificate. CLI tools can be configured similarly to also accept self-signed certificates.

When working with Invoke-RestMethod cmdlet the default certificate validation behavior for self-signed certificates results in the following error for servers using self-signed certificates:

 Invoke-RestMethod: The remote certificate is invalid according to the validation procedure.

We can override the default validation procedure by using the -SkipCertificateCheck option:

> Invoke-RestMethod -Uri https://<PUBLIC IP> -SkipCertificateCheck

Similarly, when working inside the Ubuntu VM with curl the validation can be skipped using the -k option:

$ curl https://localhost -k

When troubleshooting within a VM, you can use curl to isolate networking-related issues. If you are able to connect successfully from inside the machine, but receive a timeout when connecting externally, it indicates that an internal firewall or external network security rule is the issue.

11.5.3.1.1. Accept Email¶

The first step is accessing the email that was sent from Microsoft on your TAs behalf. The email will include a link that will allow you to associate your email address with a new account under the directory and subscription the TA administers.

Upon clicking the link you will be taken to a Microsoft web page that will prompt you to create an account in your TA’s tenant directory.

11.5.3.1.2. Create Account In the TA Tenant Directory¶

The form will come pre-populated with your email address (since you navigated to the webpage from your email client) and you will need to enter a password.

This account, and subscription, will be temporary, so we will use the same password to make things consistent. Copy the password below, then paste it in the form to avoid spelling mistakes:

• Password: LaunchCode-@zure
• Display name: your name (should default)

An email will be sent to you containing a security code. Copy the code and paste it into the verification form:

You will then be prompted to accept the invitation permissions (select Accept):

At the next prompt, you can select the Skip for now link, as this is only temporary for this final exercise:

Then select Yes to stay signed in:

11.5.3.1.3. Confirm Resources Access¶

You now have access to the resources created under the TA troubleshooting subscription. Select All Resources from the home dashboard to confirm that the broken deployment resources are available for you to view:

11.5.3.1.4. Setup AZ CLI¶

First we need to clear the AZ CLI cache:

> az account clear

Now we need to login again which will present us with the form to authenticate:

> az login

Because you selected Stay signed in, from the previous step, it will default to your account within the TA tenant directory. All you need to do is select your name from the list:

Back in your PowerShell Terminal you will see your account information output:

> az login
# output example
[
   {
      ...trimmed...
      "id": "095dea07-a8e5-4bd1-ba75-54d61d581524",
      "name": "Troubleshooting - TA <Name>",
      "user": {
         "name": "[email protected]",
         "type": "user"
      }
      ...trimmed...
   }
]

After configuring the AZ CLI to use the new subscription, let’s set up our AZ CLI defaults for the correct resource group and virtual machine:

> az configure -d group=linux-ts-rg vm=broken-linux-vm

You can verify everything worked by looking at the default VM. It should be identical to your group-mates and TA:

> az vm show

11.5.3.2.1. Import the Final Postman Collection¶

You can import this collection by selecting the Import button and then selecting the Link tab in the import window. Paste in the following link then select Continue:

• postman collection link (GitHub gist file)

11.5.3.2.2. Update Access Token Settings¶

After importing, you will need to update your access token settings to use the following pre-defined variables (from top to bottom of the access token form). As a reminder, you can access this by selecting the three dots next to the collection name, selecting Edit then from the Authorization tab select Get New Access Token:

• Token Name: {{TokenName}}
• Redirect URL: {{RedirectURL}}
• Auth URL: {{AuthURL}}
• Client ID: {{ClientID}}
• Scopes: {{Scopes}}
• State: {{State}}

After updating the form your settings should match the image below:

You can now request an access token and create a new account in this shared AADB2C tenant. After receiving your access token, leave the edit collection window open.

11.5.3.2.3. Update the baseURL Variable¶

From the Authorization tab select the Variables tab. Then (as seen in a previous studio) in the Current Value entry on the right side replace the current value, https://localhost:5001, with the public IP address of your group’s VM: