SSH Local Port Forwarding

To forward requests coming to a local TCP port to another server [host:port] connected via SSH we can use a config file or direct command line arguments. To use a config file, it should be named config and placed inside the ~/.ssh folder. Here is a sample ~/.ssh/config file with some options:

1
2
3
4
5
6
Host my-ssh-connection
  Hostname 10.20.30.40
  User penkovski
  LocalForward 7070 localhost:8080
  LocalForward 7171 localhost:8181
  ControlMaster auto

This will forward all requests hitting the local (client) TCP port 7070 to the localhost:8080 on the server and all requests for local TCP port 7171 to the localhost:8181 on the server. The above configuration will be used automatically when initiating an SSH connection by the alias:

1
ssh my-ssh-connection

Update an XML Document Node with Go

In my humble opinion, modifying an XML Node with the Go standard xml package without describing all tags in data structures is not very intuitive. Hence this post is born. There are many ways to do it with their pros and cons. I’ll describe 2 ways. The first is simpler, but loses some XML elements like comments and possibly formatting. The other is more involved but preserves everything from the initial document, including comments and formatting.

Below is the XML document we’ll use as an example and the node we’ll modify is app>description>version. We’d like to set the version value from 1.0 to 2.0. Assume the XML file is called application.xml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
<app>
	<!-- application description -->
	<description>
		<name>My App</name>
		<version>1.0</version>
	</description>
	<!-- developer maintaining the application -->
	<developer>
		<name>Developer Name</name>
		<contacts>
			<email>developer@example.com</email>
			<mobile>+359888888888</mobile>
		</contacts>
	</developer>
	<!-- markets where the app is distributed -->
	<market>
		<googleplay>https://googleplay.com/url</googleplay>
		<appstore>https://appstore.com/url</appstore>
	</market>
</app>

Print stack traces for hanged Go program

If a program using channels or mutexes is stuck because of synchronization problems, we can send an abort signal to kill the process and print the stack traces of currently running go routines to debug where the program might have hanged. Find the PID of the running program with ps and type:

1
kill -ABRT <PID>

JSON unmarshal of time.Duration in Go

The post describes how to unmarshal time.Duration which is not supported by the default JSON unmarshaler, but the example can be used to unmarshal any custom type.

Here is the JSON we’re unmarshaling:

1
2
3
4
{
	"addr": "https://example.com",
	"timeout": "5s",
}

We’d like to convert the 5s value to a time.Duration value corresponding to 5 seconds and get the JSON to unmarshal in the following data structure:

1
2
3
4
type connection struct {
	Addr 	string		`json:"addr"`
	Timeout time.Duration 	`json:"timeout"`
}

One way to do it is to implement the Unmarshaler interface and handle the conversion in our UnmarshalJSON function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
// Unmarshaler is the interface implemented by types  
// that can unmarshal a JSON description of themselves.  
// The input can be assumed to be a valid encoding of  
// a JSON value. UnmarshalJSON must copy the JSON data  
// if it wishes to retain the data after returning.  
//  
// By convention, to approximate the behavior of Unmarshal itself,  
// Unmarshalers implement UnmarshalJSON([]byte("null")) as a no-op.  
type Unmarshaler interface {  
   UnmarshalJSON([]byte) error  
}

Below is the complete example. We unmarshal the JSON into a temprary structure where 5s is a standard string. Then we convert the string to time.Duration vaue and set it in our structure.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
package mypackage

import(
	"json/encoding"
	"time"
)

type connection struct {
	Addr 	string 		  `json:"addr"`
	Timeout time.Duration `json:"timeout"`
}

func (c *connection) UmarshalJSON(data []byte) (err error) {
	var tmp struct {
		Addr 	string
		Timeout string
	}
	if err = json.Unmarshal(data, &tmp); err != nil {
		return err
	}

	c.Addr = tmp.Addr
	c.Timeout, err = time.ParseDuration(tmp.Timeout)
	return err
}

Go program design for testing with counterfeiter

counterfeiter is a command-line tool for generating fake implementation of Go interfaces.

One of the best things we can do when writing Go programs is to design our components’ dependencies to be based on local interfaces. By local I mean interfaces which are needed and defined by a given component. Consider the following example: we have a http server, a service component and a database dependency.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
package main

func main() {
	// initialize some kind of database
	db := storage.New()
	
	// our service depends on a database
	svc := service.New(db)
	
	http.HandleFunc("/example", svc.Example)
	log.Fatal(http.ListenAndServe(":8080", nil))
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
package storage

type DB struct {}

type Data struct {}

func New() *DB {
	return &DB{}
}

func (db *DB) SomeData() (*Data, error) {
	... // return data from database
}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
package service

type Storage interface {
	SomeData() (*storage.Data, error)
}

type Data struct{
	Value int
}

type Service struct {
	db Storage
}

func New(db Storage) *Service {
	return &Service{db:db}
}

func (s *Service) Example(w http.ResponseWriter, r *http.Request) {
	// do something that requires using the database
	data, err := s.db.SomeData()
	if err != nil {
		...
	}

	// do something with data and write response
	...
}

The service component has a database dependency. It may be any kind of external object - client to another service, a cache/database, message queue, whatever. This dependency is expressed as an interface inside the service package. In effect, the service is telling its initialiser: when you create me, please give me an object which implements my service.Storage interface, cause that’s what I need to execute my functions.

This design has some nice benefits. The service component doesn’t know and care how the storage dependency is implemented. All it knows and cares about is that it has a SomeData() (*storage.Data, error) function. We can supply whatever object we wish to the service, as long as it implements the interface.

Here comes the counterfeiter

Java Random within Range

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
// randInt generates pseudo-random number 
// in the range [min...max] inclusive.
public static int randInt(int min, int max) 
{
    Random random = new Random();

    // nextInt is exclusive of the top value, 
    // so add 1 to make it inclusive
    return random.nextInt((max - min) + 1) + min;
}

Git SSH Authentication

If we want to use different authentication keys for a particular Git repo, we can edit the config file of the repo and add the sshCommand setting.

Navigate to the repository’s Git configuration directory:

1
cd /path/to/repo/.git

In the folder there’s a file called config. We are interested in the core section. To use a particular SSH key for Git, we would write the command as shown below. Replace the name of the key file with your own.

1
2
3
4
5
[core]
	...
	...
	...
	sshCommand = ssh -i ~/.ssh/my_custom_git_key

Perspective on Computer Hardware Advancements

note

Excerpt from The Clean Coder by Robert Martin

“One of the first machines I ever wrote programs for was a PDP-8/I. This machine had a 1.5-microsecond cycle time. It had 4096 12-bit words in core memory. It was the size of a refrigerator and consumed a significant amount of electrical power. It had a disk drive that could store 32K of 12-bit words, and we talked to it with a 10-character-per-second teletype. We thought this was a powerful machine, and we used it to work miracles.

I just bought a new Macbook Pro laptop. It has a 2.8GHz dual core processor, 8GB of RAM, a 512GB SSD, and a 17-inch 1920x1200 LED screen. I carry it in my backpack. It sits on my lap. It consumes less than 85 watts.

My laptop is eight thousand times faster, has two million times more memory, has sixteen million times more offline storage, requires 1% of the power, takes up 1% of the space, and costs one twenty-fifth of the price of the PDP-8/I.”

gcloud configuration and projects

note

Prerequisite: Google Cloud SDK should be installed. See the docs here.

1
2
3
4
5
6
7
8
9
$ gcloud version
Google Cloud SDK 250.0.0
app-engine-go
app-engine-python 1.9.86
beta 2019.05.17
bq 2.0.43
cloud-datastore-emulator 2.1.0
core 2019.06.07
gsutil 4.38

gcloud project configurations are typically stored in the following folder in files with prefix config_. For example: config_my-project-name

1
2
3
$ cd ~/.config/gcloud/configurations && ls -la
-rw-r--r--   1 penkovski  staff   83 Apr 19 09:17 config_default
-rw-r--r--   1 penkovski  staff   83 Apr 19 09:17 config_my-project-name

To add a new project configuration, run the gcloud init command and follow the instructions.

1
gcloud init

After successfully selecting the desired project and options, a new config file will appear in the configurations folder. To switch the active configuration for a new project use the following command:

1
gcloud config configurations activate my-project-name

Set the currently active project:

1
2
$ gcloud config set project my-project-name
Updated property [core/project].

Get the currently active project:

1
2
3
$ gcloud config get-value project
Your active configuration is: [my-project-name]
my-project-name

Golang Multi Package Test Coverage in Gitlab

As part of a Go unit testing CI job, we can generate code coverage statistics. For Gitlab to display them, we must provide appropriate output from the job’s script commands (stdout). Below is a sample CI test job:

1
2
3
4
5
6
unit-tests:
  image: golang:1.12
  stage: test
  script:
    - go test -race $(go list ./... | grep -v /vendor/) -v -coverprofile=coverage.out
    - go tool cover -func=coverage.out

On line 5 we run the go test command and specify that we would like to generate coverage profile and save it in a file called coverage.out

On line 6 we use the go tool cover command to generate (stdout) output broken down by function. Example output is shown below:

1
2
3
4
5
6
7
$ go tool cover -func=coverage.out
gitlab.com/myuser/myproject/file1.go:30:	exampleFunc1	100.0%
gitlab.com/myuser/myproject/file2.go:45:	exampleFunc2	100.0%
gitlab.com/myuser/myproject/pkg1/file3.go:89:	exampleFunc3	100.0%
gitlab.com/myuser/myproject/pkg2/file4.go:23:	exampleFunc4	95.0%
gitlab.com/myuser/myproject/pkg2/file5.go:134:	exampleFunc5	0.0%
total:						(statements)	65.3%