On 11th December 2024, Bulgarian weightlifter Karlos Nasar achieved something extraordinary again by setting two new World Records in the 89kg category: a 183kg Snatch and 405kg Total.

It was amazing!

Sometimes Android wireless pairing from Device Manager UI does not work. This may happen because of WIFI network settings, which are not under our control (i.e. co-working space, cafe, hotel).

A fix that may work is using the adb CLI commands for manual pairing.

1. Go to phone Settings->Developer Options->Wireless Debugging and click Pair device with pairing code.

2. A dialog shows Pair Code plus device IP address and port.

Note that IP:PORT in this dialog is different from your local device IP:PORT reported on the previous screen. We’ll use the other one later. For pairing we use must use this one in the Dialog window.

3. Open terminal on development computer and type:

1

adb pair IP:PORT

4. Enter Pairing Code displayed on your phone screen. Pairing should succeed.

5. Now connect to the device by using the other port number displayed before the Pairing dialog popup.

1

adb connect IP:PORT

Now phone should be paired successfully with the dev computer.

Install or download yt-dlp

Open terminal.

MP3 (Audio)

1

yt-dlp --verbose --extract-audio --audio-format mp3 --audio-quality 0 https://www.youtube.com/watch\?v\=9NbeL4FtcN0

Audio quality is number from 0 (best) to 10 (worst).

MP4 (Video)

1

yt-dlp --verbose -S "res:720" -f "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best" https://www.youtube.com/watch\?v\=9NbeL4FtcN0

Works for Facebook videos too with URL like: https://www.facebook.com/watch/?v=280191971840839

Recently I’ve been working on a small Go library.

I wrote unit tests to cover most of the major functionality.

I got code coverage 33%

And I became suspicious.

Why 33%?

Because go test -race -coverprofile=coverage.out ./... includes all auto-generated Fake objects used by unit tests in the calculation. Depending on how many and how large the fake objects are, real code coverage is reduced by that much. After removing fakes from the calculation, code coverage was raised to 62%!

How we measure code coverage in Gitlab CI

1
2
3
4
5
6
7

unit tests:
  image: golang:1.22.2
  stage: test
  script:
    - go test -race -coverprofile=coverage.out ./...
    - go tool cover -func=coverage.out
  coverage: '/total:\s+\(statements\)\s+(\d+.\d+\%)/'

This will include all Fake objects code and will reduce the real code coverage of our tests.

How it should be

If we want to use go tool cover and our fake files start with the prefix fake_ (i.e. counterfeiter):

1
2
3
4
5
6
7
8

unit tests:
  image: golang:1.22.2
  stage: test
  script:
    - go test -race -coverprofile=coverage.out.tmp ./...
    - cat coverage.out.tmp | grep -v "fake_" > coverage.out
    - go tool cover -func=coverage.out
  coverage: '/total:\s+\(statements\)\s+(\d+.\d+\%)/'

Conclusion

If you made the same mistake, you must know that your code coverage is much higher than you, and everybody else think it is :)

Cheers!

I recently learned an interesting feature in the Goa framework, which we use for some of our services. How can we return a ZIP file which is created by a service function implementation, but also include additional custom headers in the response?

Big thanks to one of the Goa creators/contributors Raphael Simon, who helped me with this issue very quickly in the Goa Slack channel!

Encoding bytes

Goa provides default response encoders for JSON, XML, GOB and plain text/bytes. In my case the plain bytes response was almost enough, but I wanted to have additional response headers returned to the client.

1
2
3
4
5
6
7
8
9

	Method("ExportZip", func() {
		Description("Export a signed zip file.")
		Payload(ExportZipRequest)
		Result(Bytes)
		HTTP(func() {
			GET("/{path}/export")
			Response(StatusOK)
		})
	})

As I wanted to have Content-Type: application/zip header, I wrote a small encoder using the default TextEncoder as example and everything worked fine.

What’s the problem

I wanted to have one additional response header specifying the name of the ZIP file which was dependent on some internal resource characteristics. More specific, I wanted to include a response header Content-Disposition: attachment; filename="myfile.zip" where myfile.zip could be different with every request.

The simple text/bytes encoder cannot do this, because there’s no way I can give it more specific values from the Service implementation - I can only return []byte slice as response.

With Goa, if you need to return more specific headers, you have to define them in a Goa DSL response type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

	var ExportZipResult = Type("ExportZipResult", func() {
		Field(1, "body", Bytes, "Body is the zip file bytes.")
		Field(2, "content-disposition", String, "Content-Disposition response header containing the name of the file.")
		Required("body", "content-disposition")
	})


	Method("ExportZip", func() {
		Description("Export a signed zip file.")
		Payload(ExportZipRequest)
		Result(ExportZipResult)
		HTTP(func() {
			GET("/{path}/export")
			Response(StatusOK, func(){
				Header("content-disposition")				
			})
		})
	})

Now my small custom bytes encoder receives this object, but the body is wrapped in a generated wrapper object which I must cast to the specific ExportZipResult type in order to get the body bytes. It didn’t seem natural as it means that this bytes encoder will only be usable for that specific function and won’t be useful for other similar functions. That’s when I decided to write in the Slack channel and to my surprise the creator of Goa responded within a few hours.

The solution

It turned out that Goa provides a custom DSL function which instructs the code generation process to skip the creation of any response encoder for that function. Instead, you must implement a function which returns an object containing all header values which you’d like to return, plus an io.ReadCloser for the bytes that you want to return to the client.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

	var ExportZipResult = Type("ExportZipResult", func() {
		Field(1, "content-type", String, "Content-Type response header.")
		Field(2, "content-disposition", String, "Content-Disposition response header containing the name of the file.")
		Required("content-type", "content-disposition")
	})


	Method("ExportZip", func() {
		Description("Export a signed zip file.")
		Payload(ExportZipRequest)
		Result(ExportZipResult)
		HTTP(func() {
			GET("/{path}/export")

			// bypass response body encoder code generation, so that
			// an io.ReadCloser can be returned to the client
			// while specific response headers can be specified
			// in the ExportZipResult type.
			SkipResponseBodyEncodeDecode()

			Response(StatusOK, func(){
				Header("content-type")
				Header("content-disposition")				
			})
		})
	})

Now there is no need for writing any custom Encoder and the function that implements the creation of the ZIP archive looks like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

func (s *Service) ExportZip(ctx context.Context, req *mysvc.ExportZipRequest) (*mysvc.ExportZipResult, io.ReadCloser, error) {
	// prepare the ZIP archive []byte slice
	myzip := createSignedZip(...) 

	// prepare custom header value
	contentDisposition := fmt.Sprintf(...)

	return &mysvc.ExportZipResult{
		ContentType: "application/zip",
		ContentDisposition: contentDisposition,
	}, io.NopCloser(bytes.NewReader(myzip)), nil
}

And it works like charm! :)

Imagine you have a struct with some fields which are always there, but you also have some fields which are dynamically populated with different [key, value] pairs, which cannot be described with static types upfront.

Example Record

1
2
3
4
5
6
7

type Order struct {
    ID         string                 `json:"id"`
	ItemID     string                 `json:"item_id"`
    Category   string                 `json:"category"`
    Timestamp  int64                  `json:"timestamp"`
    Attributes map[string]interface{} `json:"attributes"`
}

The attributes field is map and may contain an arbitrary number of values for each record. This is the field that I want to store in JSON column. The other fields are stored in statically defined columns with their respective types.

First attempt

BigQuery infer schema does not work with maps. So doing something like this won’t work.

1

schema, err := bigquery.InferSchema(Order{})

Even if you make the Attributes field type json.RawMessage it won’t work again.

Second attempt

Manually create the schema and try again.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

schema := bigquery.Schema{
    {Name: "id", Type: bigquery.StringFieldType},
    {Name: "item_id", Type: bigquery.StringFieldType},
    {Name: "category", Type: bigquery.StringFieldType},
    {Name: "timestamp", Type: bigquery.IntegerFieldType},
    {Name: "attributes", Type: bigquery.JSONFieldType},
}

// ... create the table

err := table.Inserter().Put(ctx, order)

This won’t work (with the Golang client) as BigQuery won’t convert the map to JSON. If we change the field type to json.RawMessage it won’t work again.

Third attempt is OK

We make the Order type implement the bigquery.ValueSaver interface and serialize the attributes field as JSON string.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

type OrderValueSaver struct {
    order *Order
}

// Save implements the bigquery.ValueSaver interface 
// to serialize the "attributes" field as JSON string.
func (s *OrderValueSaver) Save() (row map[string]bigquery.Value, insertID string, err error) {
    attr, err := json.Marshal(s.order.Attributes)
    if err != nil {
        return nil, "", err
    }

    row = map[string]bigquery.Value{
        "id":         s.order.ID,
        "item_id":    s.order.ItemID,
        "category":   s.order.Category,
		"timestamp":  s.order.Timestamp,
        "attributes": string(attr),
    }

    return row, s.order.ID, nil
}

func (c *Client) SaveOrder(ctx context.Context, order *Order) error {
    schema := bigquery.Schema{
        {Name: "id", Type: bigquery.StringFieldType},
        {Name: "item_id", Type: bigquery.StringFieldType},
        {Name: "category", Type: bigquery.StringFieldType},
        {Name: "timestamp", Type: bigquery.IntegerFieldType},
        {Name: "attributes", Type: bigquery.JSONFieldType},
    }
	
	// ... create/get table, ideally do it in the constructor of the client once 
	
	saver := &OrderValueSaver{order: order}
    
	return table.Inserter().Put(ctx, saver)
}

Ebitengine is a Go game development engine. Games can be built for Linux, MacOS, Windows, Android, iOS and Nintendo Switch.

I did some basic experiments during a few vacation days. This post is about something which wasn’t obvious to me how to implement and took me some hours to figure out. Hopefully it may spare you some time.

Under the hood Ebitengine uses gomobile to create bindings for languages like Java and C/C++.

This article is especially interesting if you need to call Go functions from Java or C/C++ and vice-versa.

What is the problem

I make a Go game which runs on Android, but I’d like to show Ads using 3rd party SDK like Unity or Facebook Audience Network. This means I’ll have to integrate a Java or ObjC library in the native app. For example, on Android it could be implemented in the showAds method below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

import go.Seq;
import com.mycompany.mygame.mobile.EbitenView;

public class MainActivity extends AppCompatActivity {
    protected EbitenView gameView;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        Seq.setContext(getApplicationContext());

        gameView = new EbitenView(getApplicationContext());
        setContentView(gameView);
    }

    @Override
    protected void onPause() {
        super.onPause();
        gameView.suspendGame();
    }

    @Override
    protected void onResume() {
        super.onResume();
        gameView.resumeGame();
    }

    public void showAds() {
    	...
    }
}

The problem is that MainActivity doesn’t know when to call show ads, as it doesn’t know anything about the internal state of the game - game updates and state changes happen in the Go code. So, at any point in time, only the Go side knows if it’s time to show an advert. Now the problem comes down to how the Go code can call the showAds() Java method?

Solution

My current solution is presented below and it works. I don’t feel it optimal since it requires to keep a global variable reference to the game object, but for now I haven’t found an improvement (and I don’t know if it’s an issue at all).

It shows how to call Go functions from Java as well as Java functions from Go.

It looks like neither Hyperledger Aries Go framework, nor the Google Tink crypto library, which it uses under the hood, provide helper functions to convert public key bytes to JWK format (or I haven’t found the way). When you export public key bytes from the KSM, they are unmarshalled from their internal representation and serialized using x509.MarshalPKIXPublicKey(), elliptic.Marshal(), and some are marshalled as JSON using json.Marshal() to custom Aries PublicKey struct. This means that we can’t use standard Go JOSE libraries to convert these public keys to JWK directly.

This post shows how to make two kinds of keys to JWK, but I’ll try to explain the process I followed to discover them, so if you need to export other public key types, you’ll probably know where to look.

One of the key types - NISTP256ECDHKWType - is mandatory in the current implementation of Hyperledger Aries Go if you want to use authenticated encryption (authcrypt) of DIDComm messages and follow the DIDComm V2 spec.

Why would you want to convert Aries public keys to JWK?

To use them as verification methods in DID documents. It is a common and widely supported encoding of public keys in DID documents (publicKeyJwk)

Below are example functions to convert ECDSA and ECDSA-ECDH-KW keys:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109

type JWK struct {
	Kty string `json:"kty"`
	Crv string `json:"crv"`
	X   string `json:"x"`
	Y   string `json:"y"`
	Kid string `json:"kid"`
}

func main() {
	// Aries KMS is created and initialized and you have a reference to it.
	// ...

	keyID := "..."
	pubkey, keyType, err := ariesKMS.ExportPublicKeyBytes(keyID)
	if err != nil {
		log.Fatalln(err)
	}

	jwk, err := PublicKeyToJWK(keyID, keyType, pubKey)
	if err != nil {
		log.Fatalln(err)
	}

	jwkBytes, err := json.Marshal(jwk)
	if err != nil {
		log.Fatalln(err)
	}

	log.Printf("jwkBytes = %#+v\n", jwkBytes)
}


// PublicKeyToJWK converts public key bytes from the binary format of the Tink
// key to a JWK.
func PublicKeyToJWK(keyID string, kt kms.KeyType, pubKey []byte) (interface{}, error) {
	switch kt {
	case kms.ECDSAP256TypeIEEEP1363, kms.ECDSAP384TypeIEEEP1363, kms.ECDSAP521TypeIEEEP1363:
		return ecKeyToJwk(keyID, kt, pubKey)
	case kms.NISTP256ECDHKWType, kms.NISTP384ECDHKWType, kms.NISTP521ECDHKWType:
		return ecdhKWToJwk(keyID, pubKey)
	default:
		return nil, fmt.Errorf("unsupported key type: %v", kt)
	}
}

func ecKeyToJwk(keyID string, kt kms.KeyType, pubKey []byte) (interface{}, error) {
	var x, y *big.Int
	var curve elliptic.Curve

	switch kt {
	case kms.ECDSAP256TypeIEEEP1363:
		curve = elliptic.P256()
		x, y = elliptic.Unmarshal(curve, pubKey)
	case kms.ECDSAP384TypeIEEEP1363:
		curve = elliptic.P384()
		x, y = elliptic.Unmarshal(curve, pubKey)
	case kms.ECDSAP521TypeIEEEP1363:
		curve = elliptic.P521()
		x, y = elliptic.Unmarshal(curve, pubKey)
	default:
		return nil, fmt.Errorf("unsupported key type: %v", kt)
	}

	xEncoded := base64.RawURLEncoding.EncodeToString(x.Bytes())
	yEncoded := base64.RawURLEncoding.EncodeToString(y.Bytes())

	jwk := JWK{
		Kty: "EC",
		Crv: curve.Params().Name,
		X:   xEncoded,
		Y:   yEncoded,
		Kid: keyID,
	}

	return jwk, nil
}

func ecdhKWToJwk(keyID string, pubKey []byte) (interface{}, error) {
	var ariesKey cryptoapi.PublicKey
	if err := json.Unmarshal(pubKey, &ariesKey); err != nil {
		return nil, err
	}

	xEncoded := base64.RawURLEncoding.EncodeToString(ariesKey.X)
	yEncoded := base64.RawURLEncoding.EncodeToString(ariesKey.Y)

	jwk := JWK{
		Kty: ariesKey.Type,
		Crv: altCurveName(ariesKey.Curve),
		X:   xEncoded,
		Y:   yEncoded,
		Kid: keyID,
	}

	return jwk, nil
}

func altCurveName(name string) string {
	switch name {
	case "NIST_P256":
		return "P-256"
	case "NIST_P384":
		return "P-384"
	case "NIST_P521":
		return "P-521"
	default:
		return name
	}
}

How to find serializations for other supported key types?

By looking at the public key serialization process for your needed key type.

Keys are created by different KeyManager implementations.

Public keys are serialized in the pubkey_writer.go file. Best to checkout the Hyperledger Aries Go github repo and dive into this file and follow the different serialization functions for the key types you need.

We’ve been using go-watcher for local development with docker-compose to automatically rebuild a given service when its source code changes. From Go 1.20+ onwards the go-watcher binary no longer works, because it’s using a build flag -i which has been deprecated for a long time, but now is finally removed. Because of that the compilation fails with error. See it here.

The project is no longer maintained, and it seems that the authors are not going to update it.

Now I’m using a replacement called guard. Usage is almost identical as you can see in the examples below.

Note: we commit vendor, that’s why we build/install with -mod=vendor

Example Dockerfile using go-watcher to watch and rebuild a service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

FROM golang:1.19.5

RUN go install github.com/canthefason/go-watcher/cmd/[email protected]

ADD . /go/src/gitlab.com/myuser/myproject

WORKDIR /go/src/gitlab.com/myuser/myproject

RUN go install -mod=vendor ./cmd/svcmain/...

EXPOSE 8080

ENTRYPOINT ["sh", "-c", "/go/bin/watcher -run gitlab.com/myuser/myproject/cmd/svcmain -watch gitlab.com/myuser/myproject"]

Example Dockerfile using guard to watch and rebuild a service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

FROM golang:1.20.1

RUN go install github.com/ysmood/kit/cmd/[email protected]

ADD . /go/src/gitlab.com/myuser/myproject

WORKDIR /go/src/gitlab.com/myuser/myproject

EXPOSE 8080

ENTRYPOINT ["sh", "-c", "/go/bin/guard -w '**/*.go' -- go run -mod=vendor ./cmd/svcmain/..."]

Display all commit messages since the last tag:

1

git log --no-merges --oneline --format=%s $(git describe --tags --abbrev=0)..HEAD