Goa Framework Zip File Response

I recently learned an interesting feature in the Goa framework, which we use for some of our services. How can we return a ZIP file which is created by a service function implementation, but also include additional custom headers in the response?

Big thanks to one of the Goa creators/contributors Raphael Simon, who helped me with this issue very quickly in the Goa Slack channel!

Encoding bytes

Goa provides default response encoders for JSON, XML, GOB and plain text/bytes. In my case the plain bytes response was almost enough, but I wanted to have additional response headers returned to the client.

1
2
3
4
5
6
7
8
9
	Method("ExportZip", func() {
		Description("Export a signed zip file.")
		Payload(ExportZipRequest)
		Result(Bytes)
		HTTP(func() {
			GET("/{path}/export")
			Response(StatusOK)
		})
	})

As I wanted to have Content-Type: application/zip header, I wrote a small encoder using the default TextEncoder as example and everything worked fine.

What’s the problem

I wanted to have one additional response header specifying the name of the ZIP file which was dependent on some internal resource characteristics. More specific, I wanted to include a response header Content-Disposition: attachment; filename="myfile.zip" where myfile.zip could be different with every request.

The simple text/bytes encoder cannot do this, because there’s no way I can give it more specific values from the Service implementation - I can only return []byte slice as response.

With Goa, if you need to return more specific headers, you have to define them in a Goa DSL response type.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
	var ExportZipResult = Type("ExportZipResult", func() {
		Field(1, "body", Bytes, "Body is the zip file bytes.")
		Field(2, "content-disposition", String, "Content-Disposition response header containing the name of the file.")
		Required("body", "content-disposition")
	})


	Method("ExportZip", func() {
		Description("Export a signed zip file.")
		Payload(ExportZipRequest)
		Result(ExportZipResult)
		HTTP(func() {
			GET("/{path}/export")
			Response(StatusOK, func(){
				Header("content-disposition")				
			})
		})
	})

Now my small custom bytes encoder receives this object, but the body is wrapped in a generated wrapper object which I must cast to the specific ExportZipResult type in order to get the body bytes. It didn’t seem natural as it means that this bytes encoder will only be usable for that specific function and won’t be useful for other similar functions. That’s when I decided to write in the Slack channel and to my surprise the creator of Goa responded within a few hours.

The solution

It turned out that Goa provides a custom DSL function which instructs the code generation process to skip the creation of any response encoder for that function. Instead, you must implement a function which returns an object containing all header values which you’d like to return, plus an io.ReadCloser for the bytes that you want to return to the client.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
	var ExportZipResult = Type("ExportZipResult", func() {
		Field(1, "content-type", String, "Content-Type response header.")
		Field(2, "content-disposition", String, "Content-Disposition response header containing the name of the file.")
		Required("content-type", "content-disposition")
	})


	Method("ExportZip", func() {
		Description("Export a signed zip file.")
		Payload(ExportZipRequest)
		Result(ExportZipResult)
		HTTP(func() {
			GET("/{path}/export")

			// bypass response body encoder code generation, so that
			// an io.ReadCloser can be returned to the client
			// while specific response headers can be specified
			// in the ExportZipResult type.
			SkipResponseBodyEncodeDecode()

			Response(StatusOK, func(){
				Header("content-type")
				Header("content-disposition")				
			})
		})
	})

Now there is no need for writing any custom Encoder and the function that implements the creation of the ZIP archive looks like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
func (s *Service) ExportZip(ctx context.Context, req *mysvc.ExportZipRequest) (*mysvc.ExportZipResult, io.ReadCloser, error) {
	// prepare the ZIP archive []byte slice
	myzip := createSignedZip(...) 

	// prepare custom header value
	contentDisposition := fmt.Sprintf(...)

	return &mysvc.ExportZipResult{
		ContentType: "application/zip",
		ContentDisposition: contentDisposition,
	}, io.NopCloser(bytes.NewReader(myzip)), nil
}

And it works like charm! :)

Insert data in BigQuery JSON Column

Imagine you have a struct with some fields which are always there, but you also have some fields which are dynamically populated with different [key, value] pairs, which cannot be described with static types upfront.

Example Record

1
2
3
4
5
6
7
type Order struct {
    ID         string                 `json:"id"`
	ItemID     string                 `json:"item_id"`
    Category   string                 `json:"category"`
    Timestamp  int64                  `json:"timestamp"`
    Attributes map[string]interface{} `json:"attributes"`
}

The attributes field is map and may contain an arbitrary number of values for each record. This is the field that I want to store in JSON column. The other fields are stored in statically defined columns with their respective types.

First attempt

BigQuery infer schema does not work with maps. So doing something like this won’t work.

1
schema, err := bigquery.InferSchema(Order{})

Even if you make the Attributes field type json.RawMessage it won’t work again.

Second attempt

Manually create the schema and try again.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
schema := bigquery.Schema{
    {Name: "id", Type: bigquery.StringFieldType},
    {Name: "item_id", Type: bigquery.StringFieldType},
    {Name: "category", Type: bigquery.StringFieldType},
    {Name: "timestamp", Type: bigquery.IntegerFieldType},
    {Name: "attributes", Type: bigquery.JSONFieldType},
}

// ... create the table

err := table.Inserter().Put(ctx, order)

This won’t work (with the Golang client) as BigQuery won’t convert the map to JSON. If we change the field type to json.RawMessage it won’t work again.

Third attempt is OK

We make the Order type implement the bigquery.ValueSaver interface and serialize the attributes field as JSON string.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
type OrderValueSaver struct {
    order *Order
}

// Save implements the bigquery.ValueSaver interface 
// to serialize the "attributes" field as JSON string.
func (s *OrderValueSaver) Save() (row map[string]bigquery.Value, insertID string, err error) {
    attr, err := json.Marshal(s.order.Attributes)
    if err != nil {
        return nil, "", err
    }

    row = map[string]bigquery.Value{
        "id":         s.order.ID,
        "item_id":    s.order.ItemID,
        "category":   s.order.Category,
		"timestamp":  s.order.Timestamp,
        "attributes": string(attr),
    }

    return row, s.order.ID, nil
}

func (c *Client) SaveOrder(ctx context.Context, order *Order) error {
    schema := bigquery.Schema{
        {Name: "id", Type: bigquery.StringFieldType},
        {Name: "item_id", Type: bigquery.StringFieldType},
        {Name: "category", Type: bigquery.StringFieldType},
        {Name: "timestamp", Type: bigquery.IntegerFieldType},
        {Name: "attributes", Type: bigquery.JSONFieldType},
    }
	
	// ... create/get table, ideally do it in the constructor of the client once 
	
	saver := &OrderValueSaver{order: order}
    
	return table.Inserter().Put(ctx, saver)
}

Ebiten Android and Go Callbacks

Ebitengine is a Go game development engine. Games can be built for Linux, MacOS, Windows, Android, iOS and Nintendo Switch.

I did some basic experiments during a few vacation days. This post is about something which wasn’t obvious to me how to implement and took me some hours to figure out. Hopefully it may spare you some time.

Under the hood Ebitengine uses gomobile to create bindings for languages like Java and C/C++.

This article is especially interesting if you need to call Go functions from Java or C/C++ and vice-versa.

What is the problem

I make a Go game which runs on Android, but I’d like to show Ads using 3rd party SDK like Unity or Facebook Audience Network. This means I’ll have to integrate a Java or ObjC library in the native app. For example, on Android it could be implemented in the showAds method below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

import go.Seq;
import com.mycompany.mygame.mobile.EbitenView;

public class MainActivity extends AppCompatActivity {
    protected EbitenView gameView;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        Seq.setContext(getApplicationContext());

        gameView = new EbitenView(getApplicationContext());
        setContentView(gameView);
    }

    @Override
    protected void onPause() {
        super.onPause();
        gameView.suspendGame();
    }

    @Override
    protected void onResume() {
        super.onResume();
        gameView.resumeGame();
    }

    public void showAds() {
    	...
    }
}

The problem is that MainActivity doesn’t know when to call show ads, as it doesn’t know anything about the internal state of the game - game updates and state changes happen in the Go code. So, at any point in time, only the Go side knows if it’s time to show an advert. Now the problem comes down to how the Go code can call the showAds() Java method?

Solution

My current solution is presented below and it works. I don’t feel it optimal since it requires to keep a global variable reference to the game object, but for now I haven’t found an improvement (and I don’t know if it’s an issue at all).

It shows how to call Go functions from Java as well as Java functions from Go.

Convert Aries KMS public key to JWK

It looks like neither Hyperledger Aries Go framework, nor the Google Tink crypto library, which it uses under the hood, provide helper functions to convert public key bytes to JWK format (or I haven’t found the way). When you export public key bytes from the KSM, they are unmarshalled from their internal representation and serialized using x509.MarshalPKIXPublicKey(), elliptic.Marshal(), and some are marshalled as JSON using json.Marshal() to custom Aries PublicKey struct. This means that we can’t use standard Go JOSE libraries to convert these public keys to JWK directly.

This post shows how to make two kinds of keys to JWK, but I’ll try to explain the process I followed to discover them, so if you need to export other public key types, you’ll probably know where to look.

One of the key types - NISTP256ECDHKWType - is mandatory in the current implementation of Hyperledger Aries Go if you want to use authenticated encryption (authcrypt) of DIDComm messages and follow the DIDComm V2 spec.

Why would you want to convert Aries public keys to JWK?

To use them as verification methods in DID documents. It is a common and widely supported encoding of public keys in DID documents (publicKeyJwk)

Below are example functions to convert ECDSA and ECDSA-ECDH-KW keys:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
type JWK struct {
	Kty string `json:"kty"`
	Crv string `json:"crv"`
	X   string `json:"x"`
	Y   string `json:"y"`
	Kid string `json:"kid"`
}

func main() {
	// Aries KMS is created and initialized and you have a reference to it.
	// ...

	keyID := "..."
	pubkey, keyType, err := ariesKMS.ExportPublicKeyBytes(keyID)
	if err != nil {
		log.Fatalln(err)
	}

	jwk, err := PublicKeyToJWK(keyID, keyType, pubKey)
	if err != nil {
		log.Fatalln(err)
	}

	jwkBytes, err := json.Marshal(jwk)
	if err != nil {
		log.Fatalln(err)
	}

	log.Printf("jwkBytes = %#+v\n", jwkBytes)
}


// PublicKeyToJWK converts public key bytes from the binary format of the Tink
// key to a JWK.
func PublicKeyToJWK(keyID string, kt kms.KeyType, pubKey []byte) (interface{}, error) {
	switch kt {
	case kms.ECDSAP256TypeIEEEP1363, kms.ECDSAP384TypeIEEEP1363, kms.ECDSAP521TypeIEEEP1363:
		return ecKeyToJwk(keyID, kt, pubKey)
	case kms.NISTP256ECDHKWType, kms.NISTP384ECDHKWType, kms.NISTP521ECDHKWType:
		return ecdhKWToJwk(keyID, pubKey)
	default:
		return nil, fmt.Errorf("unsupported key type: %v", kt)
	}
}

func ecKeyToJwk(keyID string, kt kms.KeyType, pubKey []byte) (interface{}, error) {
	var x, y *big.Int
	var curve elliptic.Curve

	switch kt {
	case kms.ECDSAP256TypeIEEEP1363:
		curve = elliptic.P256()
		x, y = elliptic.Unmarshal(curve, pubKey)
	case kms.ECDSAP384TypeIEEEP1363:
		curve = elliptic.P384()
		x, y = elliptic.Unmarshal(curve, pubKey)
	case kms.ECDSAP521TypeIEEEP1363:
		curve = elliptic.P521()
		x, y = elliptic.Unmarshal(curve, pubKey)
	default:
		return nil, fmt.Errorf("unsupported key type: %v", kt)
	}

	xEncoded := base64.RawURLEncoding.EncodeToString(x.Bytes())
	yEncoded := base64.RawURLEncoding.EncodeToString(y.Bytes())

	jwk := JWK{
		Kty: "EC",
		Crv: curve.Params().Name,
		X:   xEncoded,
		Y:   yEncoded,
		Kid: keyID,
	}

	return jwk, nil
}

func ecdhKWToJwk(keyID string, pubKey []byte) (interface{}, error) {
	var ariesKey cryptoapi.PublicKey
	if err := json.Unmarshal(pubKey, &ariesKey); err != nil {
		return nil, err
	}

	xEncoded := base64.RawURLEncoding.EncodeToString(ariesKey.X)
	yEncoded := base64.RawURLEncoding.EncodeToString(ariesKey.Y)

	jwk := JWK{
		Kty: ariesKey.Type,
		Crv: altCurveName(ariesKey.Curve),
		X:   xEncoded,
		Y:   yEncoded,
		Kid: keyID,
	}

	return jwk, nil
}

func altCurveName(name string) string {
	switch name {
	case "NIST_P256":
		return "P-256"
	case "NIST_P384":
		return "P-384"
	case "NIST_P521":
		return "P-521"
	default:
		return name
	}
}

How to find serializations for other supported key types?

By looking at the public key serialization process for your needed key type.

Keys are created by different KeyManager implementations.

Public keys are serialized in the pubkey_writer.go file. Best to checkout the Hyperledger Aries Go github repo and dive into this file and follow the different serialization functions for the key types you need.

go-watcher replacement for Go 1.20+

We’ve been using go-watcher for local development with docker-compose to automatically rebuild a given service when its source code changes. From Go 1.20+ onwards the go-watcher binary no longer works, because it’s using a build flag -i which has been deprecated for a long time, but now is finally removed. Because of that the compilation fails with error. See it here.

The project is no longer maintained, and it seems that the authors are not going to update it.

Now I’m using a replacement called guard. Usage is almost identical as you can see in the examples below.

Note: we commit vendor, that’s why we build/install with -mod=vendor

Example Dockerfile using go-watcher to watch and rebuild a service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
FROM golang:1.19.5

RUN go install github.com/canthefason/go-watcher/cmd/[email protected]

ADD . /go/src/gitlab.com/myuser/myproject

WORKDIR /go/src/gitlab.com/myuser/myproject

RUN go install -mod=vendor ./cmd/svcmain/...

EXPOSE 8080

ENTRYPOINT ["sh", "-c", "/go/bin/watcher -run gitlab.com/myuser/myproject/cmd/svcmain -watch gitlab.com/myuser/myproject"]

Example Dockerfile using guard to watch and rebuild a service.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
FROM golang:1.20.1

RUN go install github.com/ysmood/kit/cmd/[email protected]

ADD . /go/src/gitlab.com/myuser/myproject

WORKDIR /go/src/gitlab.com/myuser/myproject

EXPOSE 8080

ENTRYPOINT ["sh", "-c", "/go/bin/guard -w '**/*.go' -- go run -mod=vendor ./cmd/svcmain/..."]

Vault Dev Server in Docker Compose

I have difficulties using the Vault dev server mode in docker-compose environment. Other people have difficulties too.

Partly it’s due to incomplete/imperfect Vault documentation. I had to skim through the source code to understand what parameters can be used for some commands and for the dev server itself. Some of these were hard to find in the documentation.

Objective

I want Vault running in compose env with these features enabled on start:

  • automatically unsealed
  • enabled Vault Transit Engine
  • two signing keys in Transit Engine, one ECDSA and one ED25519 with names key1 and key2
  • predefined root token which can be embedded in all services needing to communicate with it

Here is a solution that works well.

TL;DR

Just use this. If you want to read some details follow below.

docker-compose.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
  vault:
    hostname: vault
    container_name: vault
    image: vault:1.10.3
    environment:
      VAULT_ADDR: "http://0.0.0.0:8200"
      VAULT_API_ADDR: "http://0.0.0.0:8200"
    ports:
      - "8200:8200"
    volumes:
      - ./volumes/vault/file:/vault/file:rw
    cap_add:
      - IPC_LOCK
    entrypoint: vault server -dev -dev-listen-address="0.0.0.0:8200" -dev-root-token-id="root"

  vault-init:
    container_name: vault-init
    image: vault:1.10.3
    volumes:
      - ./vault-init.sh:/vault-init.sh
    depends_on:
      - vault
    restart: "no"
    entrypoint: sh -c "/vault-init.sh"

vault-init.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#! /bin/sh

set -e

export VAULT_ADDR=http://vault:8200

# give some time for Vault to start and be ready
sleep 3

# login with root token at $VAULT_ADDR
vault login root

# enable vault transit engine
vault secrets enable transit

# create key1 with type ed25519
vault write -f transit/keys/key1 type=ed25519

# create key2 with type ecdsa-p256
vault write -f transit/keys/key2 type=ecdsa-p256

That’s it.

Details

The dev server is always unsealed by default, so our first objective is achieved.

Convert Vault Public Key to JWK

Hashicorp Vault Transit Engine API allows signing of data and verification of signatures. It offers more features like encryption/decryption, hashing, random bytes generation, but in this post we’re concerned only with the signing part of the API and more specifically with converting public keys returned by the Vault API to JWK format.

Example response when fetching a public key from Vault:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
curl -X GET https://myvault.com/v1/transit/keys/keyname

{
    "data":
    {
        "keys":
        {
            "1":
            {
                "creation_time": "2022-06-13T12:21:27.257274629Z",
                "name": "P-256",
                "public_key": "-----BEGIN PUBLIC KEY-----\nMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAERTx/2cyYcGVSIRP/826S32BiZxSg\nnzyXgRYmKP8N2l26ec/MwCdsHIEyraX1ZYqwMUT4wO9fqFiGsRKyMBpPnQ==\n-----END PUBLIC KEY-----\n"
            }
        },
        "latest_version": 1,
        "type": "ecdsa-p256"
    }
}

Convert public key to JWK

Some keys like ED25519 are just random bytes and are returned as base64.StdEncoding string, while other key types like ECDSA and RSA have some structure and are returned as PEM encoded keys.

Below is an example function that creates a jose.JSONWebKey from a Vault public key.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
package example

import (
    "crypto/ed25519"
    "crypto/x509"
    "encoding/pem"
    "fmt"

    "github.com/square/go-jose/v3"
)	

func jwkFromPEM(keyname string, keytype string, pemKey string) (*jose.JSONWebKey, error) {
    var key interface{}
	
    switch keytype {
    case "ed25519":
        pk, err := base64.StdEncoding.DecodeString(pemKey)
        if err != nil {
            return nil, fmt.Errorf("failed to decode ed25519 key: %v", err)
        }
        key = ed25519.PublicKey(pk)
    case "ecdsa-p256", "ecdsa-p384", "ecdsa-p521", "rsa-2048", "rsa-4096":
        block, _ := pem.Decode([]byte(pemKey))
        if block == nil {
            return nil, fmt.Errorf("no public key found during PEM decode")
        }
		
        var err error
        key, err = x509.ParsePKIXPublicKey(block.Bytes)
        if err != nil {
            return nil, err
        }
    default:
        return nil, fmt.Errorf("unsupported key type: %s", keytype)
    }
	
    return &jose.JSONWebKey{
        KeyID: keyname,
        Key: key,
    }, nil
}

Example JSON encoding of a converted JWK ECDSA-P256 public key would look like:

1
2
3
4
5
6
7
{
    "kid": "keyname",
    "kty": "EC",
    "crv": "P-256",
    "x": "RTx_2cyYcGVSIRP_826S32BiZxSgnzyXgRYmKP8N2l0",
    "y": "unnPzMAnbByBMq2l9WWKsDFE-MDvX6hYhrESsjAaT50"
}

Gitlab Subgroups and Go Modules Names

I sometimes see people who use private Gitlab subgroups to name their Go
modules (mostly libraries) with a suffix .git

Let’s see an example. If we have a Go module project at the following imaginary location:

1
https://gitlab.com/myuser/foogroup/barlib

Now when running go get gitlab.com/myuser/foogroup/barlib from another Go project, even though the SSH keys are set correctly, we may get an error:

1
The project you were looking for could not be found or you don't have permissions to view it.

We start googling and reading stackoverflow and it seems that a possible solution is to rename the module to gitlab.com/myuser/foogroup/barlib.git … and the issue looks fixed.

1
2
3
4
5
6
rm go.mod go.sum
go mod init gitlab.com/myuser/foogroup/barlib.git
go mod tidy

cd /go/src/another_project
go get gitlab.com/myuser/foogroup/barlib.git

Also at this point people start to doubt the goodness of the Go tools (go mod and go get especially) and mark this as a stupid annoying (Go) bug.

The solution is simple and the reason for the issue too and it’s not a bug.

The reason behind that permissions error

Why this happens is that go get tries to discover the modules at a given path in order to find the requested Go module repository. Only after the repository is found, the tools will do git clone or git checkout and the SSH keys will be used for authentication. The issue comes down to the fact that private Gitlab subgroups cannot be listed/viewed without a Gitlab Access Token.

The fix is Gitlab Access Token

Go to Gitlab->Preferences->Access Tokens and create one. Then create (or edit) a .netrc file in your home folder: ~/.netrc The contents of the file are below:

1
2
3
machine gitlab.com
    login my_gitlab_username
    password my_gitlab_token

And we may not need to name our Go modules with a .git suffix anymore.