Get your own NeoFS HTTP Gateway

When we showed you our hilarious cat available via HTTP gateway a month ago, the gateway wasn’t open-source yet. We took some time to work out the kinks and finally released it to the public right at the end of April. Now you can enjoy it yourself — run it either as a part of a backend system for your dApp smart contract (maybe even N3 hackathon smart contract), asset storage for a traditional web application, or just for fun.

Obtain

The gateway lives on GitHub and you can grab the latest release in a standard way (single binary). However, if you prefer dockerized setup, we have this option covered as well with Docker Hub.

Run

HTTP gateway is exactly that — just a gateway between HTTP and NeoFS protocol; so it relies on NeoFS nodes for data access. It means that you need to provide it with NeoFS node address that will be used for request processing. This can be done either via -p parameter or via HTTP_GW_PEERS_<N>_ADDRESS and HTTP_GW_PEERS_<N>_WEIGHT environment variables. The gateway also supports specifying multiple NeoFS nodes with weighted load balancing, which can be used for more complex setups.

These two commands are functionally equivalent, they run the gate with one backend node from neofs-dev-env:

$ neofs-http-gw -p 192.168.130.72:8080
$ HTTP_GW_PEERS_0_ADDRESS=192.168.130.72:8080 neofs-http-gw

This is enough to provide you with the read access to public containers via default port 8082.

Access

To get an object via HTTP, you can use /get/$CID/$OID path where $CID is a container ID and $OID is an object ID. If your cat is stored as 2m8PtaoricLouCn5zE8hAFr3gZEBDCZFe9BEgVJTSocY in Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ you can obtain it as following

$ wget http://localhost:8082/get/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/2m8PtaoricLouCn5zE8hAFr3gZEBDCZFe9BEgVJTSocY

Yet, sometimes you want to get more user-friendly access, especially if you’re developing an application that needs to take assets from NeoFS and present them to user. You may have lots of named objects, and you don’t always know what their object IDs are. NeoFS objects don’t really have names. Otherwise, they have attributes, and you can set as many attributes for an object as you wish. Some of them are so-called “well-known” attributes, like FileName which is used to store object’s file name. But sky is the limit here! Any kind of string can be an attribute name, and any kind of string can be its value (beware of HTTP limitations, though, and try to fit all of this into regular ASCII).

So if you have attributes set for your objects, you can also use /get_by_attribute/$CID/$ATTRIBUTE_NAME/$ATTRIBUTE_VALUE path to get them, where $CID is still a container ID while $ATTRIBUTE_NAME and $ATTRIBUTE_VALUE are the name and the value of the attribute you’re looking for. The gateway will answer with the first object matching this name-value pair (technically, there can be multiple objects matching this criterion).

If your 2m8PtaoricLouCn5zE8hAFr3gZEBDCZFe9BEgVJTSocY object from the example above has FileName attribute set to cat.jpeg you can get it with the following request:

$ wget http://localhost:8082/get_by_attribute/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/FileName/cat.jpeg

If it has “Ololo” attribute with “100500” value, you can get it this way too:

$ wget http://localhost:8082/get_by_attribute/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ/Ololo/100500

Wrap

You may have noticed that the API provided by the gateway natively differs a little from the URL scheme given in our previous post. That’s because we expect most of the real setups to be using some proxy server (or servers) in front of the gateway, providing caching, URL rewriting, and TLS termination if needed (although TLS can be handled by the gateway itself if needed). For example, in the previous article, we’ve used nginx as a caching proxy with some URL rewriting to make access to objects more web-native.

First, we made direct pass of simple GET request to /get/ location:

location  ~ "^/[0-9a-zA-Z]{43,44}/[0-9a-zA-Z\-]{43,44}$" {
  rewrite /(.*) /get/$1 break;
  proxy_pass http://127.0.0.1:8082;
  proxy_cache neofs_cache;
  proxy_cache_methods GET;
 }

Then, we’d like to have filename-based access to objects:

location / {
  rewrite '/([0-9a-zA-Z]{43,44})/(.*)' /get_by_attribute/$1/FileName/$2 break;
  proxy_pass http://127.0.0.1:8082;
  proxy_cache neofs_cache;
  proxy_cache_methods GET;
 }

And finally, add automatic fallback to index document, to have everything needed for decentralized static site hosting on NeoFS:

location ~ "^/[0-9a-zA-Z]{43,44}/$" {
  rewrite '/([0-9a-zA-Z]{43,44})/' /get_by_attribute/$1/FileName/index.html break;
  proxy_pass <http://127.0.0.1:8082;>
  proxy_cache neofs_cache;
  proxy_cache_methods GET;
 }

This thin layer makes accessing NeoFS easier and can accommodate any kind of URL scheme you need for your dApp/WebApp or provide compatibility for legacy and third-party components. At the same time, it allows you to use proven and well-known solutions for caching and using other HTTP-specific features.

Upload

NeoFS HTTP Protocol Gateway is not just for downloading things from NeoFS, you can also put new objects via /upload/$CID URL where $CID is a container ID. FileName attribute is set automatically in this case based on uploaded file name, but you can also manually add as many attributes as you like:

$ curl -F 'file=@cat.jpeg;filename=cat.jpeg' -H "X-Attribute-Ololo: 100500" http://localhost:8082/upload/Dxhf4PNprrJHWWTG5RGLdfLkJiSQ3AQqit1MSnEPRkDZ

You’d get container and object IDs in reply for this request.

This can always be done for public containers that allow anyone to put objects into. But most probably your dApp containers won’t be that permissive and this simple approach won’t work. You will need to explicitly add your gates’ keys into container’s Extended ACL rules to allow write access. Using eACL you can also limit access only to the gateways you trust. You can, of course, provide container owner’s keys for HTTP gateway to use, but that’s not really secure, and we don’t recommend this approach.

What you can do instead is use delegation mechanism known as Bearer Tokens provided by NeoFS. This mechanism allows application backend to issue such token, hand it over to the frontend part (for example, during authentication), and then frontend (like mobile app) to pass it to the gateway via Authorization header or __context_bearer_token_key cookie. Gateway doesn’t need anything special then, making it easy to replace gateways/setup new gateways, it just takes bearer token from the request and passes it along with NeoFS request. Then NeoFS takes it and does what it’s supposed to — reliably stores your kitties.

Have fun

So, NeoFS HTTP Protocol Gateway allows you to host your dApp data on NeoFS and access it from frontend and backend via standard HTTP. In future, we plan to launch a truly distributed CDN working with NeoFS gateways, so that there will be even more functional options for NeoFS integration with existing ecosystem.

Your smart contract can also have access to NeoFS via Oracle, which lets you end up with a truly decentralized application that has no single point of failure. Try it out and send us your feedback to ops@nspcc.ru or via GitHub!