SurfStore

This is a cloud-based file storage service based on RAFT and consistent hashing called SurfStore. SurfStore is a networked file storage application that is modeled after Dropbox, and lets you sync files to and from the “cloud”. The system consits of the cloud service, and clients which interact with the service, via gRPC.

Multiple clients can concurrently connect to the SurfStore service to access a common, shared set of files. Clients accessing SurfStore “see” a consistent set of updates to files, but SurfStore does not offer any guarantees about operations across files, meaning that it does not support multi-file transactions (such as atomic move).

Protocol buffers

The gRPC service in SurfStore.proto haven been changed. You need to regenerate the gRPC client and server interfaces from our .proto service definition. We do this using the protocol buffer compiler protoc with a special gRPC Go plugin (The gRPC official documentation introduces how to install the protocol compiler plugins for Go).

protoc --proto_path=. --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative pkg/surfstore/SurfStore.proto

Running this command generates SurfStore.pb.go and SurfStore_grpc.pb.go in the pkg/surfstore directory.

Usage

1. Run your server using this:

go run cmd/SurfstoreServerExec/main.go -s <service> -p <port> -l -d (BlockStoreAddr*)

Here, service should be one of three values: meta, block, or both. This is used to specify the service provided by the server. port defines the port number that the server listens to (default=8080). -l configures the server to only listen on localhost. -d configures the server to output log statements. Lastly, (BlockStoreAddr*) are the BlockStore addresses that the server is configured with.

2. Run your client using this:

go run cmd/SurfstoreClientExec/main.go -d <meta_addr:port> <base_dir> <block_size>

3. Print block mapping using this:

go run cmd/SurfstorePrintBlockMapping/main.go -d <meta_addr:port> <base_dir> <block_size>

Examples:

Run the commands below on separate terminals (or nodes)
> go run cmd/SurfstoreServerExec/main.go -s block -p 8081 -l
> go run cmd/SurfstoreServerExec/main.go -s block -p 8082 -l
> go run cmd/SurfstoreServerExec/main.go -s meta -l localhost:8081 localhost:8082

The first two lines start two servers that services BlockStore interface and listens to localhost on port 8081 and 8082. The third line starts a server that services MetaStore interface, listens to localhost on port 8080, and references the BlockStore we created as the underlying BlockStore. (Note: if these are on separate nodes, then you should use the public ip address and remove -l)

2. From a new terminal (or a new node), run the client using the script provided in the starter code (if using a new node, build using step 1 first). Use a base directory with some files in it.

> mkdir dataA
> cp ~/pic.jpg dataA/ 
> go run cmd/SurfstoreClientExec/main.go localhost:8080 dataA 4096

This would sync pic.jpg to the server hosted on localhost:8080, using dataA as the base directory, with a block size of 4096 bytes.

3. From another terminal (or a new node), run PrintBlockMapping to check which blocks a block server has.

> go run cmd/SurfstorePrintBlockMapping/main.go localhost:8080 dataB 4096

The output willl be a map from block hashes to server names.

Makefile

Run BlockStore server:

$ make run-blockstore

Run RaftSurfstore server:

$ make IDX=0 run-raft

Test:

$ make test

Specific Test:

$ make TEST_REGEX=Test specific-test

Clean:

$ make clean