# gos3dir

A lightweight S3-compatible server that uses a local directory as storage.
Perfect for local development, testing, and offline S3 workflows.
_Do not use in production._

## Installation

```bash
go install tangled.org/juanlu.space/gos3dir@latest
```

Or build from source:

```bash
git clone <repository-url>
cd gos3dir
go build
```

## Usage

Start the server by pointing it to a directory:

```bash
gos3dir /path/to/data/dir
```

The server listens on `http://localhost:8041`.

## Configure AWS CLI v2

Add a profile to `~/.aws/config`:

```ini
[profile gos3dir-dev]
endpoint_url = http://localhost:8041
s3 =
    addressing_style = path
```

Add dummy credentials to `~/.aws/credentials`:

```ini
[gos3dir-dev]
aws_access_key_id = test
aws_secret_access_key = test
```

## Examples

List buckets:
```bash
aws s3 ls --profile gos3dir-dev
```

Create a bucket:
```bash
aws s3 mb s3://my-bucket --profile gos3dir-dev
```

Upload a file:
```bash
aws s3 cp file.txt s3://my-bucket/ --profile gos3dir-dev
```

List objects in a bucket:
```bash
aws s3 ls s3://my-bucket --profile gos3dir-dev
```

Delete an object:
```bash
aws s3 rm s3://my-bucket/file.txt --profile gos3dir-dev
```

Delete an empty bucket:
```bash
aws s3 rb s3://my-bucket --profile gos3dir-dev
```

## Supported Operations

- List buckets (`GET /`)
- List objects (`GET /{bucket}`)
- Download objects (`GET /{bucket}/{key}`)
- Create bucket (`PUT /{bucket}`)
- Upload object (`PUT /{bucket}/{key}`)
- Delete empty bucket (`DELETE /{bucket}`)
- Delete object (`DELETE /{bucket}/{key}`)

## Lakehouse formats

This has been tested with Polars + Delta Lake:

```
In [25]: df.write_delta("s3://deltalake/df", storage_options={"AWS_ENDPOINT_URL": "http://localhost:8041", "AWS_ALLOW_HTTP": "true"})

In [26]: pl.read_delta("s3://deltalake/df", storage_options={"AWS_ENDPOINT_URL": "http://localhost:8041", "AWS_ALLOW_HTTP": "true"})
Out[26]:
shape: (4, 2)
┌─────┬─────┐
│ id  ┆ col │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 0   ┆ a   │
│ 1   ┆ b   │
│ 2   ┆ c   │
│ 3   ┆ d   │
└─────┴─────┘
```

## Future work

We would like to fix these at some point:
- Proper deletion of dangling "directories"
- Whatever is needed for open table formats (DuckLake, Apache Iceberg, Delta Lake)
  to work almost perfectly
- Virtual-hosted-style addressing (for now, only path-style addressing is supported)

If you see more gaps, feel free to open an issue.
But it might be deemed out of scope (see below).

## Limitations

These are by design and will not be fixed:
- No authentication or authorization
- No multipart uploads
- No versioning
- No `.` and `..` in full key names
- No bit-by-bit compatibility with Amazon S3 server responses