Simple S3-like server for development purposes, written in Go
Update README
+30
-3
+30
-3
README.md
+30
-3
README.md
···
83
83
84
84
- List buckets (`GET /`)
85
85
- List objects (`GET /{bucket}`)
86
+
- Download objects (`GET /{bucket}/{key}`)
86
87
- Create bucket (`PUT /{bucket}`)
87
88
- Upload object (`PUT /{bucket}/{key}`)
88
89
- Delete empty bucket (`DELETE /{bucket}`)
89
90
- Delete object (`DELETE /{bucket}/{key}`)
90
91
92
+
## Lakehouse formats
93
+
94
+
This has been tested with Polars + Delta Lake:
95
+
96
+
```
97
+
In [25]: df.write_delta("s3://deltalake/df", storage_options={"AWS_ENDPOINT_URL": "http://localhost:8041", "AWS_ALLOW_HTTP": "true"})
98
+
99
+
In [26]: pl.read_delta("s3://deltalake/df", storage_options={"AWS_ENDPOINT_URL": "http://localhost:8041", "AWS_ALLOW_HTTP": "true"})
100
+
Out[26]:
101
+
shape: (4, 2)
102
+
┌─────┬─────┐
103
+
│ id ┆ col │
104
+
│ --- ┆ --- │
105
+
│ i64 ┆ str │
106
+
╞═════╪═════╡
107
+
│ 0 ┆ a │
108
+
│ 1 ┆ b │
109
+
│ 2 ┆ c │
110
+
│ 3 ┆ d │
111
+
└─────┴─────┘
112
+
```
113
+
91
114
## Future work
92
115
93
116
We would like to fix these at some point:
94
-
- Downloads
95
117
- Proper deletion of dangling "directories"
118
+
- Whatever is needed for open table formats (DuckLake, Apache Iceberg, Delta Lake)
119
+
to work almost perfectly
120
+
- Virtual-hosted-style addressing (for now, only path-style addressing is supported)
121
+
122
+
If you see more gaps, feel free to open an issue.
123
+
But it might be deemed out of scope (see below).
96
124
97
125
## Limitations
98
126
99
127
These are by design and will not be fixed:
100
-
- Path-style addressing only (virtual-hosted-style not supported)
101
128
- No authentication or authorization
102
129
- No multipart uploads
103
130
- No versioning
104
-
- No ETags for uploaded objects
131
+
- No `.` and `..` in full key names
105
132
- No bit-by-bit compatibility with Amazon S3 server responses