+3 -5

_freeze/posts/the-basics-of-duckdb-in-r/index/execute-results/html.json

reviewed

··· 1 1 { 2 2 - "hash": "1c6768c4188aa5e05a27a36f5e043069", 2 2 + "hash": "4f821c23d294514ddd79e006d14ce219", 3 3 "result": { 4 4 "engine": "knitr", 5 5 - "markdown": "---\ntitle: \"The basics of DuckDB in R\"\nauthor: Rory Lawless\ndate: 2025-03-30\nlastmod: 2026-01-01\nformat: html\n---\n\nOver the past year, [DuckDB](https://duckdb.org/docs/stable/clients/r) has gradually become an important part of my data science workflow - at first clumsily, then seamlessly. I don’t typically work with large datasets, however, integrating DuckDB has addressed some of my frustrations, especially when dealing with hardware limitations and moderately-sized but inefficiently stored data. With this in mind, here are two major benefits I’ve found since integrating DuckDB into my workflow.\n\n## Handling larger-than-memory data\n\nAs noted, I don't work with very large data often but I still run into annoying issues caused by repeated reloading of data after making mistakes. Now, this is not an issue for a .csv file containing a few hundred rows and, for larger files or those stored in legacy formats, I could add a \"backup\" step to my code, like so:\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndata <- read.csv(\"some-data-file.csv\")\ndata_backup <- data\n\n# Do some work on data, maybe make a mistake...\n\ndata <- data_backup\n```\n:::\n\n\nThis works fine, but I consider it an anti-pattern and ought to, in my opinion, be avoided. Instead of adding this extra step - likely increasing the memory used in the R session - you can use DuckDB to directly query files stored on disk, without having to load them into memory first.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(tidyverse)\nlibrary(duckdb)\n\n# Create a DuckDB connection\ncon <- dbConnect(duckdb::duckdb())\n\n# Write a SQL query to read data directly from the CSV file\ndata <- dbGetQuery(\n\tcon,\n\t\"SELECT col_1, col_2, col_4, col_10\n\tFROM 'some-data-file.csv'\n\tWHERE col_10 = 'some_value'\"\n)\n```\n:::\n\n\nThis may seem more complicated at first, and does require some knowledge of SQL, but it is a very efficient way of working with larger datasets, especially in the early stages when you're still exploring the data and working out what you're going to do with it.\n\n## {duckplyr}\n\nA game-changer for me, which really accelerated my adoption of DuckDB as a backend for processing data, was the [{duckplyr}](https://duckplyr.tidyverse.org) package. Those familiar with [{dbplyr}](https://dbplyr.tidyverse.org) will understand the theory behind this package; it allows queries to be built using the standard set of [{dplyr}](https://dplyr.tidyverse.org) functions, which are converted to SQL behind the scenes. \n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(tidyverse)\nlibrary(duckplyr)\n\n# Read CSV using DuckDB behind the scenes\ndata <- read_csv_duckdb(\"some-data-file.csv\")\n\n# Perform data manipulation using dplyr syntax\ndata <- data |>\n\tselect(col_1, col_2, col_4, col_10) |>\n\tfilter(col_10 == \"some_value\")\n```\n:::\n\n\nAside from the `read_csv_duckdb()` function, the rest of the code will be familiar to anyone who has used {dplyr} before. The main advantage of using {duckplyr} over writing SQL and using the [{DBI}](https://dbi.r-dbi.org) package is readability - using common {dplyr} functions makes it accessible to a wider range of users. This is a big benefit for teams where not everyone is comfortable reading or writing SQL.\n\nAdditionally, should the original author fall off the face of the earth, the code is still maintainable by others and readily adapted to eliminate the dependency on DuckDB.\n\n## Final thoughts\n\nDuckDB and R are a great combination, allowing me to overcome some of my (self-inflicted?) frustrations in my day-to-day data work. With {duckplyr}, querying data directly from files has smoothed out some of the rough edges in my workflow.\n", 6 6 - "supporting": [ 7 7 - "index_files" 8 8 - ], 5 5 + "markdown": "---\ntitle: \"The basics of DuckDB in R\"\ndate: 2025-03-30\nlastmod: 2026-01-01\nformat: html\n---\n\nOver the past year, [DuckDB](https://duckdb.org/docs/stable/clients/r) has gradually become an important part of my data science workflow - at first clumsily, then seamlessly. I don’t typically work with large datasets, however, integrating DuckDB has addressed some of my frustrations, especially when dealing with hardware limitations and moderately-sized but inefficiently stored data. With this in mind, here are two major benefits I’ve found since integrating DuckDB into my workflow.\n\n## Handling larger-than-memory data\n\nAs noted, I don't work with very large data often but I still run into annoying issues caused by repeated reloading of data after making mistakes. Now, this is not an issue for a .csv file containing a few hundred rows and, for larger files or those stored in legacy formats, I could add a \"backup\" step to my code, like so:\n\n\n::: {.cell}\n\n```{.r .cell-code}\ndata <- read.csv(\"some-data-file.csv\")\ndata_backup <- data\n\n# Do some work on data, maybe make a mistake...\n\ndata <- data_backup\n```\n:::\n\n\nThis works fine, but I consider it an anti-pattern and ought to, in my opinion, be avoided. Instead of adding this extra step - likely increasing the memory used in the R session - you can use DuckDB to directly query files stored on disk, without having to load them into memory first.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(tidyverse)\nlibrary(duckdb)\n\n# Create a DuckDB connection\ncon <- dbConnect(duckdb::duckdb())\n\n# Write a SQL query to read data directly from the CSV file\ndata <- dbGetQuery(\n\tcon,\n\t\"SELECT col_1, col_2, col_4, col_10\n\tFROM 'some-data-file.csv'\n\tWHERE col_10 = 'some_value'\"\n)\n```\n:::\n\n\nThis may seem more complicated at first, and does require some knowledge of SQL, but it is a very efficient way of working with larger datasets, especially in the early stages when you're still exploring the data and working out what you're going to do with it.\n\n## {duckplyr}\n\nA game-changer for me, which really accelerated my adoption of DuckDB as a backend for processing data, was the [{duckplyr}](https://duckplyr.tidyverse.org) package. Those familiar with [{dbplyr}](https://dbplyr.tidyverse.org) will understand the theory behind this package; it allows queries to be built using the standard set of [{dplyr}](https://dplyr.tidyverse.org) functions, which are converted to SQL behind the scenes. \n\n\n\n::: {.cell}\n\n```{.r .cell-code}\nlibrary(tidyverse)\nlibrary(duckplyr)\n\n# Read CSV using DuckDB behind the scenes\ndata <- read_csv_duckdb(\"some-data-file.csv\")\n\n# Perform data manipulation using dplyr syntax\ndata <- data |>\n\tselect(col_1, col_2, col_4, col_10) |>\n\tfilter(col_10 == \"some_value\")\n```\n:::\n\n\nAside from the `{{r}} read_csv_duckdb()` function, the rest of the code will be familiar to anyone who has used {dplyr} before. The main advantage of using {duckplyr} over writing SQL and using the [{DBI}](https://dbi.r-dbi.org) package is readability - using common {dplyr} functions makes it accessible to a wider range of users. This is a big benefit for teams where not everyone is comfortable reading or writing SQL.\n\nAdditionally, should the original author fall off the face of the earth, the code is still maintainable by others and readily adapted to eliminate the dependency on DuckDB.\n\n## Final thoughts\n\nDuckDB and R are a great combination, allowing me to overcome some of my (self-inflicted?) frustrations in my day-to-day data work. With {duckplyr}, querying data directly from files has smoothed out some of the rough edges in my workflow.\n", 6 6 + "supporting": [], 9 7 "filters": [ 10 8 "rmarkdown/pagebreak.lua" 11 9 ],

+3 -5

_freeze/posts/using-1password-secret-references-in-R/index/execute-results/html.json

reviewed

··· 1 1 { 2 2 - "hash": "c8d2ad5a4b0de04fbe8f87ac8fa15903", 2 2 + "hash": "688323e0ffc005182da1885cd69e6b3c", 3 3 "result": { 4 4 "engine": "knitr", 5 5 - "markdown": "---\ntitle: \"Using 1Password Secret References in R\"\nauthor: Rory Lawless\ndate: 2026-01-04\nformat: html\ndraft: true\n---\n\nIf you're like me, you store your API keys in your .Renviron file and forget about them. Not only is it risky behaviour to store them in plaintext, it is also a nightmarish way to manage and rotate out keys when needed.\n\n1Password offers a great solution managing your sensitive data, it even has a dedicated API Credentials item type for people super into taxonomy. The real magic, however, happens when you introduce yourself to [1Password CLI and its handy secret references](https://developer.1password.com/docs/cli/secret-reference-syntax) feature.\n\nI won't go into detail on installation and configuration, [the documentation does a better job than I would](https://developer.1password.com/docs/cli/get-started). Once you have migrated an API key to 1Password, you can refer to the secret using its URI instead of including it in plain text in your .Renvion or R script.\n\nThe URI of a credential comes in this format: `op://<vault-name>/<item-name>/[section-name/]<field-name>`. For example, an Anthropic API key save as an item named \"ClaudeAPI\" inside your 'Private' with the value in a field called 'credential' can be referred to as `op://Private/ClaudeAPI/credential` (note, the `[section-name/]` element is only required if the field is under a named section within the item).\n\nThe URI of your credential can then replace the plain text API key anywhere you were storing it. Using our ClaudeAPI, our key=value in .Renviron would look like:\n\n``` \nANTHROPIC_API_KEY=\"op://Private/ClaudeAPI/credential\"\n```\n\nOur key is stored inside our 1Password vault, safe from prying eyes and accidental exposure. To access this environment variable in a script, we would typically write something like `op://Private/Claude API Key/credential`, while this will technically run you will receive an error as you will be attempting to pass the literal secret reference to your API call. We will need to alter this code in two ways, the first is to use one of [the methods 1Password CLI provides](https://developer.1password.com/docs/cli/secrets-scripts) for loading secrets into code, the one we will use in our R script is `op read`. \n\nThe second change, calling the function `system2()` from base, which lets us run `op read` (or any system command) from our R script. \n\n\n::: {.cell}\n\n```{.r .cell-code}\nsystem2(\n\t\"op\",\n\targs = c(\"read\", shQuote(Sys.getenv(\"ANTHROPIC_API_KEY\"))),\n\tstdout = TRUE\n)\n```\n:::\n", 6 6 - "supporting": [ 7 7 - "index_files" 8 8 - ], 5 5 + "markdown": "---\ntitle: \"Using 1Password Secret References in R\"\ndate: 2026-01-02\nformat: html\n---\n\nIf you’re like me, you’ve stored your API keys in your .Renviron file and forget about them. Not only is it risky to store them in plaintext, it’s also a nightmarish way to manage and rotate out keys when needed.\n\n## 1Password to the rescue\n\n1Password offers a great solution for managing sensitive data. It even has a dedicated API Credentials item type for people super into taxonomy. The desktop and mobile apps are great for day-to-day internet usage but the real magic happens when you introduce yourself to [1Password CLI and its handy secret references](https://developer.1password.com/docs/cli/secret-reference-syntax) feature.\n\nI won't go into detail on installation and configuration, [the documentation does a better job than I ever could](https://developer.1password.com/docs/cli/get-started). However, once you have added an API key to 1Password, you can refer to it using its URI (also called a Secret Reference) instead of including it as plaintext in your .Renviron or R script. The URI of a credential comes in this format:\n\n``` {#uri-example style=\"text-align: center;\"}\nop://<vault-name>/<item-name>/[section-name/]<field-name>\n```\n\nFor example, an Anthropic API key saved as an item named \"ClaudeAPI\" inside our \"Private\" vault with the value in a field called \"credential\" can be referred to using the URI `op://Private/ClaudeAPI/credential` (note, the `[section-name/]` element is only required if the field is under a named section within the item). The URI of your credential can then replace the plain text key in your .Renviron or anywhere else it was lurking. Using our ClaudeAPI example, the entry in .Renviron would look like:\n\n``` {#envvar-example style=\"text-align: center;\"}\nANTHROPIC_API_KEY=\"op://Private/ClaudeAPI/credential\"\n```\n\n## Out of 1Password, into R\n\nOur key is now stored inside our 1Password vault, safe from prying eyes and accidental exposure. In our less secure days, we would typically run `{{r}} Sys.getenv(\"ANTHROPIC_API_KEY\")` to access the value of the environment variable in a script. While this still works, any API call that uses this value will likely fail as the value returned is the literal URI string, not the secret.\n\nTo address this, we need to alter the code in two ways. The first is to use one of [the methods 1Password CLI provides](https://developer.1password.com/docs/cli/secrets-scripts) for loading secrets into code, the one we will use in our R script is `op read`. This simply allows us to read the secret into our environment.\n\nThe second change, calling the function `{{r}} system2()` from {base}, lets us run `op read` (indeed, any system command) from our R script and returns the actual API key.\n\n\n::: {.cell}\n\n```{.r .cell-code}\nsystem2(\n\t\"op\",\n\targs = c(\"read\", shQuote(Sys.getenv(\"ANTHROPIC_API_KEY\"))),\n\tstdout = TRUE\n)\n\n# Note, our call to Sys.getenv() needs to be quoted using shQuote() to mitigate issues with spaces and special characters\n```\n:::\n\n\nWhen we execute the example code, 1Password will prompt us to approve access to the secret (illustrated in @fig-authimage). Once we click \"Authorize\", the secret value is returned as a character vector (setting `stdout = TRUE` in `system2()` makes this happen).\n\n{#fig-authimage fig-alt=\"A screenshot of a 1Password prompt to 'Allow Positron to get CLI access' with the option to 'Cancel' or 'Authorize'.\" width=\"390\"}\n\nWe can alter the above code to assign a name to the returned character vector or we can run the function call directly within the API call function. Furthermore, we can jettison `Sys.getenv()` entirely and pass the URI directly, like so:\n\n\n::: {.cell}\n\n```{.r .cell-code}\nsystem2(\n\t\"op\",\n\targs = c(\"read\", shQuote(\"op://Private/ClaudeAPI/credential\")),\n\tstdout = TRUE\n)\n```\n:::\n\n\nDoing this potentially comes with extra work if you choose to change the name of the item in 1Password but this alternative is there as an option, especially if you're playing around with your code and are not yet ready to commit to setting an environment variable.\n\n## Final thoughts\n\nAdmittedly, this does add some extra steps to workflows (not to mention some unsightly code), however, this trade-off is worth it for the privacy and security benefits it offers. Moreover, the increased cognitive load of switching out expired keys is abated somewhat as we can update secrets in a single place, a 1Password vault, without touching our R scripts or environment. This is especially helpful if you use these secrets across multiple contexts.", 6 6 + "supporting": [], 9 7 "filters": [ 10 8 "rmarkdown/pagebreak.lua" 11 9 ],

+1

_quarto.yml

reviewed

··· 47 47 fontcolor: "#070a0c" 48 48 fontsize: 14pt 49 49 linkcolor: "#183e4d" 50 50 + code-overflow: wrap 50 51 metadata-files: 51 52 - drafts.yml

+1 -1

drafts.yml

reviewed

··· 1 1 website: 2 2 drafts: 3 3 - - posts/using-1password-secret-references-in-R/index.qmd 3 3 + - none

+1 -2

posts/the-basics-of-duckdb-in-r/index.qmd

reviewed

··· 1 1 --- 2 2 title: "The basics of DuckDB in R" 3 3 - author: Rory Lawless 4 3 date: 2025-03-30 5 4 lastmod: 2026-01-01 6 5 format: html ··· 66 65 filter(col_10 == "some_value") 67 66 ``` 68 67 69 69 - Aside from the `read_csv_duckdb()` function, the rest of the code will be familiar to anyone who has used {dplyr} before. The main advantage of using {duckplyr} over writing SQL and using the [{DBI}](https://dbi.r-dbi.org) package is readability - using common {dplyr} functions makes it accessible to a wider range of users. This is a big benefit for teams where not everyone is comfortable reading or writing SQL. 68 68 + Aside from the `{{r}} read_csv_duckdb()` function, the rest of the code will be familiar to anyone who has used {dplyr} before. The main advantage of using {duckplyr} over writing SQL and using the [{DBI}](https://dbi.r-dbi.org) package is readability - using common {dplyr} functions makes it accessible to a wider range of users. This is a big benefit for teams where not everyone is comfortable reading or writing SQL. 70 69 71 70 Additionally, should the original author fall off the face of the earth, the code is still maintainable by others and readily adapted to eliminate the dependency on DuckDB. 72 71

posts/using-1password-secret-references-in-R/20260102-1password-prompt.png

reviewed

This is a binary file and will not be displayed.

+43 -12

posts/using-1password-secret-references-in-R/index.qmd

reviewed

··· 1 1 --- 2 2 title: "Using 1Password Secret References in R" 3 3 - author: Rory Lawless 4 4 - date: 2026-01-04 3 3 + date: 2026-01-02 5 4 format: html 6 6 - draft: true 7 5 --- 8 6 9 9 - If you're like me, you store your API keys in your .Renviron file and forget about them. Not only is it risky behaviour to store them in plaintext, it is also a nightmarish way to manage and rotate out keys when needed. 7 7 + If you’re like me, you’ve stored your API keys in your .Renviron file and forget about them. Not only is it risky to store them in plaintext, it’s also a nightmarish way to manage and rotate out keys when needed. 10 8 11 11 - 1Password offers a great solution managing your sensitive data, it even has a dedicated API Credentials item type for people super into taxonomy. The real magic, however, happens when you introduce yourself to [1Password CLI and its handy secret references](https://developer.1password.com/docs/cli/secret-reference-syntax) feature. 9 9 + ## 1Password to the rescue 12 10 13 13 - I won't go into detail on installation and configuration, [the documentation does a better job than I would](https://developer.1password.com/docs/cli/get-started). Once you have migrated an API key to 1Password, you can refer to the secret using its URI instead of including it in plain text in your .Renvion or R script. 11 11 + 1Password offers a great solution for managing sensitive data. It even has a dedicated API Credentials item type for people super into taxonomy. The desktop and mobile apps are great for day-to-day internet usage but the real magic happens when you introduce yourself to [1Password CLI and its handy secret references](https://developer.1password.com/docs/cli/secret-reference-syntax) feature. 14 12 15 15 - The URI of a credential comes in this format: `op://<vault-name>/<item-name>/[section-name/]<field-name>`. For example, an Anthropic API key save as an item named "ClaudeAPI" inside your 'Private' with the value in a field called 'credential' can be referred to as `op://Private/ClaudeAPI/credential` (note, the `[section-name/]` element is only required if the field is under a named section within the item). 13 13 + I won't go into detail on installation and configuration, [the documentation does a better job than I ever could](https://developer.1password.com/docs/cli/get-started). However, once you have added an API key to 1Password, you can refer to it using its URI (also called a Secret Reference) instead of including it as plaintext in your .Renviron or R script. The URI of a credential comes in this format: 16 14 17 17 - The URI of your credential can then replace the plain text API key anywhere you were storing it. Using our ClaudeAPI, our key=value in .Renviron would look like: 15 15 + ``` {#uri-example style="text-align: center;"} 16 16 + op://<vault-name>/<item-name>/[section-name/]<field-name> 17 17 + ``` 18 18 + 19 19 + For example, an Anthropic API key saved as an item named "ClaudeAPI" inside our "Private" vault with the value in a field called "credential" can be referred to using the URI `op://Private/ClaudeAPI/credential` (note, the `[section-name/]` element is only required if the field is under a named section within the item). The URI of your credential can then replace the plain text key in your .Renviron or anywhere else it was lurking. Using our ClaudeAPI example, the entry in .Renviron would look like: 18 20 19 19 - ``` 21 21 + ``` {#envvar-example style="text-align: center;"} 20 22 ANTHROPIC_API_KEY="op://Private/ClaudeAPI/credential" 21 23 ``` 22 24 23 23 - Our key is stored inside our 1Password vault, safe from prying eyes and accidental exposure. To access this environment variable in a script, we would typically write something like ``r Sys.getenv("ANTHROPIC_API_KEY")``, while this will technically run you will receive an error as you will be attempting to pass the literal secret reference to your API call. We will need to alter this code in two ways, the first is to use one of [the methods 1Password CLI provides](https://developer.1password.com/docs/cli/secrets-scripts) for loading secrets into code, the one we will use in our R script is `op read`. 25 25 + ## Out of 1Password, into R 24 26 25 25 - The second change, calling the function `system2()` from base, which lets us run `op read` (or any system command) from our R script. 27 27 + Our key is now stored inside our 1Password vault, safe from prying eyes and accidental exposure. In our less secure days, we would typically run `{{r}} Sys.getenv("ANTHROPIC_API_KEY")` to access the value of the environment variable in a script. While this still works, any API call that uses this value will likely fail as the value returned is the literal URI string, not the secret. 28 28 + 29 29 + To address this, we need to alter the code in two ways. The first is to use one of [the methods 1Password CLI provides](https://developer.1password.com/docs/cli/secrets-scripts) for loading secrets into code, the one we will use in our R script is `op read`. This simply allows us to read the secret into our environment. 30 30 + 31 31 + The second change, calling the function `{{r}} system2()` from {base}, lets us run `op read` (indeed, any system command) from our R script and returns the actual API key. 26 32 27 33 ```{r system2-example} 28 34 #| eval: false ··· 33 39 stdout = TRUE 34 40 ) 35 41 36 36 - ``` 42 42 + # Note, our call to Sys.getenv() needs to be quoted using shQuote() to mitigate issues with spaces and special characters 43 43 + 44 44 + ``` 45 45 + 46 46 + When we execute the example code, 1Password will prompt us to approve access to the secret (illustrated in @fig-authimage). Once we click "Authorize", the secret value is returned as a character vector (setting `stdout = TRUE` in `system2()` makes this happen). 47 47 + 48 48 + {#fig-authimage fig-alt="A screenshot of a 1Password prompt to 'Allow Positron to get CLI access' with the option to 'Cancel' or 'Authorize'." width="390"} 49 49 + 50 50 + We can alter the above code to assign a name to the returned character vector or we can run the function call directly within the API call function. Furthermore, we can jettison `Sys.getenv()` entirely and pass the URI directly, like so: 51 51 + 52 52 + ```{r system2-example2} 53 53 + #| eval: false 54 54 + 55 55 + system2( 56 56 + "op", 57 57 + args = c("read", shQuote("op://Private/ClaudeAPI/credential")), 58 58 + stdout = TRUE 59 59 + ) 60 60 + 61 61 + ``` 62 62 + 63 63 + Doing this potentially comes with extra work if you choose to change the name of the item in 1Password but this alternative is there as an option, especially if you're playing around with your code and are not yet ready to commit to setting an environment variable. 64 64 + 65 65 + ## Final thoughts 66 66 + 67 67 + Admittedly, this does add some extra steps to workflows (not to mention some unsightly code), however, this trade-off is worth it for the privacy and security benefits it offers. Moreover, the increased cognitive load of switching out expired keys is abated somewhat as we can update secrets in a single place, a 1Password vault, without touching our R scripts or environment. This is especially helpful if you use these secrets across multiple contexts.