Personal Data
Projects for taking control of your own data, and to use this for betterment.
Personal Data is split into roughly three phases:
- Data retrieval: You need to get your data.
- Data storage: You need to store your data somehow.
- Data presentation: You need some way to use the data.
Data Retrieval
These are systems for retrieving personal data:
- Scrobbing is the word for tracking your personal music history. Entertainment oriented.
- Maloja is a self-hosted scrobble site. Easy setup through docker-compose.
- Multi-Scrobbler is a system for importing data to your Scrobble server (like Maloja) from some number of music services (Spotify, Google Music, etc.)
- Gandalf Network is an attempt to allow users to query their own data. Currently supports selected endpoints from YouTube, Uber Eats, PlayStation, X. Entertainment oriented.
- Step counting involves automatic counting of the number of steps you take in a day. Most modern smart phones have built-in step counter hardware, though dedicated non-smart step counters are available (not useful for this list). Health oriented.
- Privacy Friendly Pedometer App by SecUSo for Android. Has a basic export function. I used to use this, but it suddenly died on my, and the project appears mostly dead.
- garminconnect library for Garmin Watches allows for authenticating using the unofficial Garmin API.
- Withings Pseudo-smart Watches exposes an API, though it is unclear whether individuals have easy access to the API.
- Walkscape can expose the step count of any individual users, allowing a scraper to download the steps, and perform the differential between scraped amounts to determine steps/hour/day.
Data Retrieval, Own Projects
My own projects for retrieving data:
- Fin Depo is a library for implementing financial depository fetching. It mainly supports crypto-currencies and investment banks. Personal finance oriented
- Personal Data (Project) is a system for downloading some types of personal data from various sources. Fetchers are implemented when they fill one of my needs. Sub-projects:
- (Git) Time Tracker: small program for heuristically estimating time spent doing something. Originally built for tracking time spent working on different projects, but scope expanded to support most event-based data.
- Obsidian Import: Too for importing event-based and daily statistics into Obsidian.
Data Storage & Data Presentation
There are many ways to store your data, many of them with integrated presentation system. Some of the ways I've seen data being stored are:
- Just a bunch of CSV files. No presentation inherit. Needs third party system to process, such as NuShell.
- In Excel sheets. Some presentation inherit. Easy to query over.
- As tables in a SQL Database. Little if any presentation inherit. Easy to query over.
- As metadata in an Obsidian vault. Some presentation inherit. This is the one I am presently preferring, due to the clear advantage of being able to associate free-style notes with structured data.
- As a relational database, such as Wikibase. Some presentation inherit, but not a lot.
- Special built systems such as sloph.
Background
The 21st century is the century of information and data. We produce data at alarming rates, most of which goes to shadowy companies and governments, who use it in an attempt to sell us random crap. But this is your data, and you should be able to control who has access to it, and should be able to inspect it yourself. GDPR in the EU has helped with the first goal, but the second is still way out of reach.
This list focuses on accessing, viewing and understanding data you are already producing, or could produce without changing your workflows or life. A requirement for this list is that you do not need to manually input a lot of data, so this list will not cover nutrient tracking or similar, as this is often quite labour intensive.
