This is the first development log for my project Exist Integrations.

I’ve been using the site Exist for years. Back in July 2021, I announced the launch of a project I was working on for a while, Exist Integrations. Exist Integrations pulls values from data sources: Trakt, WhatPulse, Toggl Track, and You Need a Budget and stores them in your Exist Dashboard. This allows you to see the data and link it to Exist’s correlation engine.

Exist went through a major overhaul in the backend. They announced the V2 of their API in August. It brought several exciting enhancements: create custom attributes and incremental updates!

I started work and then things stalled. I finally got back to work and today I’m excited to show a sneak peak of what will be coming with the new Exist Integrations v2:

In this post I will go through the different things that I had to handle in order to accomplish the progress. I’ll plan on doing this as I move towards the launch of Exist Integrations v2.

Lessons learned

Starting over fresh is rewarding to a point. The original code base for Exist Integrations was written as I was learning Laravel and getting back into development. There was a lot of choices that I made that I did not like. When looking at the Exist API V2 I decided to rewrite the application on a new server. Things started really quickly and I was having fun. Then edge cases came up and plans to migrate everyone to the new server and I got overwhelmed. Work stalled for two months. Now I’m just trying to get things back up and running so I can take advantage of the new API and migrate over everyone before the V1 is deprecated.

New Attributes will be really exciting! This is exciting to me. For those that don’t know, before Exist V2 came around you could only send data to the defined attributes on the site. From WhatPulse, the only relevant attribute was keystrokes. With V2, I’m able to pull more information from WhatPulse.

Incremental Update will make life easier in the long term and harder in the short term. One of the reasons why I wanted to rewrite the code base is that shortly before the V2 API was released my server experienced some major issues. The CPU was maxing out when processing the YNAB data. This was due to my data retention and the need to aggregate all information by day to send to Exist. I was able to stabalize the server but I realized Exist Integrations does need to store data long-term. It should just take data from the source and send it to Exist. With the incremental update, Exist Integrations can send just the transactional data and Exist will summarize it. However, when the rubber hit the road it presents some interesting issues. Exist Integrations has to keep track of what it has sent so it doesn’t increment too high. Then it’ll be an interesting challenge on the migration since Exist Integrations v1 sends the entire data and v2 will use the incremental logic. V2 can’t just send incremental data as anything still loaded to Exist will then be duplicated. I haven’t fully solved this problem yet, but there are some options.

Next steps

I have until February to get everything migrated. Thankfully we’re going into the holiday season and I’ll be taking off a lot of time from work. I have only done the WhatPulse integration. It’s running for my account so I can monitor the logic. The UI has to be addressed since it is very ugly. Once WhatPulse is done, I’ll move to the other three integrations and the migration.

I look forward to getting through this and writing more about it in the next couple weeks.